yuris/titik-gambut-next
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

feat(telemetry): live telemetry ingestion, server-side clustering, and GeoParquet historical archive on R2 #36

New issue
Closed
opened 2026-10-14 22:28:34 +00:00 by yuris · 0 comments
yuris commented 2026-10-14 22:28:34 +00:00
Owner
Copy link

What to build

Implement an end-to-end telemetry pipeline that:

  1. Performs server-side spatial grid clustering (O(N)) and payload pruning on live KLHK SiPongi+ hotspots to eliminate client-side UI freezing.
  2. Implements background Stale-While-Revalidate (SWR) caching and snapshot uploads to Cloudflare R2 (telemetry/current/latest-clusters.json).
  3. Compiles historical SiPongi satellite detections into partitioned OGC GeoParquet archives on Cloudflare R2 (telemetry/archive/year=YYYY/month=MM/data.parquet).
  4. Implements a serverless historical query endpoint (/api/hotspots/historical) using DuckDB to query remote GeoParquet archives over HTTP byte-range requests without requiring a stateful database.

Technical specifications

  1. Target files:

    • src/lib/gis/spatial-clustering.ts: Refactor clusterHotspots() to use a 2D spatial grid hash map (Math.floor(lon / eps), Math.floor(lat / eps)), checking only adjacent 3x3 cells. Execution time must drop from O(N^2) (~8.6s) to O(N) (<50ms) for 20,000 points.
    • src/app/api/hotspots/route.ts: Pre-cluster and prune payload on the server. Return pre-calculated clusters (~120 KB) and compact coordinate arrays instead of 7.8 MB of raw GeoJSON. Add background revalidation so clients never block on SiPongi upstream latency.
    • scripts/ingest-hotspots-geoparquet.py: Python CLI ingestion script fetching SiPongi historical periods, validating coordinates against Indonesia bounding box [94.0, -11.5, 141.5, 6.5], generating GeoParquet with Snappy/Zstd compression and row-group bounding boxes, and syncing to R2.
    • src/lib/services/geoparquet-query.ts: Embedded DuckDB query interface executing spatial SQL queries (read_parquet('https://tiles.gambut.nokopia.com/telemetry/archive/...')) using HTTP range requests.
    • src/app/api/hotspots/historical/route.ts: Historical query API route supporting temporal (startDate, endDate), spatial (provinsi, bbox), and attribute (confidence_level) filtering.
    • .forgejo/workflows/sync-tiles-r2.yml: Add step to run daily historical telemetry ingestion and Parquet sync to R2.

  2. GeoParquet Schema & Partitioning:

    • Columns: hs_id (VARCHAR), geometry (GEOMETRY / WKB Point), lon (DOUBLE), lat (DOUBLE), waktu_utc (TIMESTAMP), waktu_wib (VARCHAR), sumber (VARCHAR), confidence_level (VARCHAR), counter (INTEGER), provinsi (VARCHAR), kabkota (VARCHAR), kecamatan (VARCHAR), desa (VARCHAR), peat_depth (VARCHAR), is_gambut (BOOLEAN).
    • Partition structure on R2: telemetry/archive/year=YYYY/month=MM/data.parquet.
    • Compression: Parquet Snappy/Zstd dictionary encoding with row group statistics and spatial bbox metadata.

  3. Performance & SLA Requirements:

    • Live endpoint /api/hotspots response time: <50ms on cache HIT, <200ms on server-side re-clustering.
    • Client payload: <200 KB (down from 7.8 MB raw GeoJSON).
    • Historical range query latency: <500ms for single-month queries via DuckDB HTTP range requests.

Acceptance criteria

  • Spatial grid clustering executes in <50ms for 20,000 points with 100% test coverage for boundary points and density clusters.
  • Client payload size for live hotspots is reduced by >90% with zero UI freezing or dropped frames during map mount.
  • GeoParquet ingestion script produces valid OGC GeoParquet 1.1 compliant files with embedded coordinate reference system (EPSG:4326) and row-group bounding boxes.
  • Ingestion script uploads daily and monthly partitioned Parquet files to Cloudflare R2 bucket.
  • /api/hotspots/historical successfully answers time-series queries using DuckDB HTTP range requests without downloading entire Parquet files.
  • Full Vitest and Python test suites pass without warnings or errors.

Authoritative Specifications & Reference Documentation

### What to build Implement an end-to-end telemetry pipeline that: 1. Performs server-side spatial grid clustering ($O(N)$) and payload pruning on live KLHK SiPongi+ hotspots to eliminate client-side UI freezing. 2. Implements background Stale-While-Revalidate (SWR) caching and snapshot uploads to Cloudflare R2 (`telemetry/current/latest-clusters.json`). 3. Compiles historical SiPongi satellite detections into partitioned OGC GeoParquet archives on Cloudflare R2 (`telemetry/archive/year=YYYY/month=MM/data.parquet`). 4. Implements a serverless historical query endpoint (`/api/hotspots/historical`) using DuckDB to query remote GeoParquet archives over HTTP byte-range requests without requiring a stateful database. ### Technical specifications 1. **Target files**: - `src/lib/gis/spatial-clustering.ts`: Refactor `clusterHotspots()` to use a 2D spatial grid hash map (`Math.floor(lon / eps)`, `Math.floor(lat / eps)`), checking only adjacent 3x3 cells. Execution time must drop from $O(N^2)$ (~8.6s) to $O(N)$ (&lt;50ms) for 20,000 points. - `src/app/api/hotspots/route.ts`: Pre-cluster and prune payload on the server. Return pre-calculated clusters (~120 KB) and compact coordinate arrays instead of 7.8 MB of raw GeoJSON. Add background revalidation so clients never block on SiPongi upstream latency. - `scripts/ingest-hotspots-geoparquet.py`: Python CLI ingestion script fetching SiPongi historical periods, validating coordinates against Indonesia bounding box `[94.0, -11.5, 141.5, 6.5]`, generating GeoParquet with Snappy/Zstd compression and row-group bounding boxes, and syncing to R2. - `src/lib/services/geoparquet-query.ts`: Embedded DuckDB query interface executing spatial SQL queries (`read_parquet('https://tiles.gambut.nokopia.com/telemetry/archive/...')`) using HTTP range requests. - `src/app/api/hotspots/historical/route.ts`: Historical query API route supporting temporal (`startDate`, `endDate`), spatial (`provinsi`, `bbox`), and attribute (`confidence_level`) filtering. - `.forgejo/workflows/sync-tiles-r2.yml`: Add step to run daily historical telemetry ingestion and Parquet sync to R2. 2. **GeoParquet Schema & Partitioning**: - Columns: `hs_id` (VARCHAR), `geometry` (GEOMETRY / WKB Point), `lon` (DOUBLE), `lat` (DOUBLE), `waktu_utc` (TIMESTAMP), `waktu_wib` (VARCHAR), `sumber` (VARCHAR), `confidence_level` (VARCHAR), `counter` (INTEGER), `provinsi` (VARCHAR), `kabkota` (VARCHAR), `kecamatan` (VARCHAR), `desa` (VARCHAR), `peat_depth` (VARCHAR), `is_gambut` (BOOLEAN). - Partition structure on R2: `telemetry/archive/year=YYYY/month=MM/data.parquet`. - Compression: Parquet Snappy/Zstd dictionary encoding with row group statistics and spatial bbox metadata. 3. **Performance & SLA Requirements**: - Live endpoint `/api/hotspots` response time: &lt;50ms on cache HIT, &lt;200ms on server-side re-clustering. - Client payload: &lt;200 KB (down from 7.8 MB raw GeoJSON). - Historical range query latency: &lt;500ms for single-month queries via DuckDB HTTP range requests. ### Acceptance criteria - [x] Spatial grid clustering executes in &lt;50ms for 20,000 points with 100% test coverage for boundary points and density clusters. - [x] Client payload size for live hotspots is reduced by &gt;90% with zero UI freezing or dropped frames during map mount. - [x] GeoParquet ingestion script produces valid OGC GeoParquet 1.1 compliant files with embedded coordinate reference system (EPSG:4326) and row-group bounding boxes. - [x] Ingestion script uploads daily and monthly partitioned Parquet files to Cloudflare R2 bucket. - [x] `/api/hotspots/historical` successfully answers time-series queries using DuckDB HTTP range requests without downloading entire Parquet files. - [x] Full Vitest and Python test suites pass without warnings or errors. --- ### Authoritative Specifications & Reference Documentation - **Titik Gambut Architecture**: [`docs/ARCHITECTURE.md`](https://git.nokopia.com/yuris/titik-gambut-next/src/branch/main/docs/ARCHITECTURE.md) - **OGC GeoParquet 1.1.0 Specification**: [OGC GeoParquet](https://geoparquet.org/) — Columnar vector data format on Apache Parquet. - **DuckDB Spatial Extension**: [DuckDB Spatial Documentation](https://duckdb.org/docs/extensions/spatial/overview.html) — Zero-database spatial analytics over HTTP range requests. - **KLHK SiPongi+ API Contract**: [`src/lib/services/sipongi.ts`](https://git.nokopia.com/yuris/titik-gambut-next/src/branch/main/src/lib/services/sipongi.ts)
yuris changed title from Issue 23: [Backend] Automated live telemetry ingestion & spatial enrichment worker (rustgis ingest-telemetry) to [Backend] Automated live telemetry ingestion & spatial enrichment worker (rustgis ingest-telemetry) 2026-10-14 22:31:29 +00:00
yuris changed title from [Backend] Automated live telemetry ingestion & spatial enrichment worker (rustgis ingest-telemetry) to Automated live telemetry ingestion & spatial enrichment worker (rustgis ingest-telemetry) 2026-10-14 22:38:35 +00:00
yuris changed title from Automated live telemetry ingestion & spatial enrichment worker (rustgis ingest-telemetry) to feat(telemetry): live telemetry ingestion & spatial peatland enrichment service 2026-10-15 01:26:42 +00:00
yuris changed title from feat(telemetry): live telemetry ingestion & spatial peatland enrichment service to feat(telemetry): live telemetry ingestion, server-side clustering, and GeoParquet historical archive on R2 2026-10-15 05:48:13 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/issue-51-concessions-layer
feat/issue-46-dual-mode-tile-url-resolver
feat/issue-44-45-protomaps-basemap-cutover
No results found.
Labels
Clear labels   
backend
Backend task

  
frontend
Frontend task

No labels
backend
frontend
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yuris/titik-gambut-next#36
No description provided.