Raster Serving — titiler + Cloud-Optimized GeoTIFF¶

Climate-Lama serves hazard rasters (river flood, wildfire, European windstorm) as Cloud-Optimized GeoTIFFs through titiler, a FastAPI app that translates GET /cog/tiles/{tileMatrixSetId}/{z}/{x}/{y}.png?url=<cog>&bidx=<band> into on-the-fly raster tiles. MapLibre renders them via a plain raster source — no GeoJSON round-trip, no server-side rasterisation of vector pixels.

The browser never talks to titiler directly (issue #355 / #385): MapLibre requests the backbone's own same-origin tile proxy (GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png), which resolves dataset_id to the dataset's cog_path and fetches from titiler server-side. No titiler host or S3 path is ever visible in an API response or a browser network request.

See ADR-029 for the bucket layout that underpins this pipeline.

Architecture¶

MapLibre GL (browser)
    │  GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png[?bidx=N]  (same-origin, cookie-authed)
    ▼
Climate-Lama backbone tile proxy (api/v1/tiles.py, ADR-029 read-through cache)
    │  GET /cog/tiles/WebMercatorQuad/{z}/{x}/{y}.png?url=s3://climate-lama/processed/<org>/<ds>/hazard.tif&bidx=1
    ▼
titiler           (stateless, reads COGs from object storage)
    │  range-GETs the COG header + overviews
    ▼
MinIO / Hetzner Object Storage (COG at processed/<org>/<ds>/hazard.tif per ADR-029)

The worker produces the COG up front (see Conversion pipeline below); titiler serves it on demand. Because COGs embed their own overviews, titiler does not need to generate pyramids — it seeks into the pre-built levels via HTTP range requests.

Conversion pipeline¶

Celery task — `convert_to_cog(hazard_dataset_id, raw_raster_key, org_id?)`¶

Source: src/climate_lama/worker/tasks.py

The task:

Downloads the raw raster at raw_raster_key from MinIO (the ADR-029 raw/ layout).
Runs rio-cogeo's cog_translate with the deflate profile (tiled, with overviews, DEFLATE compression — titiler's recommended defaults for web tiles).
Uploads the result to processed/{org_id_or_public}/{dataset_id}/hazard.tif per ADR-029.
Updates hazard_datasets.cog_path with the new key so GET /v1/hazards/{id}/layers starts emitting a same-origin proxy tile URL and GET /v1/tiles/cog/{dataset_id}/... has a COG to resolve.

The task is routed to the existing compute Celery queue; no new worker container is required. The rio-cogeo dependency lives in the worker optional-dependency group (pip install -e .[worker]).

Celery task — `build_dataset_cog(hazard_dataset_id, org_id?)`¶

Source: src/climate_lama/worker/tasks.py

The builder the ingest pipeline actually uses (issue #548). Instead of a raw raster it reads the dataset's committed intensity artifact, hazards/{dataset_id}/intensity.npz, which since #542 carries the grid definition and the per-column pixel indices alongside the CSR matrix. The task:

Reads hazard_datasets.npz_path and fetches the artifact from MinIO.
Densifies the CSR back onto its own grid as a multi-band GeoTIFF — one band per event, in return_periods order, written in row blocks so the dense (height, width) band never materialises. Cells the sparse matrix does not store are written as 0.0 and declared as the raster's nodata sentinel: the sparse reader drops nodata and zero cells identically, so an absent cell is exactly a cell the source declared as "no hazard here", and tiling it as nodata leaves dry land transparent.
Runs the same cog_translate, uploads to the same processed/... key, and stamps cog_path — steps 2–4 above.

Building from the committed artifact makes the COG agree with the dataset by construction: the same grid its centroids are indexed against, one band per return period, and whatever clip/resampling the ingest ran with.

Pre-#542 artifacts have no grid keys and are refused with a pointer to scripts/backfill_grid_geometry.py --apply, which rewrites them to v2.

Auto-trigger on ingest¶

aggregate_and_commit — the ingest chord's terminal callback — queues build_dataset_cog for every dataset it commits (_schedule_cog_build). Every ingest path ends in that callback, so every dataset gets a COG with no per-caller parameter involved. The enqueue is best-effort: a broker hiccup costs the dataset its map layer, never its data, and scripts/backfill_dataset_cogs.py re-queues anything missed.

ingest_hazard still honours a raw_raster_key param by enqueueing convert_to_cog at dispatch time. That path is for a raster pre-staged under the ADR-029 raw/ layout by something other than the chord; no producer of ingest_hazard params sets it, and one usefully cannot — the chord's own staged copy at staging/{ingest_job_id}/... is deleted by aggregate_and_commit on the success path, so a conversion scheduled at dispatch time would race the cleanup. Before #548 this was the only trigger, which is why cog_path was NULL for every dataset ever ingested.

Backfilling existing datasets¶

scripts/backfill_dataset_cogs.py enqueues build_dataset_cog for every dataset with cog_path IS NULL. Dry-run by default; --apply queues the builds. Datasets whose NPZ predates #542 are reported as blocked — run scripts/backfill_grid_geometry.py --apply first, then re-run.

--rebuild (#901) is the same script pointed the other way: datasets that already have a cog_path and whose stack is deep enough that core.cog_layout.cog_interleave would lay them out band-interleaved. On-disk layout is chosen when a COG is written, so #885's band interleaving reaches an existing artifact only by re-rendering it. Depth comes from count(hazard_events); a dataset with a COG and no event rows is reported rather than assumed shallow. Dry-run by default here too, and unlike the backfill lane this one is not self-limiting — re-running re-renders the same datasets.

LayerSpec wiring¶

GET /v1/hazards/{id}/layers returns the UI-facing LayerSpec list. For raster hazards with a populated cog_path, the raster spec's source_url is filled in by layer_specs_for_hazard_dataset() as a same-origin, relative tile-proxy URL template (issue #355 / #385 — _proxy_raster_tile_url()):

{
  "type": "raster",
  "source_url": "/v1/tiles/cog/3f9e2b1a-.../{z}/{x}/{y}.png",
  "style": {"colormap": "Blues", "opacity": 0.7},
  "interactivity_config": {"tooltip_fields": ["intensity"]}
}

MapLibre substitutes {z}/{x}/{y} per-tile. dataset_id is threaded straight into the URL — the proxy (GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png, api/v1/tiles.py::get_cog_tile) resolves it to the dataset's cog_path and builds the titiler ?url=s3://... query itself; no S3 path or titiler host ever appears in an API response.

UI contract: source_url is relative, not absolute. The UI resolves it against its configured API base URL before handing it to MapLibre (the same base URL it already uses for every other /v1/... fetch) — no separate "tile server origin" config is needed anymore. Auth follows the same rule: the proxy is gated by require_role(Role.VIEWER) via get_auth_context (api/dependencies.py), which already accepts the browser's access_token session cookie (via="cookie") as an alternative to a bearer header — so as long as the UI's API requests are same-origin (or its cross-origin cookie/CORS setup already covers its other API calls), MapLibre's tile <img>/fetch requests carry the session automatically. No bearer-header support is needed in MapLibre's tile loader.

Datasets without a cog_path (not yet converted, or not a raster hazard) get source_url: "". The UI treats an empty string as "no layer to render" and falls back to loading states.

The titiler origin and S3 prefix remain configured via two settings, but as of #385 they are internal to the tile proxy (api/v1/tiles.py, core/tile_cache.py) — they are never embedded in a LayerSpec or exposed to the browser:

Setting	Default	Notes
`titiler_base_url`	`http://localhost:7800`	Backbone-reachable titiler origin (server-to-server only).
`titiler_s3_prefix`	`s3://climate-lama`	`s3://` scheme + bucket. Swap for the prod bucket: `s3://climate-lama-storage`.

Local dev¶

# 1. Start the full stack.
docker compose up

# 2. Verify titiler is healthy.
curl http://localhost:7800/healthz

# 3. With a COG at s3://climate-lama/processed/public/<ds>/hazard.tif,
#    fetch a tile:
#    The TileMatrixSet segment and &bidx are both mandatory: titiler >= 0.15
#    publishes no TMS-less tile route (it 404s at the route level), and a
#    multi-band COG cannot be PNG-encoded without picking one band (#831).
curl -o /tmp/tile.png \
  "http://localhost:7800/cog/tiles/WebMercatorQuad/5/16/11.png?url=s3://climate-lama/processed/public/<ds>/hazard.tif&bidx=1"
file /tmp/tile.png   # → PNG image data

titiler auto-discovers the MinIO endpoint via the env vars set in docker-compose.yml (AWS_S3_ENDPOINT, AWS_VIRTUAL_HOSTING=FALSE). No config file is involved.

Production notes¶

Cache: raster tiles are regenerable from the COG (ADR-029 invariant 4). The backbone exposes a read-through cache at GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png (issue #232), with an optional ?bidx= band selector defaulting to 1. On cache miss the proxy fetches from titiler, writes the bytes back to tiles/{dataset_id}/b{bidx}/{z}/{x}/{y}.png in the object-store bucket, and returns them — the band is part of the key so two bands of the same tile cannot collide (#831). Admin flushes the prefix via POST /v1/datasets/{id}/invalidate-tiles. Prometheus counters climate_lama_tile_cache_hits_total{tile_type} and climate_lama_tile_cache_misses_total{tile_type} track hit ratios.
Colormapping: titiler applies colormaps server-side via the &colormap_name=... query parameter. The MVP embeds the colormap in the MapLibre style rather than the tile URL; Phase 6 may move it into LayerSpec.style for server-side rendering depending on how heavy the client-side rescale is.
Worker image size: rio-cogeo is optional — it installs only when the worker image is built with pip install -e .[worker]. The core API image does not include it.

Alternatives considered¶

Rasterize to GeoJSON: rejected — payload scales with pixel count, not data complexity, and kills browsers at continental extents.
Pre-baked PNG pyramids: works but doubles object-storage usage and makes colormap iteration expensive. COG + on-the-fly titiler lets us swap colormaps without re-generating pyramids.
Embedded titiler library (import titiler.core into the backbone): smaller network hop, but couples the backbone to GDAL and rasterio at request time. The container path keeps the backbone slim and lets ops scale the tile server independently.