Climate-Lama -- Complete Project Reference¶
Purpose: Document for the backbone platform. Everything needed to understand, extend, and deploy Climate-Lama.
Repository:
github.com/CortoMaltese3/climate-lama(private) License: Apache 2.0 (confirmed, review before making repo public) Python: 3.12 (pinned for CLIMADA/Docker compatibility) Status: PoC complete (River Flood, Greece, Buildings, EAD, GeoJSON). Control panel UI on main.
1. What This Is¶
Positioning superseded 2026-07-26 by the goal reset in docs/plan/plan.md (Extended End Goal): personas are now ranked (P1 regional/mid-market insurer, conditional — see plan), the business model is commercial white-label instances, and licensing is deferred pending validation. The paragraphs below predate that reset; treat plan.md as authoritative where they disagree. Technical reference content in this file remains valid.
Climate risk intelligence platform built for extensibility. Backend-first, API-driven, self-hostable.
Target users: researchers, government agencies (GIZ, UNU, EIOPA-type), NGOs, development banks, smaller insurers -- organizations that need transparency, reproducibility, and self-hosting.
Not competing with: enterprise SaaS (delta-climate, Jupiter Intelligence). Serves the development/research/regulatory market where open-source, auditable methods matter.
What it provides:
- Unified data access to hazard, exposure, and vulnerability datasets
- Model-agnostic compute with climate-lama-engine as the default (pluggable via ModelInterface)
- Clean REST API for downstream consumers (web apps, desktop apps, BI tools, AI agents)
- Offline-capable architecture for sensitive data environments
2. Architecture¶
Two deployable units¶
HTTP
Clients -----> Core Service (FastAPI monolith, port 8000)
| | |
v v v
PostgreSQL MinIO Redis
+ PostGIS (S3) (queue/cache)
^
| Celery tasks
Compute Worker
Core Service (FastAPI):
- API layer (REST, /v1/ versioned)
- Ingest module (data acquisition, ETL)
- Data module (normalization, validation)
- Catalog module (search, metadata)
Compute Worker (Celery):
- climate-lama-engine adapter (~200MB container)
- ModelInterface abstraction (pluggable engines)
- Async job processing
Storage: - PostgreSQL 16 + PostGIS 3.4 -- normalized domain data, spatial queries - MinIO -- raw files, intensity matrices (NPZ), result arrays (S3-compatible) - Redis 7.x -- Celery broker/backend, caching
Source tree¶
src/climate_lama/
api/v1/ # HTTP layer only -- no business logic
exposures.py # Upload CSV, list, CRUD
hazards.py # Ingest trigger, list, centroid assignment
compute.py # Impact calculation trigger
jobs.py # Job status polling
results.py # Result retrieval, GeoJSON generation
impact_functions.py # Seed and list impact functions
core/ # Business logic -- no HTTP, no DB queries
hazard_ingest.py # GeoTIFF parsing, NPZ creation, centroid bulk insert
centroid_assignment.py # KDTree nearest-neighbor, bulk SQL update
exposure_service.py # CSV parsing, dedup, create exposures
impact_function_seeder.py # Seed JRC flood impact functions
models/ # Pydantic + SQLAlchemy
hazard.py # HazardDataset, HazardEvent, HazardCentroid
exposure.py # ExposureDataset, Exposure, ExposureCentroidAssignment
job.py # Job (status, celery_task_id, timestamps)
result.py # ImpactResult (ead, aai, impact_per_event, eai_exp_path)
impact_function.py # ImpactFunction (mdd_x/y, paa_x/y curves)
enums.py # HazardType, ExposureType, JobStatus, JobType
db/
session.py # Async (asyncpg) + sync (psycopg2) session factories
base.py # SQLAlchemy declarative base
repositories/
hazard_repository.py # CRUD + get_by_name_and_type (dedup)
exposure_repository.py # CRUD + exists_by_name (dedup)
job_repository.py
result_repository.py
impact_function_repository.py
worker/
celery_app.py # Celery config, queue routing
tasks.py # compute_impact, ingest_hazard (+ centroid assignment)
models/
base.py # ModelInterface ABC + ImpactResult dataclass
engine_adapter.py # EngineAdapter (wraps climate-lama-engine)
storage/ # MinIO/S3 abstraction
config.py # Pydantic Settings (env vars)
main.py # FastAPI app, router registration, lifespan
docker/
core.Dockerfile # API container
worker.Dockerfile # Worker container (installs climate-lama-engine)
migrations/ # Alembic
tests/ # pytest + pytest-asyncio
Dependency rules¶
api/callscore/, neverdb/directlycore/callsdb/repositories/for data accessworker/is independent -- communicates via Redis queue and shared DB- Engine imports only in
worker/models/engine_adapter.py - No circular imports -- dependency flows downward
3. Tech Stack¶
| Layer | Choice | Version |
|---|---|---|
| Language | Python | 3.12 |
| API Framework | FastAPI | 0.115+ |
| Async Server | Uvicorn | latest |
| Task Queue | Celery | 5.4+ |
| Message Broker | Redis | 7.x |
| Database | PostgreSQL + PostGIS | 16 + 3.4 |
| Object Storage | MinIO | latest |
| ORM | SQLAlchemy | 2.x (async) |
| Migrations | Alembic | latest |
| Validation | Pydantic | 2.x |
| Geospatial | GeoAlchemy2, Shapely | latest |
| Containerization | Docker + Compose | latest |
| Linting/Formatting | Ruff | latest |
| Testing | pytest + pytest-asyncio | latest |
| Package manager | uv | latest |
| Engine | climate-lama-engine | 0.1.0+ |
4. API Design¶
Versioning and format¶
All endpoints: /v1/ prefix. Breaking changes require /v2/.
Success response:
Error response (ADR-028):
{ "error": { "code": "E_VALIDATION", "severity": "error", "message_en": "...", "details": {} }, "meta": { ... } }
Request validation is in the envelope too (issue #1357). A request FastAPI
refuses before the endpoint runs — a body that fails its Pydantic model, a query
or path parameter out of range — answers 422 E_VALIDATION, with FastAPI's
per-error list under details.errors (each entry's type, loc, msg,
input and, where the rule has one, ctx; a validator's own exception message
arrives under ctx.error as a string). POST /v1/auth/refresh with {}:
{ "error": { "code": "E_VALIDATION", "severity": "error", "message_en": "Input data failed validation.",
"details": { "errors": [ { "type": "missing", "loc": ["body", "refresh_token"], "msg": "Field required", "input": {} } ] } },
"meta": { ... } }
This is a wire change: before #1357 these refusals were FastAPI's raw
{"detail": [...]}, outside the envelope and without a code. The status is
unchanged, and the list is the same data one level deeper (detail →
error.details.errors). No client in this org's repos parsed the raw shape;
the Python SDK already maps E_VALIDATION.
Exceptions: GET /v1/results/{id}/geojson returns a raw FeatureCollection, and
GET /v1/results/{id}/report returns a raw summary object, CSV, or PDF bytes
depending on ?format= — all without the envelope. Errors from both still use it.
Endpoints¶
| Method | Path | Description |
|---|---|---|
| GET | /health |
Health check |
| GET | /v1/info |
Build and deploy identity, unauthenticated — data carries exactly version, environment, git_sha, data_region, live_feed_tick_seconds and features. data_region (issue #1669, SH-24) is the deployment setting CL_DATA_REGION (e.g. "European Union"), shown read-only in Organisation settings, and null when unset. live_feed_tick_seconds (issue #1444) is the interval the live-feed-tick beat runs at — the finest cadence any live feed on this deployment is actually polled at — and is a top-level sibling of features, never a key inside it, because that object carries booleans only. features is a feature-advertisement object (issue #1280): booleans only, never a URL, host, provider name or key, so a client can decide whether to mount a surface without the endpoint becoming a configuration dump. live_mode is what mounts the UI's Live nav entry; geocoding is geocoder_enabled and geocoder_advertised, never the master switch alone; features.assistant (issue #1493) is assistant_enabled and anthropic_api_key, never the flag alone. scripts/verify-deploy.sh reads git_sha — see below |
| POST | /v1/assistant/messages |
One turn of the bounded assistant (issue #1493, ADR-086), VIEWER+. Body: messages (client-held history, at most 40 entries of {role:"user",text}, {role:"assistant",text?,calls:[{id,name,input}]} or {role:"tool_results",results:[{call_id,output,is_error}]}), actions (the UI's named-action manifest, at most 64 {name,description,input_schema}, name matching ^[a-z][a-z0-9_]{0,63}$, each input_schema at most 8 KB; extra manifest keys such as group/title are ignored) and optional context (UI state, at most 2 KB). Answers {reply, calls, append, stop, withheld_figures}: calls are proposals the UI executes visibly and posts back as tool_results (only names from the posted manifest ever appear), append is what the UI adds to its history verbatim (assistant turns and the results of the backbone's read-only query tools get_result_summary / get_coverage (issue #1495) and find_portfolios(q) → [{id,name,asset_count}], find_exposure_datasets(country, region?) → {clip, datasets:[{id,name,value_unit,exposure_type,is_default}]} get_job(job_id) → {status,result_id,error} (issue #1603) and list_hazard_options({portfolio_id | exposure_dataset_ids}) → [{haz_type,climate_driven,state,reason,money,money_reason,cells}] (issue #1604) and get_portfolio_rollup(portfolio_id, scenario?, horizon_year?) → {id,name,scenario,horizon_year,member_count,headline,hazards,value_weighted,ead_by_hazard} (issue #1650)), stop is end_turn, action_calls, refused, max_tokens or max_rounds, and withheld_figures counts the unsourced figures replaced by [figure withheld]. The backbone never executes a named action and stores no transcript; an answered request leaves assistant.message, assistant.action_proposed and assistant.tool_called audit rows (issue #1496, see "Assistant audit and request budget" below); an action named like a query tool is 422. See "Assistant query tools and the figure guard" below. Over-limit or malformed bodies are 422 E_VALIDATION; 429 E_RATE_LIMITED (details {limit, retry_after_seconds}) once the caller's organization has spent its ASSISTANT_RATE_LIMIT budget; 503 E_ASSISTANT_UNAVAILABLE with details.reason disabled (ASSISTANT_ENABLED off), no_key, no_provider_adapter, provider_error or timeout — see "Assistant settings" below |
| GET | /v1/assistant/options |
The assistant's model choices (issue #1602, AS-1 (b)), any authenticated role. Answers exactly provider (anthropic), models (ASSISTANT_MODEL_CHOICES, in order), efforts (["low","medium","high"]), default (the deployment's ASSISTANT_MODEL, which may lie outside models), default_effort (the deployment's ASSISTANT_EFFORT, unchanged by an override) and effective ({model, effort, source}, source org when the organization's override supplies them, else deployment). Never carries the provider key. An org admin sets the override with PUT /v1/orgs/{id}/settings {"assistant": {"model", "effort"}} (a model outside the allowlist or another effort is 422 E_VALIDATION, {"assistant": null} clears it; each such write leaves one org_settings.updated audit row), and the next POST /v1/assistant/messages from that organization sends it and records model and effort on its assistant.message audit row |
| GET | /v1/engines |
List registered compute engines — availability, default, and declared supported_hazards (issue #1054), each engine's installed version and its runtime (in_process or http_sidecar, issue #1669, EN-8) — see below |
| GET | /v1/admin/engine-stats |
Platform admin (require_platform_admin, issue #1669, SH-15): ?days=30 (1–366) → {days, items}, one {engine, runs, median_seconds, failed_share} per registered engine over every organisation's usage events of the window. The engine is the one the metered job's params.engine_name names (the default engine when absent); a cache hit is not a run; an engine with no run reads runs: 0 and the other two null. An org admin without the flag gets 403 |
| GET | /v1/admin/data-requests |
ORG_ADMIN (issue #1669, SH-26, BC-9): the caller's organisation's data requests only, newest first → {items, total}, each {id, name, kind, how, source, requested_by_user_id, requested_by_name, size_bytes, status, detail, created_at}. A request is a pending_ingests row naming the asking org_id — a boundary file POST /v1/admin-boundaries/uploads queued (how: "upload") or a catalogue dataset POST /v1/admin/catalog/{name}/request-ingest asked for (how: "download", source the provider) — or a catalogue download of the organisation (how: "download", source its URL). status is read, never stored: licence (terms not accepted), waiting (detail "Waiting for platform approval"), ingesting (approved; a download pending/running), rejected (rejected or ingest_failed; a download failed/cancelled, detail its error_code) or ready (an approved boundary file at once, any other request once a data_ingest job naming it completed; a download succeeded). Withdrawn and dismissed rows are left out. An analyst gets 403 |
| POST | /v1/admin/data-requests/{id}/accept-licence |
ORG_ADMIN (issue #1669): accept a licence request's provider terms for the organisation — it moves to waiting and stamps licence_accepted_at/licence_accepted_by_user_id. 404 E_NOT_FOUND for another organisation's request; 409 E_VALIDATION for a request in another state |
| POST | /v1/admin/data-requests/{id}/withdraw |
ORG_ADMIN (issue #1669): withdraw a licence or waiting request — it leaves both the organisation's list and the platform's Ingest approvals, and approve/reject answer 404 for it. 404 for another organisation's request, 409 once decided |
| POST | /v1/admin/data-requests/{id}/dismiss |
ORG_ADMIN (issue #1669): hide a rejected request from the organisation's list. 404 for another organisation's request, 409 for a request in another state |
| GET | /v1/admin/queues |
Platform admin (issue #1669, EN-1): the impact lane's queue over every organisation's jobs → {impact: {waiting, running, workers, oldest_wait_seconds}, ingest: {waiting, running}}. The impact lane is the impact_calc jobs compute_impact runs on the impact queue (an event's impact, params.live_event_id, runs on compute and is left out); workers is the impact_worker_concurrency setting (#1662, null when unset); oldest_wait_seconds is the age of the oldest pending impact job (null when none waits); ingest counts the data_ingest jobs apart. An org admin without the flag gets 403 |
| GET | /v1/admin/failures |
Platform admin (issue #1669, PF-4): ?days=7 (1–90) → {days, items, total}, newest first, each {kind, at, org_id, org_name, actor_user_id, code, reason, exposure, haz_type, cells, job_id, job_type, scenario_id}. kind job_failed is every organisation's failed job of the window (code/reason its error_code/error_message, scenario_id the saved History item holding its run — params.scenario_id, or the Scenario whose inputs.batch_id lists the job among its batch's cells); kind run_refused is a compute refused at submission, recorded as an audit event compute.refused (below). An org admin without the flag gets 403 |
| GET | /v1/admin/feeds |
Platform admin (issue #1680, SH-26, K45): the feeds of every open watch of every organisation → {items, orgs}, each item {org_id, org_name, live_event_id, live_event_name, feed_id, kind, enabled, paused_reason, last_polled_at, cadence_seconds, effective_cadence_seconds, last_error, last_poll_summary, max_snapshots, snapshot_count}; orgs counts the organisations read. Each organisation is read through the request session scoped to it (read_each_org). 422 E_LIVE_MODE_DISABLED while live mode is off. An org admin without the flag gets 403 |
| PATCH | /v1/admin/feeds/{feed_id} |
Platform admin (issue #1680): Pause or resume any organisation's feed — body {enabled?, max_snapshots?} (extra="forbid") → 200 with the feed's /v1/admin/feeds row. The member route's rules, run under the owning organisation's scope: enabled: true clears paused_reason; a feed paused at its snapshot cap resumes for good only with max_snapshots above its snapshot_count. 404 E_LIVE_FEED_NOT_FOUND; 403 without the flag |
| POST | /v1/admin/feeds/{feed_id}/poll |
Platform admin (issue #1680): Poll now on any organisation's feed → 202 {feed_id, queued: true}, as the member route (the poll is queued on live under the feed's organisation). Same refusals: 409 E_LIVE_FEED_PAUSED, 422 E_LIVE_EVENT_CLOSED, 404, 429 past live_feed_poll_rate_limit; 403 without the flag |
| GET | /v1/admin/jobs |
Platform admin (issue #1680, SH-26): every organisation's jobs in flight — ?status=pending&status=running (repeatable; both when absent; any other state 422 E_VALIDATION), ?limit= 1–500 (default 200) → {running, pending, items}, items newest first, each {org_id, org_name, job_id, job_type, title, origin, status, created_at, started_at}. running/pending count every organisation's jobs in that state, uncapped; feed-driven jobs are included (origin). No progress fraction. An org admin without the flag gets 403 |
| GET | /v1/catalog/coverage |
What the caller's org holds, as a dense country x hazard-type grid — every loaded country x every HazardType, empty cells included as explicit zeroes (issue #1191), VIEWER+ — see below |
| GET | /v1/admin-boundaries |
List and search the loaded GADM units (?level=&iso3=&q=&limit=&offset=), {items, total}, never a polygon (issue #1468), VIEWER+ — see below |
| GET | /v1/admin-boundaries/{boundary_id} |
One loaded GADM unit by id, in exactly the list's item shape, never a polygon; 404 E_NOT_FOUND for an unknown id (issue #1589), VIEWER+ — see below |
| GET | /v1/admin-boundaries/geometry |
Outlines of one level's units intersecting a required bbox, simplified for zoom (?level=&bbox=&zoom=&limit=), a bare GeoJSON FeatureCollection with total and truncated (issue #1469), VIEWER+ — see below |
| POST | /v1/admin-boundaries/uploads |
Hold an organisation's boundary file (GeoJSON FeatureCollection, each feature a polygon with name and iso3; form level 1/2/3/custom) for a platform admin's review as a pending_ingests row of source user_upload; loads nothing until approved (issue #1670), ANALYST+ — see below |
| POST | /v1/exposures |
Upload an exposure file — CSV, CLIMADA entity XLSX, or CLIMADA Exposures HDF5 (issue #1172) — see below |
| POST | /v1/exposures/column-suggestions |
Suggest which headers of a CSV/XLSX upload are lat/lon/value (multipart file, same accept-list and byte cap as the upload); reads the header and at most 20 rows, writes nothing, no model call; an HDF5 store is 422 E_VALIDATION (issue #1497), ANALYST+ — see below |
| GET | /v1/exposure-datasets |
List exposure datasets — the primary exposure resource (?search=&country=&exposure_type=&include_superseded=&limit=&offset=), {items, total, default_dataset_id} (issues #1166, #1660). Excludes superseded rows by default so the list agrees with GET /v1/catalog/coverage (issue #1205); pass ?include_superseded=true to see them — see below |
| GET | /v1/exposure-datasets/{id} |
Read one exposure dataset: the list fields plus value_total and dropped_columns (issue #1633); another org's dataset is 404, not 403 |
| DELETE | /v1/exposure-datasets/{id} |
Delete one exposure dataset and its point rows (issue #1912), ANALYST+ and gated like DELETE /v1/hazards/{id}: only the uploader (owner_user_id) or an org admin, so a seeded or ingested dataset is admin-only (403 otherwise); another org's dataset is 404; 409 CONFLICT while an impact result, an attaching portfolio (the used-by read) or a live-event watch still uses it, with details {results, portfolios, live_events}; writes an exposure_dataset.deleted audit entry — {deleted: id} |
| GET | /v1/exposure-datasets/{id}/rows |
Page one dataset's points (?bbox=&limit=&offset=&sort=value\|name), {items, total}; every row carries a human-readable name |
| GET | /v1/exposure-datasets/{id}/used-by |
What uses one exposure dataset (issue #1634) — {results, analyses, analyses_total, portfolios}; see "Dataset used-by and versions" below |
| GET | /v1/exposure-datasets/{id}/versions |
The other ingests sharing its source identity, newest first (issue #1634) — {items, reason}; see below |
| GET | /v1/exposure-datasets/{id}/download |
Download one exposure dataset as CSV in the upload's own columns (issue #1635), VIEWER+ — text/csv, Content-Disposition: attachment; filename="<name>.csv", header name,lat,lon,value plus one column per properties key (sorted); name is the row's properties.name, blank when it has none, so the file uploads back through POST /v1/exposures with the dataset's exposure_type/value_unit to the same rows. Rows are read in keyset pages of 500 inside the request's session and only the text is streamed; another org's dataset is 404 |
| GET | /v1/exposures |
List exposure points — legacy alias of /v1/exposure-datasets; gained an optional ?dataset_id= filter in #1166 |
| GET | /v1/exposures/catalog |
List exposure datasets by grouping the point table — legacy alias of /v1/exposure-datasets (no total, no search/country, no per-dataset bbox) |
| POST | /v1/hazards/ingest |
Trigger hazard ingestion from a pre-staged server path (async job) — GeoTIFF or a CLIMADA Hazard HDF5 (issue #1174); NetCDF is not reachable through this route — see below |
| POST | /v1/hazards/upload |
Direct browser upload of a hazard GeoTIFF, CLIMADA Hazard HDF5 or NetCDF (netcdf_variable, one declared layout; issue #1670), then ingest (async job); single_event + event_date dates one observed event — see below |
| POST | /v1/hazards/footprints |
Store an event footprint uploaded as a GeoJSON outline with no intensity — a single-event dataset that reads outline_only: true, is drawn, and is refused by every compute route (issue #1670) — see below |
| GET | /v1/hazards |
List hazard datasets — omitting ?status= now excludes superseded rows by default (issue #873); pass ?status=superseded explicitly to see them. Each row carries event_axis/computable (issue #920) and a derived country (issue #1369) — ?country= filters to one ISO3 — see below |
| GET | /v1/hazards/ingest-jobs/{id} |
Fine-grained progress for one hazard ingest, VIEWER+ — status walks pending → running → chunking → writing → aggregating → committed → succeeded (terminal failed/cancelled), plus progress_pct, chunks_total/chunks_done, error_code (the same ADR-028 Code GET /v1/jobs/{id} reports, issue #1182), dataset_id, the timestamps, and paginated logs (?logs_offset=N; the response carries next_offset so a client pages forward without re-reading) |
| POST | /v1/hazards/ingest-jobs/{id}/cancel |
Cancel an in-flight hazard ingest, ANALYST+ — 200 {"cancelled": "<id>"} when the job is in a non-terminal state; 409 ALREADY_TERMINAL when it is already succeeded/failed/cancelled; 404 when unknown or another org's |
| GET | /v1/hazards/quota |
The caller's hazard-upload storage — {used_bytes, limit_bytes}, VIEWER+. Sums file_size_bytes over datasets whose owner_user_id is the caller, so seeded and script-ingested datasets never count against a person. Exposure uploads are metered on a separate counter that has no read route — see below |
| GET | /v1/hazards/{id}/used-by |
What uses one hazard dataset (issue #1634) — {results, analyses, analyses_total}; see "Dataset used-by and versions" below |
| GET | /v1/hazards/{id}/versions |
The other ingests sharing its source identity, newest first (issue #1634) — {items, reason}; see below |
| GET | /v1/hazards/{id}/download |
Download one hazard dataset as its stored processed COG (issue #1635), VIEWER+ — image/tiff, Content-Disposition: attachment; filename="<name>.tif", read from MinIO 1 MiB at a time and passed through the API (never a presigned URL, #1484). 404 with details.reason: "no stored raster" when cog_path is NULL (a point-only dataset, or one never converted) or the object is gone; another org's dataset is a plain 404 |
| POST | /v1/hazards/{id}/assign-centroids |
Trigger centroid assignment (async job) |
| POST | /v1/impact-functions/seed |
Seed default impact functions |
| GET | /v1/impact-functions |
List impact functions (filterable by ?haz_type= and ?exposure_type=; every row carries exposure_type and version) |
| POST | /v1/impact-functions |
Create a user-defined impact function (Phase 4+, #69); exposure_type (default buildings) and is_default (default false), a second default for one (haz_type, exposure_type) answering 409 CONFLICT (issue #1670) |
| DELETE | /v1/impact-functions/{id} |
Delete a user-defined function; builtin functions are protected (Phase 4+, #69) |
| POST | /v1/measures/seed |
Idempotently seed the cited reference measure catalogue (core/measure_seeder.py) for the caller's org — 201 {"seeded": N}; rows carry source: "builtin", cost: null and a per-entry license/citation/source_url (the two flood protections carry rp_protect: 100, issue #1566), and list under GET /v1/measures?source=builtin. A select-then-write keyed on (org, source="builtin", name) under a transaction-scoped advisory lock, so a second call refreshes the same rows. Runs only on this explicit call, never at startup; not demo data under ADR-077 (the ADR-023 impact-function pattern). Builtin rows are read-only: PATCH on one is 422, a user POST/PATCH with source: "builtin" is 422, and POST /v1/compute/cost-benefit naming a measure with no cost is 422 "measure … has no cost" — ANALYST+ (issue #1485) |
| PATCH | /v1/measures/{id} |
Edit a saved adaptation measure in place, keeping its id — any POST /v1/measures field, only the sent keys change, extra="forbid"; {"region": null} clears a nullable field, while a null name/haz_type/cost/cost_unit or a cost ≤ 0 is 422; id/org_id/created_at are not patchable; another org's id is 404 — ANALYST+ (issue #1483) |
| PUT | /v1/measures/{id}/image |
Upload or replace a measure's picture (multipart file) — the type is sniffed from the bytes (PNG, JPEG or WebP; SVG and anything else is 422 whatever the declared type), more than 5 MiB is 413; stored at measures/{org_id}/{measure_id}/image in MinIO and returned as image_url on the measure body; another org's id is 404 — ANALYST+ (issue #1484) |
| GET | /v1/measures/{id}/image |
Stream the picture with its sniffed content type (never a presigned MinIO URL); 404 when the measure has none — VIEWER+ (issue #1484) |
| DELETE | /v1/measures/{id}/image |
Remove the picture (204, a no-op when there is none); deleting the measure removes its picture too. Measures also carry sketch, a gallery key matching ^[a-z0-9-]{1,32}$ on POST and PATCH — ANALYST+ (issue #1484) |
| POST | /v1/adaptation-plans |
Save an adaptation plan (issue #1632, AD-8) — {name, portfolio_id \| exposure_dataset_ids, scope?, haz_type, scenario?, horizon_year?, measure_ids, custom_order?, discount_rate, value_growth_rate?, time_horizon_years}, the compute request's own names; measures in plan order, scope stored exactly as sent. Refused as a run is: no selector or two (422 "name one exposure selector"), a dataset outside the org, a clip of unknown units or of mixed level or country, half a cell, a duplicate or unknown measure, or one of another hazard than haz_type — each 422 with details.reason; a portfolio outside the org is 404. last_appraisal_job_id is not a request field (422). ANALYST+ — see "AdaptationPlan" |
| GET | /v1/adaptation-plans |
The org's plans, most recently changed first (?haz_type=&limit=&offset=), {items, total} — each with appraisal_status, avoided_loss and avoided_loss_unit read through last_appraisal_job_id (AD-17, FT-14); ?haz_type= serves the run form's "Use a saved plan" (FT-7). Two statements for any page. VIEWER+ |
| GET | /v1/adaptation-plans/{id} |
One plan, the same body as a list item; another org's id is 404 E_NOT_FOUND. VIEWER+ |
| PATCH | /v1/adaptation-plans/{id} |
Change only the sent fields; scope, scenario, horizon_year and the selector fields may be null, any other explicit null is 422. Sending one selector replaces the other; measure_ids replaces the list in the order sent. The merged plan is refused exactly as on create. Changing any field but name clears last_appraisal_job_id. Another org's id is 404. ANALYST+ |
| DELETE | /v1/adaptation-plans/{id} |
Delete a plan, 204 — the row only; its appraisal jobs stay. Another org's id is 404. ANALYST+ |
| POST | /v1/compute/impact |
Trigger impact calculation (async job) — scenario_id alone replays a saved scenario (issue #1186); the exposure set is exposure_dataset_ids, exposure_dataset_id or portfolio_id (issue #1586, "Exposure selector on impact compute"); optional scope clip with one of admin_boundary_ids, admin_boundary_id or bbox (issue #1588, "Scope clip on impact compute"); optional exposure_growth_rate in [-0.5, 0.5] (issue #1593, "Exposure growth on impact compute"); optional measure_ids and adapted_from_result_id (issue #1594, "Adaptation measures on impact compute") — see below |
| GET | /v1/compute/hazard-options |
Each hazard type's state, reason, climate_driven, money, money_reason and cells for an exposure selector and optional clip, plus the clipped scope_total — a read, VIEWER+ (issue #1661, "Hazard options for an exposure selector") — see below |
| POST | /v1/compute/impact/matrix |
Submit a scenario x horizon matrix batch (async group, issue #380); takes the same exposure_dataset_ids / portfolio_id selector, scope clip (issue #1588), exposure_growth_rate (issue #1593) and measure_ids / adapted_from_result_id (issue #1594), and explicit cells with haz_type/region (issue #1587); each dispatched cell's job carries the cache_key of what it computes and completes with a stored result of the same inputs instead of computing (issue #1662, "Matrix cells reuse stored results") |
| GET | /v1/compute/impact/matrix |
List the org's compute groups, newest first (?limit=&offset=) — matrices and engine comparisons since issue #1186, every row tagged with kind and carrying inputs_resolved (issue #1270) and the submitter's created_by_user_id and created_by_email (null once the user is gone, issue #1662) — see below |
| GET | /v1/compute/impact/matrix/{batch_id} |
Batch status + per-cell result refs (partial-failure semantics) — scenario matrices only; a comparison id is 404 here. The batch carries created_by_user_id and created_by_email; each cell carries hazard_dataset_id (the dataset it resolved to, null for a cell that resolved none) and cached (true when its completed job's result was written by another job, i.e. reused from a stored result of the same inputs; derived from impact_results.job_id, never stored) — issue #1662. params is the batch's stored request; a clipped run carries its clip as params.scope in the stored form its cell jobs get (ExposureSelector.scope_record: {"admin_boundary_ids": [...]} sorted, or {"bbox": [w, s, e, n]}), written at submit so a batch in flight or one whose every cell failed still says where it ran; an unclipped batch, and one written before #1806, has no scope key |
| DELETE | /v1/compute/impact/matrix/{batch_id} |
Delete a scenario-matrix group; the tracking row goes, its Jobs and results stay (issue #1270), ANALYST+ — a comparison id is 404 here |
| POST | /v1/compute/impact/matrix/{batch_id}/report |
Queue a template-driven PDF render for a scenario-matrix batch (async job, issue #415) — the batch-shaped sibling of POST /v1/results/{id}/report |
| GET | /v1/compute/impact/matrix/{batch_id}/report |
Fetch the stored PDF pack for a scenario-matrix batch |
| POST | /v1/compute/impact/compare |
Run one scenario through N engines (async group); takes exposure_dataset_ids or portfolio_id, a scope clip (issue #1588), exposure_growth_rate (issue #1593), measure_ids / adapted_from_result_id (issue #1594) and a scenario + horizon_year cell like /impact — see below |
| POST | /v1/compute/score-delta |
Appraise measures on an indicator hazard (heatwave days, wildfire FWI) as a score delta, never a BCR (AD-16, issue #1627): takes measure_ids, portfolio_id or exposure_dataset_ids, an optional scope clip and a scenario + horizon_year cell → 200 with per-measure baseline_band / adapted_band (integer bands) and baseline_mean / adapted_mean (the unrounded means), both band distributions and both shares in high or severe; 202 {job_id, job_type: "score_delta"} for a dataset set above rollup_max_portfolio_members rows — see "The score delta of a measure on an indicator hazard" — ANALYST+ |
| GET | /v1/compute/impact/compare/{id} |
Side-by-side per-engine metrics + divergence %, plus queued_ahead — how many queued messages a worker takes before the group's oldest pending cell (issue #1640) |
| DELETE | /v1/compute/impact/compare/{id} |
Delete an engine-comparison group; kind-scoped like its matrix sibling (issue #1270), ANALYST+. Its pending cells end cancelled and their tasks are revoked; a running cell finishes (issue #1639) |
| POST | /v1/live-events |
Open a live-event watch (haz_type, name, source, exactly one of portfolio_id/exposure_dataset_id, optional started_at/aoi/auto_compute/kind, and the watch's uniform insurance terms deductible (≥ 0) and cover (> 0), a negative value is a 422, issue #1611); auto_compute: true is a 422 no vulnerability curve for exposure type <type> when the hazard has no default curve for the selector's exposure type (the curve gate, issue #1615); an optional feed (the body of POST /v1/live-events/{id}/feeds) attaches the watch's feed in the same request, both written in one savepoint, so a refused feed (E_VALIDATION with details.problems, E_LIVE_FEED_KIND_UNKNOWN) answers its own code and leaves no watch, and the feed sets the watch's kind as on the feeds route (EV-20, issue #1807) — ANALYST+ (issues #1026, #1287) — see below |
| GET | /v1/live-events/replay-series |
The replay series this deployment holds → [{series_dir, name, frames}]: each directory holding a valid manifest.json up to two levels under an allowed series root (dataset_source_download_dir, data_mount_root), name the manifest's series, frames its observation count, ordered by series_dir, at most 200; every series_dir passes the replay feed's params check. A root that does not exist is skipped, so no data volume is 200 []; the scan runs off the event loop and is the only code that reads the data roots — the feed's own series_dir check stays lexical. Declared above GET /v1/live-events/{id} — ANALYST+, E_LIVE_MODE_DISABLED (422) while live mode is off (EV-20, issue #1807) |
| GET | /v1/live-events |
List the org's live events, newest-started first (?portfolio_id=&status=&limit=&offset=) — every row carries the watch's summary (snapshot_count, latest_valid_at, latest_event_loss with its latest_event_loss_unit and latest_insured_event_loss (issue #1611), latest_severity_level, feed, and events: [{event_key, first_valid_at, last_valid_at, snapshot_count}] ordered by first_valid_at, issue #1606) resolved in one statement (issue #1287; one more for the page, for the headline results' units, when a watch on it has a result, issue #1608); 200 with an empty page when live mode is off |
| GET | /v1/live-events/{id} |
Read one live event, same summary shape as the listing (including the watch's events list, each event_key with its first_valid_at/last_valid_at/snapshot_count, and feed.last_poll_summary: what the feed's last completed poll looked at and found — polled_at, observations, events_in_window, criterion, window_start, refused — null for a feed never polled, issue #1614); E_LIVE_EVENT_NOT_FOUND (404) when absent, cross-org, or live mode is off |
| PATCH | /v1/live-events/{id} |
Revise a running watch's name/auto_compute/aoi/kind/deductible/cover — nothing else, extra="forbid"; {"aoi": null} clears the area and {"cover": null} drops the cover; edited terms apply to computes dispatched after the edit (issue #1611); switching auto_compute on passes the same curve gate as the create (issue #1615) — ANALYST+ (issue #1287) — see below |
| GET | /v1/live-events/{id}/snapshots |
Replay the event's hazard-dataset snapshots, valid_at descending, filtered valid_at <= as_of (?as_of=) and to one event inside the watch (?event_key=, an unknown key is an empty page — issue #1606); each row carries live_feed_id (null for a pushed observation), event_key and cumulative. The listing holds the frames, one per overpass; ?cumulative=true lists only the event-to-date snapshots a wildfire feed mints on every poll (cell-wise maximum over the overpasses so far, issue #1607). Every snapshot_count and the feed's max_snapshots cap count frames only |
| GET | /v1/live-events/{id}/snapshots/{snapshot_id}/portfolios |
The organisation's portfolios with members inside one snapshot's footprint (issue #1687, SH-16): {snapshot_id, valid_at, items: [{portfolio_id, name, asset_count, value, value_unit}]}. asset_count counts members on a cell whose intensity is above zero, value is Σ weight × value over them in value_unit, both null when they mix units; items with a value by value descending, then the others by asset_count descending, none without a counted member. Counted at read time and cached in the object store at hazards/{snapshot_id}/portfolios/{org_id}/{fingerprint}.json until the organisation's portfolios or attachments change. A snapshot that is not the watch's answers 404 E_LIVE_EVENT_NOT_FOUND |
| POST | /v1/live-events/{id}/close |
Close the watch — one-way; a second call is E_LIVE_EVENT_CLOSED (422) — ANALYST+ |
| POST | /v1/live-events/{id}/compute |
Submit a deterministic-event impact job for one snapshot (async, 202 {job_id}); it runs on the watch's own deductible and cover (issue #1611), and optionally with measure_ids and adapted_from_result_id (issue #1643) — ANALYST+ (issue #1027) — see below |
| GET | /v1/live-events/{id}/results |
List the event's computed jobs, oldest snapshot first — every row is a completed job and carries cumulative from the snapshot it scored (issue #1608) and insured_event_loss beside event_loss (null for a run without terms, issue #1611) |
| GET | /v1/live-events/{id}/results/{job_id} |
Full event-lane result: event_loss, insured_event_loss (the same footprint after the watch's deductible and cover, in unit, at most event_loss, null for a run without terms, issue #1611), unit (the one value_unit every scored row shares, null when they mix, K27 — the same reading over the same rows as the watch's latest_event_loss_unit when this result is its headline), impact_per_event, valid_at, source, grade/grade_note, exposure selector, exposed_count/exposed_value (issue #1293), ranked assets (?limit=&offset=, each with a display name + full dataset_name and intensity, the footprint's intensity at the asset's cell, null for a result scored before it was stored, issue #1612), admin_units (event_loss and asset_count per GADM unit over every asset, null when the result has no stored sums, issue #1612), avoided_event_loss (an adapted result's source event_loss minus its own, in unit, computed on read, null without provenance.adapted_from_result_id, issue #1643), provenance — no ead/aai anywhere (phase guardrail 6) |
| POST | /v1/live-events/{id}/feeds |
Attach a polling schedule (kind, params, cadence_seconds, optional max_snapshots) — sets the watch's kind: replay for a replay feed, else near_real_time — ANALYST+ (issue #1295) — see below |
| GET | /v1/live-events/{id}/feeds |
The watch's feeds, ordered by kind, unpaged (one per provider); each row carries the schedule, its state (enabled/paused_reason/cursor/last_polled_at/last_error/last_poll_summary) and the derived snapshot_count |
| PATCH | /v1/live-events/{id}/feeds/{feed_id} |
Revise enabled/cadence_seconds/params/max_snapshots — enabled=true clears paused_reason, whatever wrote it; kind is not patchable — ANALYST+ (issue #1295) |
| DELETE | /v1/live-events/{id}/feeds/{feed_id} |
Remove the schedule — 204, and the snapshots it produced are kept (hazard_datasets.live_feed_id is ON DELETE SET NULL) — ANALYST+ |
| POST | /v1/live-events/{id}/feeds/{feed_id}/poll |
Poll now, out of band with the cadence → 202 {feed_id, queued}; rate-limited by live_feed_poll_rate_limit (default 10/minute, then 429), a closed watch is E_LIVE_EVENT_CLOSED (422), a disabled feed is E_LIVE_FEED_PAUSED (409) — ANALYST+ (issue #1295) — see below |
| POST | /v1/live-events/{id}/observations |
Push one observed raster onto an open watch — multipart file (GeoTIFF only), valid_at (required, timezone-aware ISO-8601, never defaulted to now), intensity_unit (required, one a seeded curve for the watch's peril consumes), optional label/source → 202 {job_id, ingest_job_id}; the mode-3 ingest POST /v1/hazards/ingest dispatches, jobs.origin = user, its snapshot carries live_feed_id: null and auto-computes on an auto_compute watch; the upload route's cap and quota apply (413) — ANALYST+ (issue #1294) — see below |
| GET | /v1/live-events/{id}/results/{job_id}/delta |
What changed between two of this watch's results (?against=<job_id>&limit=&offset=, required against) → {job_id, against_job_id, valid_at, against_valid_at, event_loss_delta, exposed_count_delta, exposed_value_delta, assets[{exposure_id, name, lat, lon, loss_before, loss_after, delta}], total_assets}, assets ranked by |delta| descending; E_RESULT_VECTOR_ABSENT (409) when either side stored no per-asset vector, E_RESULT_STALE (409) on exposure drift, E_LIVE_EVENT_NOT_FOUND (404) when against is another watch's (issue #1293) — see below |
| GET | /v1/jobs |
List the org's jobs, newest first (?status=&job_type=&include_automated=&limit=&offset=; include_automated defaults to false and excludes feed-driven rows — jobs.origin = 'feed' — from the page and from total, issue #1287) — {items, total}, VIEWER+; every row carries a server-derived non-null title so a client renders "Data ingest — Attica flood" without a second round trip (issue #1184); status repeats (?status=pending&status=running) and the envelope carries impact_workers (issue #1662) — see below |
| GET | /v1/jobs/{id} |
Poll job status — includes error_code (a §11 ADR-028 Code, null unless the job failed) alongside the free-text error_message (issue #1182), and result: the stored answer of a completed score_delta job (jobs.result_payload, issue #1627), null on every other job; cancellable and ingest_job_id as on the list (issue #1654); portfolio_id and the run inputs (exposure_dataset_ids, hazard_dataset_id, haz_type, scenario_label, horizon_year, country_iso3) as on the list (issue #1901) |
| POST | /v1/jobs/{id}/cancel |
Cancel a queued impact job, ANALYST+ (issue #1654, SH-16, EX-20's Stop is one call per cell that has not started) — a pending impact_calc job ends cancelled in the request's commit, then its Celery task is revoked (never terminate=True; the worker also refuses a job that reads cancelled, #1639) and a job.cancelled audit entry is written; 200 with the job. 409 CONFLICT naming the status for any job that is not pending (a running one finishes and keeps its result); 422 for a job that is not impact_calc (a hazard ingest is cancelled through POST /v1/hazards/ingest-jobs/{ingest_job_id}/cancel); 404 for another organization's job |
| GET | /v1/users/me/notifications |
The caller's bell feed, newest first (?unread=&limit=&offset=) — {items, total, unread_count}; dismissed rows never appear |
| POST | /v1/users/me/notifications/{id}/read |
Mark one notification read; 404 when it is not the caller's |
| POST | /v1/users/me/notifications/{id}/dismiss |
Remove one notification from the feed (implies read) |
| POST | /v1/users/me/notifications/read-all |
Clear the whole inbox in one statement — {count}, idempotent (issue #1187) — see below |
| GET | /v1/orgs/me/branding |
Branding for the caller's own org, any org member — 200 with the platform defaults when the org has configured none (issue #1190), never 404 — see below |
| GET | /v1/orgs/{org_id}/branding |
Branding for a named org, any org member — 404 when that org has configured none; another org's id is 404 too |
| POST | /v1/orgs/{org_id}/branding/logo |
Upload the org's logo, org admin — an SVG or PNG at least 256 px wide, at most 1 MiB; sets branding.logo_url (issue #1689) — see below |
| GET | /v1/orgs/{org_id}/branding/logo |
The org's stored logo, any org member, under default-src 'none'; 404 when none is stored |
| POST | /v1/scenarios |
Save a scenario — inputs is an open dict and a create never fails on its shape; anything uninterpretable comes back as warnings[] (issue #1186), ANALYST+ |
| GET | /v1/scenarios |
List the org's scenarios (?limit=&offset=), {data, total}, VIEWER+ — every row carries inputs_resolved since issue #1270, so naming a run needs no per-row detail fetch; warnings[] stays on the detail route |
| GET | /v1/scenarios/{id} |
Read one scenario: the raw inputs unchanged plus inputs_resolved (names, computable, unresolved_ids) and warnings[] (issue #1186) — see below |
| PATCH | /v1/scenarios/{id} |
Rename / retag / edit description; inputs is deliberately not editable — save a new scenario instead |
| DELETE | /v1/scenarios/{id} |
Soft-delete a scenario; subsequent reads are 404 |
| GET | /v1/portfolios/summary |
The org's top books ranked by exposed value or EAD (?limit=&sort=), {items} — one call for Home's Portfolios card (issue #1181). Declared above /{portfolio_id}; below it, it 422s — see below |
| DELETE | /v1/portfolios/{id} |
Hard-delete a portfolio, ORG_ADMIN — {deleted}. 409 CONFLICT while an open watch names it (EV-12, issue #1609): the message names the watch ("Portfolio is watched by open watch '…'. Close the watch first."), details.live_events lists {id, name} of every open one, and nothing is deleted. A closed watch does not block: its portfolio_id is cleared in the same transaction. Unknown or another org's is 404 — see "The watch: selector, AOI, summary" |
| POST | /v1/portfolios/{id}/datasets |
Attach a whole exposure dataset — or a bbox slice, or an explicit row list — to a portfolio in one call, ANALYST+ (issue #1168) — see below |
| GET | /v1/portfolios/{id}/datasets |
List a portfolio's dataset attachments, {items, total}; row_count is what the selection resolves to, not the dataset's full count |
| DELETE | /v1/portfolios/{id}/datasets/{dataset_id} |
Detach one dataset — 204, no body; not attached (or another org's) is 404, ANALYST+ |
| GET | /v1/portfolios/{id}/members |
Page every member of a portfolio (?limit=&offset=), {items, total} — exposure_id, dataset_id, dataset_name, name, lat, lon, value, value_unit, weight. One read, no client-side join |
| POST | /v1/portfolios/{id}/members/remove |
Remove assets from a portfolio without deleting them (PF-12, issue #1688): {exposure_ids} (1–500) → {removed, member_count}. Each id is recorded on the attachment covering it — excluded_row_ids on an all/bbox selection, a shorter row_ids on a rows one; an id the portfolio does not hold is a 422 listing it, nothing written; ANALYST+ — see below |
| GET | /v1/portfolios/{id}/results |
The impact results this portfolio's own runs produced (?limit=), {items, total}, newest first — see below |
| GET | /v1/portfolios/{id}/thumbnail |
A card's small map (PF-7, issue #1666): {bbox, countries, weight, value_unit, cells: [{lon, lat, value}]}, at most 256 cells of a 16 × 16 grid summing to the exposed value, VIEWER+ — see "What a portfolio card names" |
| GET | /v1/portfolios/{id}/spread |
"Geographic spread" (PF-7, issue #1666), ?haz_type=&scenario=&horizon_year=: {admin_level, weight, value_unit, areas: [{boundary_id, name, exposed_value, member_count, band}]}, the five largest areas, VIEWER+ — see "What a portfolio card names" |
| POST | /v1/portfolios/{id}/exposures |
Deprecated (#1168), kept for one release — add one exposure point; also maintains the equivalent rows attachment |
| DELETE | /v1/portfolios/{id}/exposures/{exposure_id} |
Deprecated (#1168), kept until no released UI calls it — remove one point through /members/remove's path (issue #1688): a point an all/bbox attachment covers is excluded (200, where #1214 answered 409); 404 only when no attachment covers it and no junction row holds it |
| GET | /v1/portfolios/{id}/exposures |
Deprecated (#1168), kept for one release — unpaged {data, total} of (portfolio_id, exposure_id, weight), no coordinates and no name. Use /members |
| GET | /v1/results |
Stored results of given run inputs (EX-10, issue #1664): GET /v1/results?exposure_dataset_ids=<id>&cells=baseline@today (or portfolio_id), with the clip admin_boundary_ids/bbox=w,s,e,n, haz_type, cells (repeated, cell grammar), exposure_growth_rate, measure_ids, impact_function_id and engine, answers {items: [{result_id, haz_type, scenario_label, horizon_year, ead, value_unit, value_weighted_average_band, indicator_score, created_at}], total} — the newest stored result per hazard type and cell over exactly those inputs (an omitted curve or engine reads what a run would use), and per requested cell of an indicator hazard of the selector's countries a row with only indicator_score {band, label, metric_unit, hazard_dataset_id, band_index, rows_sampled} (band_index: the dataset band read, issue #1833). A read: no job, no cache, no total row (K26); the selector and clip are refused as /impact refuses them — VIEWER+ |
| GET | /v1/results/assets/{exposure_id} |
One asset's stored figure per hazard type and cell (EX-14, issue #1685): the query of GET /v1/results less haz_type → {exposure_id, items: [{haz_type, scenario_label, horizon_year, result_id, eai, band, value_unit, created_at}]}, eai and band the asset's entries of each stored result's eai_exp and band_exp — see "One asset's and one admin unit's figures per hazard" — VIEWER+ |
| GET | /v1/results/admin-units/{boundary_id} |
One admin unit's stored total per hazard type and cell (EX-14, issue #1685): the same query → {boundary_id, admin_level, items: [{haz_type, scenario_label, horizon_year, result_id, eai, points, value_unit, created_at}]}, each item the unit's stored result_admin_cells row — VIEWER+ |
| GET | /v1/results/{id} |
Get result summary (EAD, AAI, per-event, the shared vocabulary curve_basis/aal/pml_200, plus rows_computed/scope_total, value_unit and the #1595 figures exposed_value, band_distribution, share_high_or_severe, count_high_or_severe, design_return_period, value_weighted_average_band, impact_by_exposure_type — see below — and checksum, the SHA-256 of the stored arrays, null on a result written before issue #1636; see "Result checksum") |
| GET | /v1/results/{id}/admin-cells |
One result's stored admin units at one tier ranked by expected annual impact (SH-16, issue #1636): ?admin_level=0..2&limit= (default 10, capped at 50; another admin_level is 422) → {cells_status, items: [{admin_boundary_id, name, admin_level, eai, points}]}, eai descending; geometry=simplified adds each unit's polygon — see "Ranked admin units of a result" — VIEWER+ |
| GET | /v1/results/{id}/geojson |
Get result as GeoJSON FeatureCollection |
| GET | /v1/results/{id}/freq-curve |
The loss-exceedance curve {result_id, curve_basis, points} sorted by return period; ?rp=N reads one stored point, answered only at a return period the curve samples inside the hazard dataset's [RP_min, RP_max] (its return_periods) — any other rp, or any rp on a dataset without return periods (a live snapshot), is 422 E_VALIDATION with details.reason naming the range, details.range and details.return_periods (the sampled periods); never interpolated (issue #1596) |
| GET | /v1/tiles/results/{id}/{z}/{x}/{y}.mvt |
The result's stored aggregates or asset points as a one-layer Mapbox Vector Tile (?layer=admin\|h3\|assets&level=), layer picked by zoom when absent; 404 E_NOT_FOUND with details.cells_status until metadata.cells.status is built (issues #1472, #1473); ?minus=<base result id> adds eai_avoided to every feature for an adapted result (issue #1628), VIEWER+ — see "Result vector tiles" |
| GET | /v1/tiles/exposures/{dataset_id}/{z}/{x}/{y}.mvt |
An exposure dataset's assets as a one-layer Mapbox Vector Tile (?layer=points\|bins, or ?layer=admin&level=0-2 for the exposed value per GADM unit): query-time hexagon bins below z 11, points from z 11 when no layer is named; 404 for another org's or an unknown dataset (issue #1474), VIEWER+ — see "Exposure vector tiles" |
| GET | /v1/tiles/portfolios/{portfolio_id}/{z}/{x}/{y}.mvt |
A portfolio's members (not its datasets' other rows) as a one-layer Mapbox Vector Tile in the exposure tiles' shape (?layer=points\|bins, or ?layer=admin&level=0-2 per GADM unit; mixed value_unit members serve count only); 404 for another org's or an unknown portfolio (issues #1598, #1686), VIEWER+ — see "Portfolio vector tiles" |
| GET | /v1/tiles/scope-scores/{hazard_dataset_id}/{z}/{x}/{y}.mvt |
A scope's indicator score per GADM unit as a one-layer (admin) Mapbox Vector Tile: selector exposure_dataset_ids (repeated) or portfolio_id, clip admin_boundary_ids/bbox=w,s,e,n, cell scenario/horizon_year, level=1\|2 (absent: z 0-6 level 1, z >= 7 level 2); features boundary_id, iso3, name, admin_level, band, label, score, rows_sampled; 404 for a money hazard or another org's dataset (issue #1833), under the vector tile guard, VIEWER+ — see "Scope score vector tiles" |
| GET | /v1/results/{id}/per-exposure |
Page the individual exposure points behind a result (?limit=&offset=&sort=eai\|value), {items, total} — see below |
| POST | /v1/results/{id}/report |
Queue a template-driven PDF render (async job, ADR-039) |
| GET | /v1/results/{id}/report |
Export the result — ?format=json\|csv\|pdf; PDF serves the stored document |
| POST | /v1/report-documents |
Save a report document — {name, template, scope, widgets}, widgets in document order (issue #1487, ADR-085). 422 E_VALIDATION for an unknown widget kind (details.errors[].loc names the index), or a referenced id outside the caller's org or a 51st widget (details.widget_index). 422 E_REPORT_TEMPLATE_UNKNOWN for an unregistered template, as on the render routes. ANALYST+ — see "Report documents" |
| GET | /v1/report-documents |
The org's report documents, newest first (?limit=&offset=), {items, total}, VIEWER+ |
| GET | /v1/report-documents/templates |
Every registered report template, {items: [{key, name, subject: result\|batch\|document, availability: available\|wording_to_verify}], total} sorted by key, from core/reports/composer.TEMPLATE_INFO (issue #1488). Declared before /{id}. VIEWER+ |
| GET | /v1/report-documents/{id} |
One report document; another org's id is 404 E_NOT_FOUND, VIEWER+ |
| POST | /v1/report-documents/{id}/render |
Queue a PDF render through the document's template (issue #1489) — 202 {job_id, document_id, template}. 422 E_VALIDATION for an impact/orsa document without exactly one result/batch widget. ANALYST+ — see "Report documents" |
| GET | /v1/report-documents/{id}/pdf |
The document's last complete render, application/pdf; 404 E_REPORT_NOT_RENDERED before one exists. VIEWER+ |
| GET | /v1/report-documents/{id}/checklist |
The document's coverage checklist for its template (issue #1491), computed at read time — {template, items: [{id, label, status: ok\|partial\|missing, evidence, wording: internal\|to_verify\|verified, source}], ok, partial, missing}, no score. Another org's id is 404 E_NOT_FOUND. VIEWER+ — see "Report documents" |
| PATCH | /v1/report-documents/{id} |
Change only the sent fields of {name, template, scope, widgets} (issue #1488); an explicit null is 422. widgets replaces the whole list in the order sent (to reorder, send the new order) and is refused exactly as on create. Another org's id is 404 E_NOT_FOUND. ANALYST+ |
| DELETE | /v1/report-documents/{id} |
Delete a report document, 204 — the row only; rendered PDFs stay in the object store (ADR-029 leaves). Another org's id is 404 E_NOT_FOUND. ANALYST+ |
| GET | /v1/risk/lookup |
Multi-hazard score card for one coordinate (?lat&lon[&scenario&horizon®ion&hazards&dataset_id]) — see below |
| GET | /v1/risk/admin-units/{boundary_id}/impact |
One admin unit's expected annual impact per hazard, from saved runs (LK-15, issue #1684) → {boundary_id, admin_level, hazards: [{haz_type, result_id, scenario, points, eai, value_unit, exposure_dataset_id, exposure_name, exposure_type, computed_at}], buildings_in_data} — see below — VIEWER+ |
| GET | /v1/risk/assets/{id} |
Retired (#1175, ADR-080) — always 410 Gone. The assets identity plane it scored has had no writer since the demo seeder was deleted (ADR-077) and prod measured portfolio_assets = 0 on 2026-09-07. The envelope carries E_ASSET_PLANE_RETIRED, names ADR-080, and points at GET /v1/risk/portfolios/{id} (a book) or GET /v1/risk/lookup (one coordinate). Authentication is still required and no database is read; a 404 was deliberately rejected because it invites a retry with another id. The physical table drop is Phase 15 |
Dataset used-by and versions (issue #1634, SH-16; core/dataset_usage.py
over db/repositories/dataset_usage_repository.py). They back the Data
drawer's "Used by" and "Versions" blocks. Both are VIEWER+, and another
organization's dataset is 404.
- Used by.
resultscounts theimpact_resultswhosehazard_dataset_id(orexposure_dataset_id) is the dataset.analyseslists the saved analyses as{id, name, created_at}, newest first, at most 10, andanalyses_totalcounts all of them. A saved analysis is a livescenariosrow (deleted_at IS NULL) whoseresult_idnames such a result. The exposure read addsportfolios: [{id, name}], every portfolio attaching the dataset on theportfolio_datasetsplane, listed once whatever its selection. Each read is a fixed 2 (hazard) or 3 (exposure) statements, however many results reference the dataset. - Versions. The organization's other datasets that share the ADR-048
source identity. That is the same non-null
providerand the samedata_type, plushaz_type,regionandscenariofor a hazard, orexposure_typefor an exposure. Two nulls count as equal on every column exceptprovider. Items are{id, name, upstream_version, status, created_at, checksum}, newest first.checksumis the storedsha256(hazard) or the row-levelexposures.dataset_sha256(exposure), and it isnullwhen no hash was stored. The list groups ingests by identity and claims no succession, because nothing records which ingest replaced which. A dataset with noprovideranswersitems: []withreason: "no source identity recorded". Otherwisereasonisnull.
Upload formats — exposures and hazards, at a glance (issue #1176)¶
The two upload-shaped routes from the table above, side by side. This is the index; the full field lists, refusal tables and rationale live in the per-route sections below — not duplicated here.
| Route | Accepted formats |
|---|---|
POST /v1/exposures |
CSV, CLIMADA entity .xlsx, CLIMADA Exposures .h5/.hdf5 (issue #1172) — see below |
POST /v1/hazards/upload |
GeoTIFF (.tif/.tiff), CLIMADA Hazard .h5/.hdf5 (issue #1174), NetCDF .nc in one declared layout (issue #1670) — see below |
POST /v1/hazards/footprints |
GeoJSON outline (one Feature or a FeatureCollection of polygons, EPSG:4326) (issue #1670) — see below |
POST /v1/admin-boundaries/uploads |
GeoJSON FeatureCollection of polygons with name and iso3 (issue #1670) — see below |
POST /v1/hazards/ingest |
GeoTIFF, CLIMADA Hazard .h5/.hdf5 (issue #1174) — from a pre-staged server path rather than a multipart body; format is chosen by suffix (.h5/.hdf5 resolves the CLIMADA adapter, anything else the GeoTIFF one) since IngestRequest carries no source_format field — see Hazard Ingest Input Modes below. NetCDF is not reachable here — see the operator-only note below |
Both HDF5 branches share one reader, core/ingest/climada_hdf5.py, and one
security property stated once here rather than twice per route: h5py
only, never pandas.read_hdf / HDFStore / pickle. CLIMADA persists its
stores through pandas.HDFStore; a pandas fixed-format object block is a
raw pickle stream, so reading one back with a pandas HDF reader deserialises
whatever a downloaded artefact contains — a remote-code-execution primitive
gated only on the file server staying honest. Every column readable only by
unpickling is refused (InvalidSourceError) rather than decoded; a Point
geometry column stored as WKB is read with shapely.from_wkb instead — a
binary parser, not an object deserialiser.
The two accepted CLIMADA Hazard HDF5 shapes, exactly:
- (a) a return-period set — every event's
frequencystrictly positive and strictly monotonic across the event axis;return_periods = 1/frequency, kept as the unrounded floats the division produces, bands = events. - (b) exactly one event — the upload endpoint sets
params["event"]carrying{"label", "valid_at"}, withvalid_atdecoded from CLIMADA's proleptic Gregorian ordinaldate[0](the{"label", "valid_at"}docstring atworker/tasks.py:1862documents this key, not the chord declaration). The chord's ownsingle_event_declaration— whathazard_events.event_namederives from — carries the label only (worker/tasks.py:2109);valid_attravels separately onparams["event"]and is parsed there (worker/tasks.py:2158), never riding the declaration.
Anything else is refused, naming both accepted shapes and pointing at where
the rejected shape belongs. A dated multi-event catalogue — every event
sharing one frequency, which is what a stochastic set looks like — fails the
strict-monotonicity test exactly like a genuinely stochastic event set, and
both are the Phase 15 event-set track, not this path
(core/ingest/climada_hazard.py::_ACCEPTED_SHAPES). Full refusal tables,
with HTTP status and machine code, are under the per-route sections below.
Two independent ceilings, one per axis, env-overridable:
| Guard | Default | Env override | Bounds |
|---|---|---|---|
hazard_h5_max_events |
100 | HAZARD_H5_MAX_EVENTS |
n_events |
hazard_ingest_max_grid_cells |
1,000,000,000 | HAZARD_INGEST_MAX_GRID_CELLS |
n_centroids — the same quantity the GeoTIFF/NetCDF paths bound by grid cells |
hazard_h5_max_densified_bytes |
68,719,476,736 (64 GiB) | HAZARD_H5_MAX_DENSIFIED_BYTES |
n_events × n_centroids × 4 — the intensity matrix's densified footprint (#1239) |
The first two are independent, one per axis (#1225), so neither bounds the
n_events × n_centroids product on its own — 100 events over a 1e9-cell grid
is 1e11 nonzeros, ~400 GB densified, and both per-axis checks pass it.
hazard_h5_max_densified_bytes is the third ceiling that does bound it
(#1239), applied in the worker where the file is readable so
POST /v1/hazards/upload and POST /v1/hazards/ingest admit the same
products — the ingest route charges no quota, so it had no other backstop
(rationale under Hazard Upload — CLIMADA Hazard HDF5 below).
Two ingest paths are operator-only, reachable only from a script, never
from any HTTP route: the GeoTIFF→exposure raster path (ADR-055) and hazard
NetCDF ingest (ADR-051). NetCDF is the stricter case — it is not merely
undocumented on POST /v1/exposures / POST /v1/hazards/upload, it cannot
be reached even through POST /v1/hazards/ingest's pre-staged path: the
driving pack scripts call the Celery task ingest_hazard.delay(...) directly
with source_format="netcdf", bypassing every HTTP endpoint entirely. See
docs/DATA_SOURCES.html §9 for the driving scripts.
Cross-reference: docs/plan/phase-14-solidify.md D10 / W1.4c.
Upload size limits — two quota counters, one read route (issues #1170, #1174)¶
The ceilings above bound what a file may contain. These bound what a person may store, and the two upload routes meter it on separate counters that never see each other's bytes:
POST /v1/hazards/upload |
POST /v1/exposures |
|
|---|---|---|
| Per-file cap | hazard_upload_max_file_bytes, 2 GiB |
exposure_upload_max_file_bytes, 50 MiB (issue #1170 — it replaced a literal) |
| Per-user quota | hazard_upload_quota_bytes, 5 GiB |
exposure_upload_quota_bytes, 5 GiB |
| Summed from | hazard_datasets.file_size_bytes where owner_user_id is the caller |
exposure_datasets.file_size_bytes where owner_user_id is the caller (columns added by migration 0090) |
| Read route | GET /v1/hazards/quota |
none |
All four settings take their env var from the field name uppercased
(HAZARD_UPLOAD_QUOTA_BYTES, EXPOSURE_UPLOAD_MAX_FILE_BYTES, …) — Settings
declares no env_prefix, so there is none to add.
- Enforcement is incremental, not post-hoc. Each handler reads the
uploader's existing total once, before a byte is staged, then streams the body
to the staging path in
upload_chunk_bytes(8 MiB) chunks and checks both bounds before each chunk reaches the disk — so an oversized upload is refused part way through the body and the partial file is unlinked, rather than the whole file being written and then judged. One implementation for both routes:core/upload_streaming.py::stream_upload_to_pathraisesUploadTooLargeError/UploadQuotaExceededError, andapi/uploads.py::upload_limit_faultstranslates them into413 FILE_TOO_LARGE/413 QUOTA_EXCEEDED. - Only uploads are charged.
owner_user_idis NULL for anything an ingest, ascripts/packs/run, or a pre-migration upload wrote, which is exactly what keeps seeded catalog data out of a person's quota. POST /v1/hazards/ingestcharges nothing and reads nothing, and cannot: it stages a path rather than a body, so there are no bytes to charge and its params carryorg_idwith noowner_user_idto charge them to. The absolute ceilings above, not the quota, are what bound that route — see its own section below.- A CLIMADA
.h5hazard upload is admitted on one number and billed on another (issue #1228, superseding W1.4b's original plan). The admission bound is the densified footprintn_events × n_centroids × 4, computed from the store's metadata after staging and refused against the quota there — a small compressed file that would explode in memory does not get in. The ledger isfile_size_bytes: the uploaded source's byte count, stamped once at row creation, never corrected to the committed COG afterwards. That is the figure the quota later sums, the oneGET /v1/hazardsreports, and the one the upload's own202body echoes. For a CLIMADA store the two differ by orders of magnitude, which is why the handler logs them side by side (size=andadmission_bound=). - The exposure counter has no read route. A client can discover its hazard
headroom before uploading and cannot do the same for exposures; the only signal
there is the
413. Deliberate for now — the symmetricGET /v1/exposures/quotadoes not exist, so do not document or call one.
/v1/exposure-datasets — the dataset-first exposure resource (issue #1166)¶
exposure_datasets is the primary exposure resource: a picker searches
datasets, never pages points. The point list exists — one production Greek
GHSL dataset holds ~216 000 rows, so paging it was never a substitute for a
catalog. Three routes, all VIEWER+, all org-scoped:
| Route | Returns |
|---|---|
GET /v1/exposure-datasets |
{items, total, default_dataset_id}, newest first. Each item: dataset_id, name, exposure_type, value_unit, row_count, bbox, country, provider, created_at, upstream_version and checksum (the stored source SHA-256; both null when not recorded, issue #1912), plus the shared attribution quartet source/license/citation/source_url (rendered by api/attribution.py, so it matches the hazard and impact-function catalogs field for field). Filters: search (case-insensitive ILIKE on name, wildcards escaped), country (ISO3), exposure_type, include_superseded (default false). limit 1–200 (default 50), offset ≥ 0; total is the unpaged count |
GET /v1/exposure-datasets/{id} |
The same fields plus value_total (the summed value, in value_unit) and dropped_columns (issue #1633: the sorted source columns an upload did not keep; [] when none, null when never recorded). A dataset owned by another organization answers 404, not 403 — it is indistinguishable from an absent one, and a 403 would confirm the id exists |
GET /v1/exposure-datasets/{id}/rows |
{items, total} of {exposure_id, name, lat, lon, value, value_unit, properties}. bbox is minLon,minLat,maxLon,maxLat in EPSG:4326 through the shared parse_bbox helper (a malformed value is a 400 E_VALIDATION, as on every other dataset-fetch route); sort is value (descending) or name (ascending) and anything else is a 422; limit 1–500 (default 100) |
default_dataset_id — a country's default exposure layer (issue #1660,
EX-18). With ?country=<ISO3> the envelope carries, beside total, the id
core/exposure_service.py::default_exposure_dataset picks over every
non-superseded dataset of the organization whose derived country is that ISO3: a
money unit first (core/value_units.py::is_money_unit), then exposure type
buildings, then the newest created_at. It is an envelope key, not a per-row
flag, because the default may sit on another page or outside the search or
exposure_type filter, so neither those nor limit/offset move it. It is
null without country and when no dataset of that country qualifies. The UI
ranks nothing; every surface that needs a country's default reads this key.
An upload carries its extent (issue #1660). POST /v1/exposures (CSV, XLSX
and HDF5) writes the dataset's bbox as the envelope of its stored points,
[min_lon, min_lat, max_lon, max_lat], in the upload's own transaction
(core/exposure_bbox.py::record_exposure_bbox), so an upload derives a
country and can be a default. Uploads stored before that change are filled by
scripts/backfill_exposure_bbox.py --apply, a thin shim over
fill_missing_exposure_bbox, which binds each organization's
app.current_org_id in turn, fills only a missing bbox of a dataset that
holds points, logs seen=<n> filled=<k>, writes nothing without --apply, and
exits 2 when it saw no row unless --allow-empty is passed. A dataset with no
bbox is in no country's list and is never a default.
The list excludes superseded datasets by default (issue #1205), by the
same predicate GET /v1/catalog/coverage counts with — the two are one click
apart, so they must agree. The rule, and why it is one rule rather than two, is
stated once under GET /v1/catalog/coverage below; ?include_superseded=true
is the only way to see retired rows here, and it widens total alongside the
page.
POST /v1/exposures — the upload request¶
ANALYST+, multipart. The file is a CSV (lat, lon, value, plus any
extra columns, which are kept in properties), a CLIMADA entity .xlsx
(read from its assets sheet, latitude/longitude aliased per ADR-029), or
a CLIMADA Exposures .h5/.hdf5 (issue #1172). GeoTIFF exposures are still
unsupported.
The upload runs as an ingest job (issue #1667, DT-7, PF-12). The route
stages the file in the object store (uploads/exposures/{org_id}/…), enqueues
a data_ingest job whose params carry the upload's kind
(exposure_layer, or portfolio_import when it names a portfolio) and the
uploader's user_id, and answers 202 {job_id, kind, skipped_existing:
false, message} — before #1667 it parsed in the request and answered 201
with the rows. The worker (ingest_exposure_upload) parses the file, writes
the catalogue row, its points and its bbox in one transaction, and ends the
job: GET /v1/jobs/{job_id} then serves result: {dataset_id, count,
dropped_columns, portfolio_id, supersedes_dataset_id}. A file fault (missing
column, unknown unit, a CSV named .h5) fails the job with its error_code
rather than the request. While in flight the job row serves step: {index,
count, name} (index from 1) — "Uploading the file", "Checking format, units
and extent", "Writing to storage", "Registering in the catalogue" for a layer;
"Uploading the file", "Mapping columns and units", "Writing the assets" for a
portfolio import — and a pending upload is cancellable through
POST /v1/jobs/{job_id}/cancel, which ends it cancelled, deletes the staged
object and writes no row. A running upload is never interrupted (409).
The suffix and Content-Type are a request pre-filter, not the parser
selector. The HDF5 branch is taken on the file's magic bytes via
core/ingest/climada_hdf5.is_hdf5, so a CLIMADA store uploaded as
points.csv is read as HDF5, and a CSV uploaded as points.h5 is refused as
malformed rather than parsed as a CSV under a name that says otherwise. HDF5
stores are decoded by climada_hdf5.read_exposures, which never unpickles: a
column readable only by deserialising a pandas object block is refused. Its
region_id and impf_* columns land in properties uninterpreted — per-asset
impact-function binding is Phase 15, not this route.
Dropped columns (issue #1633). Every other column of an HDF5 store
(deductible, cover, category_id, …) is not read, and the completed job's
result names them, sorted, in dropped_columns. The list is stored on the dataset
(exposure_datasets.dropped_columns, migration 0115) and served by
GET /v1/exposure-datasets/{id}. The columns never reach properties:
insurance terms stay uniform per run. A CSV/XLSX keeps every extra column in
properties and answers dropped_columns: []. A dataset written before 0115,
or by an ingest, reads dropped_columns: null (not recorded).
| Form field | Default | Notes |
|---|---|---|
name |
(required) | Dataset name. Re-using one with no supersedes_dataset_id is a name-idempotent no-op at submission: 200 with job_id: null and skipped_existing: true, nothing staged or enqueued (issues #550, #1667) |
country |
null |
ISO3 the points lie in (issue #1667). When the points' extent derives another country (the catalogue's derived country, #1166) the job fails E_VALIDATION "the points lie in |
reference_year |
null |
1900 to the current year, else 422 (issue #1667). Stamped into every row's properties.reference_year, which the growth resolver reads as {year, source: "properties.reference_year"} |
supersedes_dataset_id |
null |
The exposure dataset of this organisation the upload versions, else 422 (issue #1667). The new dataset may keep the name; the predecessor reads status: superseded once the job completes, and GET /v1/exposure-datasets/{id}/versions lists the lineage although an upload stamps no provider |
portfolio_name / portfolio_owner |
null |
A portfolio import (issue #1667, PF-12): the job creates the portfolio with that owner and holds the new dataset with selection all. A name the organisation already uses is 409 at submission |
portfolio_id |
null |
A new version of that portfolio's file: the job holds the new dataset in place of the one its previous upload created, which reads superseded; results computed before keep their rows |
source / license / citation / source_url |
null |
ADR-013 attribution, stamped on every row |
exposure_type |
buildings |
buildings / population / infrastructure (issue #584) |
value_unit |
USD |
e.g. USD, m3 (issue #584) |
crs |
EPSG:4326 |
The only accepted value (issue #1172). Nothing on this path reprojects, and a CLIMADA artefact states its CRS only in a pickled attribute the reader will not decode — so the caller declares the frame and a mismatch is refused, never reinterpreted |
column_map |
null |
CSV/XLSX only (issue #1497): a JSON object {"lat": "<header>", "lon": "<header>", "value": "<header>"} (any subset), usually confirmed from POST /v1/exposures/column-suggestions. Applied before the required-column check; unmapped headers keep the latitude/longitude aliasing and land in properties. A header displaced by the map only through an alias (Latitude beside a mapped lat) keeps its lower-cased name in properties. An unknown target key, a header the file lacks, one header mapped twice, a displaced header literally named lat/lon/value, or a map on an HDF5 store is 422 E_VALIDATION. Omitted, nothing changes |
Column-mapping suggestions (issue #1497). POST /v1/exposures/column-suggestions
takes the same multipart file and answers 200 with
{headers, suggestions: {lat, lon, value}, properties, sampled_rows}. Each
suggestion is {suggested, confidence: high|medium|low, reason, alternatives},
or suggested: null when no header qualifies — never a guess. The heuristic
(core/column_mapping.py) is a synonym table on normalised headers
(latitude/lat_dd/y, longitude/lng/long/x,
tiv/total_insured_value/sum_insured/replacement_value/exposure/amount)
plus checks on the sampled values: latitude in [-90, 90], longitude in
[-180, 180], value numeric and ≥ 0. A header whose samples fail is not
suggested. Suggestions never apply themselves: only a column_map posted with
the upload changes parsing.
Refusals split by fault class:
| Fault | Status | Code |
|---|---|---|
Suffix and Content-Type both outside the accept-list |
400 | INVALID_FILE_TYPE |
Not HDF5 despite an .h5/.hdf5 name; unreadable store layout; non-Point geometry; missing value; a column readable only by unpickling |
400 | INVALID_FILE_TYPE, carrying the reader's own reason |
Unknown exposure_type, blank value_unit, crs other than EPSG:4326, unparseable CSV/XLSX |
422 | E_VALIDATION |
Body over EXPOSURE_UPLOAD_MAX_FILE_BYTES, or over the uploader's quota |
413 | FILE_TOO_LARGE / QUOTA_EXCEEDED (issue #1170) |
The SDK mirrors all of it: client.exposures.upload(path, ..., exposure_type=,
value_unit=, crs=) on both ClimateLama and AsyncClimateLama (SDK 0.2.0).
country is derived, not stored — no column, no migration. It is the
ISO3 of the GADM admin-0 boundary in admin_boundaries overlapping the
dataset's ADR-048 bbox most (ties broken by iso3, so the
answer is deterministic). Largest-overlap rather than contains-the-centroid
because a national bbox's centroid frequently is not in the country at all —
Greece's (19.0, 34.5, 28.5, 42.0) centres on open water in the Aegean. It is
null — never an error — when the derivation has nothing to work with: no
bbox (an upload from before issue #1660 not yet filled), a malformed one, no admin-0 boundary
loaded, or no overlap. null means not derivable, not in no country. It
is deliberately not inferred from the dataset's name or source: ADR-048
rejected heuristic identity backfills, and a substring match on free text is
exactly that.
The single definition lives in
db/repositories/admin_boundary_repository.py::largest_overlap_admin0_iso3, so
this and any future caller ask the question one way. Which boundaries are
candidates comes from ST_Intersects on the full stored polygon — an exact
filter; only the ranking among candidates uses ST_Envelope of that polygon
as a proxy. That matters at GADM's real resolution: the Greece admin-0 feature
is 29 909 vertices, and ST_Area(ST_Intersection(geom, envelope)) over it
measured 27.0 ms against 2.2 ms for the envelope proxy — and this
subquery runs once per catalog row, so the exact form cost ~1.4 s on a 50-row
page. The proxy loses nothing real: the thing being ranked is already a
bounding box.
Per-row name is properties.name verbatim when the row carries a
non-blank one, and otherwise "<dataset name> #<row index>", where the index
is the row's 0-based position in the dataset's stable order (created_at
then id ascending). The index is computed over the whole dataset — before
the bbox filter and independently of sort — so a row keeps one label
through any page or viewport. This closes the class of issue #1156: a gridded
ingest stamps the dataset's own name onto every point it writes
(exposures.name is NOT NULL and a LitPop/GHSL cell has no name of its own),
so an unfixed listing repeats one string on every line. The ranked-asset
surfaces solve the same problem with a coordinate-based label instead
(core/display_names.py, described under the live-event results below); this
resource is positional because a picker's rows need a stable ordinal, not a
location.
What the ordinal is, and what it is not. It is deterministic, not
meaningful. created_at is a server_default of now(), and PostgreSQL's
now() is the transaction timestamp — so every row of a bulk ingest, which is
one transaction, carries the same created_at and the order falls through
to the random-UUID primary key. #0 is an arbitrary point of the dataset, not
its first source-file row, and re-ingesting the same file relabels every point.
What the ordinal does guarantee is what a picker needs: within one dataset the
labels are distinct, cover #0..#n-1 exactly once, and do not move between
reads, pages, viewports or sort orders. Two things do move them — deleting a
row renumbers everything after it (DELETE /v1/exposures/{id} is a live
route), and so does adding rows. Both properties are pinned by tests
(test_row_index_under_a_production_bulk_insert,
test_deleting_a_row_renumbers_the_rows_after_it). Giving the ordinal
source-file meaning would need a stored per-dataset ingest sequence — a schema
change, out of scope for #1166.
Numbering before the filter costs a window over the whole dataset per request,
and the idx_exposures_geometry GiST index cannot serve a bbox that sits
above the window: measured at 402 ms cold / ~275 ms warm on a synthetic
216 000-row dataset (a production Greek GHSL upload's size) with a bbox and
limit=100. A covering (dataset_id, created_at, id) index would make the
sort an index scan, but that is also a migration.
Legacy aliases. GET /v1/exposures (per-point) and
GET /v1/exposures/catalog (datasets derived by grouping the point table) are
kept and unchanged — not deprecated, and with no removal date. /v1/exposures
gained an optional ?dataset_id= filter; omitting it returns exactly what it
always returned.
Portfolio membership — attachments and /members (issue #1168)¶
A portfolio holds datasets, not points. portfolio_datasets (migration
0091) carries one row per attached exposure dataset plus a selection naming
which of its rows the portfolio holds, so a 216 000-row GHSL upload is one
attachment rather than 216 000 junction rows.
| Route | Role | Returns |
|---|---|---|
POST /v1/portfolios/{id}/datasets |
ANALYST+ | 201 with the created attachment: {dataset_id, dataset_name, exposure_type, row_count, selection, weight, created_at} |
GET /v1/portfolios/{id}/datasets |
VIEWER+ | {items, total} of that same shape. Not paged — total is len(items) |
DELETE /v1/portfolios/{id}/datasets/{dataset_id} |
ANALYST+ | 204, no body |
POST /v1/portfolios/{id}/members/remove |
ANALYST+ | {removed, member_count} for a body {exposure_ids} of 1–500 ids (issue #1688) — see "Removing assets" below |
GET /v1/portfolios/{id}/members?limit=&offset= |
VIEWER+ | {items, total} of {exposure_id, dataset_id, dataset_name, name, lat, lon, value, value_unit, weight}; limit 1–500 (default 100), total is the unpaged member count |
GET /v1/portfolios/{id}/aggregate?scenario=&time_horizon=&limit=&offset= |
VIEWER+ | {portfolio_id, scenario, time_horizon, total_ead, total_aai, currency, coverage_pct, by_exposure, by_exposure_total}. Totals span the whole book; by_exposure is a page of {exposure_id, name, weight, has_result, ead, aai} with limit 1–500 (default 100). 422 when the book resolves to no members (issue #1215) |
GET /v1/portfolios/{id}/results?limit= |
VIEWER+ | {items, total} of {result_id, created_at, hazard_dataset_name, impact_function_id, ead, aai, scenario, horizon_year}, newest first; limit 1–100 (default 20). A book with no runs is an empty page, not a 404 (issue #1179) |
GET /v1/portfolios/summary?limit=&sort= |
VIEWER+ | {items} of {portfolio_id, name, exposed_value, value_unit, ead, aai, member_count, latest_result_id, latest_result_at}, ranked descending; limit 1–50 (default 5), sort exposed_value (default) or ead. No total — a ranked cut has no page two (issue #1181) |
selection is one of three shapes, validated by DatasetSelection
(models/portfolio.py) — the same model the ORM column runs on every
assignment, so the API and the database agree by construction rather than by
two validators staying in step:
{"kind": "all"}— every row of the dataset.{"kind": "bbox", "bbox": [minLon, minLat, maxLon, maxLat]}— EPSG:4326 degrees, resolved withST_Intersectsagainstexposures.geometry. A malformed, out-of-range or inverted box (including a degenerate zero-area one) is a 422.{"kind": "rows", "row_ids": [...]}— an explicitexposures.idlist. Every id must belong to that dataset; any that does not makes the request a 422 listing the offending ids (an id in another org counts as outside the dataset — reporting it as a 404 would confirm the row exists).
An empty row_ids is a valid stored selection — "this portfolio holds
this dataset, currently none of its rows" — but not a valid request:
POST …/datasets with {"kind": "rows", "row_ids": []} is a 422, as is
the row_ids key being absent altogether. The state exists only because the
legacy mirror needs it (see the point routes below, issue #1214); it resolves
to no members and reads back with row_count: 0.
Removing assets — excluded_row_ids (PF-12, issue #1688). An all or a
bbox selection may carry excluded_row_ids, the exposure ids removed from the
portfolio although the selection covers them: {"kind": "all",
"excluded_row_ids": [...]}. Absent means none; a rows selection never
carries it. It is written only by POST /v1/portfolios/{id}/members/remove
(an attach carrying it is a 422), which records each id on the attachment
covering it and keeps that attachment's kind — or, for a rows attachment,
drops the id from row_ids, an emptied one following the #1408 weight rule
(kept holding [] unless its weight equals a departing junction row's). No
exposure row is deleted and no exposure layer changes. Every membership reader
— /members, /aggregate, GET /v1/portfolios and /summary, the rollup,
and compute's order, preflight probe and member counters — reads the exclusion
through core.portfolio_membership.selection_match_sql. An all attachment
with an exclusion no longer selects its whole dataset, so a dataset-level
result stops standing in for the portfolio's EAD (PF-8), and the changed
selection is a new membership_digest, so a result cached before the removal
is never served after it. The request is all or nothing: an id the portfolio
does not hold is a 422 E_VALIDATION whose details.exposure_ids lists
each one; a viewer is 403, another organization's portfolio 404.
Statuses. An unknown or cross-org portfolio or dataset is 404, never
403 — the org-scoped lookup simply misses, and a 403 would confirm the id
exists. Re-attaching a pair already attached is 409 naming the existing
attachment (uq_portfolio_datasets_pair), never a 500.
DELETE …/datasets/{dataset_id} clears both planes: the attachment and
any portfolio_exposures junction rows for that dataset's points, which the
deprecated routes below may have written. The invariant is no junction row
survives whose dataset has no attachment — without it, GET …/exposures
kept listing a point GET …/members and /aggregate had never heard of, and
nothing later healed it (issue #1214).
row_count is what the selection resolves to, not the dataset's full row
count: a bbox attachment reports the points inside the window, a rows
attachment the length of its id list less any row deleted since. Every
attachment's count comes from one GROUP BY, and a member page is one
statement whatever the number of attachments — the selections are ORed into
a single predicate. Measured on the test fixture: 4 statements for a portfolio
with one attachment and 4 for one with three.
name is the shared display-name rule, so a point reads identically here
and in GET /v1/exposure-datasets/{id}/rows: properties.name verbatim when
non-blank, else "<dataset name> #<row index>" with the index numbered over
the whole dataset before any selection filter. One implementation, in
db/repositories/exposure_naming.py.
weight is the attachment's, applied to every row the selection resolves
to — the dataset plane weights a selection, not a point. It is a JSON number
here, unlike the deprecated point routes' Decimal-as-string.
Envelope divergence, deliberate. These routes return {items, total},
matching every paged resource shipped since /v1/exposure-datasets.
GET /v1/portfolios still returns {data, total}; changing that is a
breaking change to a live contract and is not made here.
Portfolio extent — bbox (EX-14, issue #1597)¶
The portfolio read (GET /v1/portfolios/{id}, and PUT on the same path) and
every GET /v1/portfolios row carry bbox [west, south, east, north] in
EPSG:4326 degrees, so Explore can fit the map to a portfolio on entry.
bbox is the union of the members' extents, one per attachment:
allcontributes its dataset'sexposure_datasets.bbox.bboxcontributes the selection box intersected with the dataset'sbbox(a box that misses the dataset contributes nothing).rowscontributes the extent of exactly the listed rows, never the dataset's.
A dataset with no declared bbox falls back to the extent of the rows its
selection resolves to. bbox is null for an empty portfolio. It is derived
on read, with no stored column: one statement
(PortfolioRepository.list_bboxes, kinds read in
core/portfolio_membership.py::selection_extent_sql) serves the list page and
the single read, so the two always agree. GET /v1/portfolios/summary does not
carry it.
What a portfolio card names (PF-7, issue #1666)¶
The portfolio read (GET/PUT /v1/portfolios/{id}, and POST's answer) and
every GET /v1/portfolios row carry:
owner— a free-text team label, at most 120 characters (portfolios.owner, migration 0117),nullwhen unset. A label, not a team: access stays organisation-wide.category—property,hotels,publicorenergy(a CHECK, no ENUM),nullwhen unset. The fifth kind of the UI, "From exposure layers", is derived from the dataset link (#1667), never stored.geocoded_precise_share— the share of members, 0 to 1, whose row does not carryproperties.geocode_low_confidence: true, the flag the address upload writes as an extra CSV column (text"true"or a JSON boolean). A row uploaded with its own coordinates counts as precise;nullwith no member. One statement for the page (PortfolioRepository.list_precise_shares), the same one the single read issues.
POST and PUT take owner and category; a category outside the four ids
or an owner over 120 characters is 422 E_VALIDATION. PUT changes either
only when the body names it, so an explicit null clears it. The list envelope
adds owners, the organisation's distinct owner labels sorted (never another
organisation's), for the upload dialog's select:
{data, total, owners}.
GET /v1/portfolios/{id}/thumbnail serves what the card draws as its small
map, {bbox, countries, weight, value_unit, cells: [{lon, lat, value}]}:
bbox is the extent above, countries the sorted ISO3 codes derived for the
member datasets (the #1166 rule), and cells at most 256 entries, each the
centre of an occupied cell of a 16 × 16 grid over bbox with the members'
attachment-weighted value summed inside it, so the cells sum to the
portfolio's exposed value. A mixed-unit portfolio sums member counts instead
(weight: "member_count", value_unit: null, K27). An empty portfolio answers
bbox: null and cells: []; removed rows (#1688) are not counted. No tile and
no basemap.
GET /v1/portfolios/{id}/spread?haz_type=&scenario=&horizon_year= serves the
synopsis's "Geographic spread", {admin_level, weight, value_unit, areas:
[{boundary_id, name, exposed_value, member_count, band}]}, the five largest
areas:
admin_levelis the deepest tier ofresult_cell_admin_levelsholding units for every country the members stand in (the admin-0 units their points fall in): a Cyprus book, with no level 2 loaded, answers at level 1.nullwithareas: []when no member stands in a loaded country.exposed_valueandmember_countsum every remaining member standing in the unit, largest first. A mixed-unit portfolio ranks bymember_count, withweight: "member_count",value_unit: nulland a nullexposed_value(K27).bandis the mean band of the unit's members that the portfolio rollup banded forhaz_type—rollup_portfolio's read of the stored answer-plane cells for the largestrollup_max_portfolio_membersmembers, on #1580's ladder — weighted as the rollup weighs them and rounded to 4 decimals;nullwhere none was banded, and everywhere whilescore_bands_enabledis off. No per-member band is stored, no job is queued and nothing new is banded (core/portfolio_spread.py).
Both reads are VIEWER+ and answer 404 for an unknown or another org's portfolio.
Portfolio EAD per hazard (PF-8, issue #1571)¶
Every GET /v1/portfolios row carries ead_by_hazard, a list of
{haz_type, ead, result_id, scenario, time_horizon} with one entry per hazard
the portfolio has an EAD for, ordered by haz_type. scenario and
time_horizon are the values stored on the result the entry read, so the card
names the cell it shows. scenario is the job's scenario_label, else the
hazard dataset's scenario, the rule GET /v1/portfolios/{id}/results uses.
time_horizon is the job's horizon_year, else the result's time_horizon
metadata, and a Today result has none. The entries are never summed into a cross-hazard
total (K26, the engine has no aggregator) and carry no unit field. For each
hazard, under the list's scenario and time_horizon filters:
- The portfolio's own run. The newest result whose job addressed this
portfolio (
jobs.params->>'portfolio_id') supplieseadas computed. - A dataset-level stand-in, only for an all-
allportfolio. With no such run for the hazard, and only when every attachment selectsall, each attached dataset's newest dataset-level result (its job named no portfolio) counts once, weighted by its attachment. All of them come from one cell (scenario and horizon), the newest cell in which every attached dataset has a result.eadis the sum ofweight x eadover the datasets andresult_idnames the newest of them. A dataset with no result in any shared cell leaves the hazard without an entry, so a partial total never reads as the portfolio's. - Otherwise no entry. A
bboxorrowsattachment holds part of a dataset, so a dataset's EAD is not the portfolio's. Such a portfolio has no entry for the hazard until a run addresses it, and the UI reads the absence as "Not computed for this portfolio".
total_ead and total_aai keep their place and type on the list row, on
/aggregate and, as ead and aai, on GET /v1/portfolios/summary. They are
the entry of the hazard whose contributing result is the newest, never a sum
across hazards. Before #1571 they summed weight x ead over member rows
of each dataset's newest result of any hazard, so a two-row dataset with EAD
1 000 reported 2 000 and the figure named whichever peril ran last. All of it
is one statement (PortfolioRepository.list_metrics), so the three surfaces
still read the same numbers and Home's summary still costs three statements.
/aggregate — totals for the book, a page of the members (issue #1215)¶
GET /v1/portfolios/{id}/aggregate answers two questions in one response, and
since #1215 they are answered at different grains — which is the whole of
the contract change:
total_ead,total_aai,coverage_pct,currencydescribe the whole book. One SQL aggregate (PortfolioRepository.list_metrics) over every member. The totals are one hazard's figures under the PF-8 rule above, andcoverage_pctis the share of members that figure covers: every member for the portfolio's own run, the members of the datasets theallstand-in read, or none. They do not move withlimit/offset.by_exposureis a page, bounded bylimit(1–500, default 100) andoffset— the same parameters and boundsGET …/membersuses — ordered by the resolver's(dataset_id, row_index), sooffsetskips the same rows on every request.by_exposure_totalis the unpaged member count.
Why it had to be bounded. Membership is dataset-grained: {"kind": "all"}
over a gridded LitPop/GHSL upload is one attachment, one POST, and tens of
thousands of members — every one of which this response previously materialised
into a single array. An over-page book now states that it was cut
(by_exposure_total > len(by_exposure)) rather than returning a short array
silently.
Why the count is not called total. GET …/members spells it total, but
here it would sit beside total_ead and total_aai and read as a third money
total, so it is by_exposure_total. The paging parameters are unchanged from
/members.
One statement for the totals, and it is the same one three surfaces read.
GET /v1/portfolios' list row, GET /v1/portfolios/summary's ranking and this
response all read list_metrics. Their agreement used to be a claim about two
independent implementations — /aggregate summed a materialised Python list —
and is now true by construction; tests/test_db/test_portfolio_aggregate_paging.py
pins it against literals for a book of mixed coverage.
coverage_pct is covered_count / exposure_count from that same aggregate,
never a counter over the returned page. Counted over a page it would be a
property of limit — 100% on a page of covered members, 40% on the next — which
is the bug paging introduces if the count is not moved into SQL with the totals.
currency names the unit of the money in the totals, which is the
covered members' shared value_unit — only covered members contribute to
total_ead/total_aai, so a book of EUR buildings with a result plus USD
buildings without one holds a EUR loss total. It falls back to the whole book's
shared unit when nothing is covered (the totals are then 0.0), and to "USD"
when the covered members genuinely disagree. It was previously the first
covered member's unit, which no page can answer for without varying with
offset; where that old rule was well defined this returns the same string.
Note this is a narrower filter than GET /v1/portfolios/summary's
value_unit, which labels exposed_value — a sum over every member — and is
correctly the whole book's unit.
The whole response costs four statements, flat in both the size of the book
and the size of the page: the portfolio lookup, the rollup aggregate, one member
page, and one resolution of that page's datasets to their latest matching impact
results. That resolution kept its shape under PF-8. It cites each dataset's
newest matching result of any hazard, so a by_exposure row can cite another
hazard's run than the totals. Both apply the same scenario and
time_horizon filters.
Breaking for clients that read the full array. A book with more members than
limit now yields a page where it used to yield everything, and the response
carries one new required field. The UI mirror — client method and type,
including by_exposure_total — is assigned to
ui#304 by the
Phase 14 orchestration plan (docs/plan/phase-14-solidify.md, session U6:
"ui#304's dispatch gets one explicit exception to consumer-only: it owns the
aggregate client method + type for the paged shape"). That assignment lives in
the plan, not in ui#304's own body, which predates this decision — so the plan
row is the record to check, and ui#304's body is not.
The three deprecated point routes¶
POST /v1/portfolios/{id}/exposures, DELETE …/exposures/{exposure_id} and
GET …/exposures are the point-grained plane this replaces. They are kept
working for one release and will be removed in a later phase; GET
…/exposures in particular is unpaged and carries neither coordinates nor a
name, which is why clients had to page the point list separately and join,
dropping whatever failed to match.
Until they go, the two writes maintain the equivalent rows attachment so
both planes agree and a book built the old way still reads through /members:
adding a point creates a rows attachment for its dataset at the
request's own weight (default 1.0 when the request names none — issue
1408; before it this always minted 1.0 regardless of what the request¶
sent) or appends the id to the existing one, which never changes that attachment's weight; removing a point drops the id.
When dropping the id would leave the selection empty, what happens depends on whether the attachment carries anything this route could not recreate — it may discard only what it could have created (issue #1214 §2, re-derived by
1408 now that the mint weight is no longer a fixed constant): compare the¶
attachment's weight to the departing point's own weight.
| Departing point's weight vs. attachment's weight | Emptying legacy removal |
|---|---|
| equal | attachment removed; a later add naming that same weight recreates it identically, so nothing is lost |
differ (set via POST …/datasets, or an earlier point added at a different weight into the same dataset-grained attachment) |
attachment kept, selection rewritten to {"kind": "rows", "row_ids": []} — the weight is recorded nowhere else and this route cannot reproduce it |
Known limitation (issue #1408): because a rows attachment holds one
weight for every row it selects, re-adding the same point at a different
weight (or adding a second point at a different weight) leaves the
attachment's weight "wrong" for whichever point didn't set it, and can leave
an attachment permanently orphaned — kept empty at a weight no removal can
match — depending on the order points are removed in. Fixing this needs
either a selection-edit route or real per-attachment provenance tracking,
neither of which exists yet; see
core.portfolio_service.sync_rows_attachment_on_remove's docstring for the
full mechanics.
A kept-but-empty attachment resolves to no members and reports
row_count: 0; re-adding the point through the deprecated route appends to it,
so the weight round-trips.
That route is the only way to refill it. There is no PATCH for an
attachment's selection (see the known gap below) and POST …/datasets for the
pair is a 409 because it is attached, so the alternative repair is detach and
re-attach — which discards the weight. Strictly better than the behaviour it
replaces, which discarded the weight unconditionally; but retiring the point
routes should land a selection-edit route or migrate these attachments.
Against an all or bbox attachment a per-point change is either already
true or not expressible at all — DatasetSelection has no "all except this"
kind, and one attachment per (portfolio, dataset) leaves nowhere to record
the difference. The point write is therefore checked before anything is
written and is a no-op, a refusal, or — on the removal side — a 404 because
there was nothing to write; never a silent divergence:
The removal side asks about membership first. A DELETE …/exposures/{id}
for a point with no portfolio_exposures row removes nothing and moves neither
plane, so it is the route's own 404 whatever the attachment covers. The 409
is reserved for a removal that really would leave the planes disagreeing, which
requires the point to be an actual member (issue #1214). The rows below
therefore separate "selected by the attachment" from "on the point plane":
| Attachment | Selected by it? | On the point plane? | Legacy add | Legacy remove |
|---|---|---|---|---|
rows |
— | — | append the id | drop the id (404 if not a member) |
all / bbox |
yes | yes | 201, selection untouched | 409 |
all / bbox |
yes | no | 201, selection untouched | 404, nothing to remove |
all / bbox |
no | yes | 409 | 200, selection untouched |
all / bbox |
no | no | 409 | 404, nothing to remove |
The 409 carries code: "CONFLICT" with selection_kind and operation in
details, and tells the caller to edit the attachment instead; the 404 carries
the ordinary E_NOT_FOUND and says only that the point is not in this
portfolio. The 409 can only fire for a caller mixing the deprecated routes with
the new ones: migration 0091 backfilled every pre-existing portfolio as a
rows attachment, and a non-rows one can only be created through
POST /v1/portfolios/{id}/datasets.
A point whose dataset_id is NULL (an ungrouped legacy row) has no dataset to
attach to and stays readable only through the deprecated route.
Known gap, not addressed here: there is no route to edit an existing
attachment's selection, so a dataset already attached (including by the legacy
mirror) can only be changed by DELETE then POST. Adding a PATCH is new
product surface and out of scope for issue #1168.
Compute and rollup no longer read portfolio_exposures: W0.4c (issue #1169)
moved every membership read onto the attachments, resolved through the one
predicate in core/portfolio_membership.py. The junction table survives only
as the write plane behind the deprecated routes above, and
tests/test_no_portfolio_exposures_reads.py is what keeps a reader from
reappearing on it.
GET /v1/catalog/coverage — the country x hazard-type grid (issue #1191)¶
The aggregate side of resources that already exist per id: one read that tells a user what their organization actually holds, which the Home Data coverage and Start here cards render directly. Role floor VIEWER.
{
"countries": [
{
"country": "Greece",
"iso3": "GRC",
"admin_levels": [0, 1, 2],
"hazards": [
{"hazard_type": "river_flood", "dataset_count": 2, "computable_count": 1,
"indicator": false, "computability": "modelled", "climate_driven": true,
"latest_ingest_at": "2026-06-30T12:00:00+00:00"},
{"hazard_type": "earthquake", "dataset_count": 1, "computable_count": 1,
"indicator": false, "computability": "screening", "climate_driven": false,
"latest_ingest_at": "2026-07-14T10:00:00+00:00"},
{"hazard_type": "tropical_cyclone", "dataset_count": 0, "computable_count": 0,
"indicator": false, "computability": "none", "climate_driven": false,
"latest_ingest_at": null},
{"hazard_type": "wildfire", "dataset_count": 7, "computable_count": 0,
"indicator": true, "computability": "indicator", "climate_driven": true,
"latest_ingest_at": "2026-08-01T09:00:00+00:00"}
],
"exposure_datasets": 1
}
],
"undetermined": {"hazards": [...], "exposure_datasets": 0},
"generated_at": "2026-09-05T08:00:00+00:00"
}
The grid is dense. Every loaded country x every HazardType member is
present, an empty cell included as an explicit dataset_count: 0 /
latest_ingest_at: null. A cell is never omitted for being empty, because a
missing key cannot be told apart from "this country does not exist" — and
seeing the gap is the point of the card. Inert and indicator-only perils
(drought, heatwave) are ordinary members whose counts simply happen to be
zero or non-computable.
The country axis is every distinct iso3 among the admin_boundaries rows
at admin_level = 0 — org-less GADM reference data, so the rows are
identical for every tenant and only the counts hung off them differ. One row
per ISO3, not per boundary row: that table's uniqueness constraint is
(source, iso3, admin_level, name), so nothing stops a GADM release shipping
two admin-0 rows under one ISO3, and a duplicated country would silently double
its counts. The surviving country name is MIN(name) among them — chosen for
determinism, not for meaning. On an install where few boundaries have been
loaded the grid is correspondingly short: that is correct behaviour, and the
endpoint never fabricates a country the platform has not loaded. With no
boundaries loaded at all, countries is [].
A dataset's country is derived, not stored — one overlap rule for both
axes, db.repositories.admin_boundary_repository.largest_overlap_admin0_iso3
(issue #1166): the ISO3 of the admin-0 boundary overlapping the dataset's
extent most, and never inferred from name, source or region (ADR-048
rejected heuristic identity backfills). What counts as the extent differs by
axis, because the two catalogs record it in different columns. An exposure
dataset is asked about its ADR-048 bbox, which core/ingest/exposure.py
writes on every ingest (derived_country_iso3). A hazard dataset is asked
about COALESCE(footprint, bbox)
(hazard_repository._hazard_dataset_country_iso3, issue #1318): no hazard
ingest path writes hazard_datasets.bbox at all, so the only extent such a row
has is footprint — the EPSG:4326 envelope of the ingested grid (ADR-045
topic 2 / ADR-057, issue #596) — and reading bbox alone placed every ingested
hazard dataset in undetermined. The derivation yields NULL when there is no
usable extent: an exposure upload from before issue #1660 with no bbox yet, a
hazard ingest that resolved no grid (footprint and bbox both NULL — repair
with scripts/backfill_dataset_footprint.py, see the
runbook), a malformed value, no admin-0 boundary loaded, or no
overlap. NULL means "not derivable", not "in no country", so those
datasets are reported under undetermined — which carries the same dense
hazard axis — rather than dropped. A silently discarded dataset is exactly the
invisible gap this endpoint exists to expose.
computable_count is a promise about the compute gate, not a second
opinion: it counts the cell's datasets that
core.eai_eligibility.classify_eai accepts on the unit and event axes — the
identical classifier behind GET /v1/hazards' computable field (ADR-069,
issue #920) — so it is always <= dataset_count. latest_ingest_at is the
newest created_at among the cell's datasets — when the catalog row was
committed by its ingest.
indicator is true iff the cell holds at least one dataset and every one
of them is indicator-only — the same classifier's verdict.is_indicator,
which is EaiVerdict's own negation of is_computable (EaiPosture has
exactly two members), never a second rule and never inferred from
computable_count after the fact even though the two happen to agree. It is
false for a mixed cell, a fully computable one, or an empty one (issue
1375, Phase 16 D3). This is the per-cell counterpart of the flat indicator¶
flag GET /v1/hazards already serialises per dataset (issue #600): a peril
like wildfire — a curve exists, but every open FWI dataset publishes a unit no
damage function can consume (#481) — shows as 7 datasets, 0 computable
without this flag, which reads as broken rather than as the deliberate
indicator-only posture it is. Home's "Start here" chips (Phase 16 W2.2) label
the chip indicator from this field rather than re-deriving it client-side
(guardrail 8).
computability folds the cell's verdicts into one of four words (issue
1467, Phase 17 D14), checked in this order: none when dataset_count is 0;¶
indicator when the cell's indicator flag is true; otherwise screening
when core.eai_eligibility.eai_grade declares the peril's EAI screening-grade
(earthquake, ADR-060; coastal flood, ADR-076) and modelled when it does not.
It is read from the counts, flag and grade the cell already carries, never a
second rule, so none ⇔ dataset_count == 0 and indicator ⇔
indicator: true on every cell, including undetermined's; a mixed cell (one
computable dataset among indicator ones) reads modelled/screening, because
it can be computed. Every scope option, card and measure renders "not
available for X" from this field instead of showing a zero, and the UI renders
it as given rather than re-deriving it from the counts (D14).
climate_driven says whether the scenario axis changes the hazard (issue
1592, run model § 3.4): true iff the cell holds a dataset declaring a¶
scenario other than baseline/historical (its scenario or any of its
scenarios). An empty cell and a cell of baselines only (earthquake) read
false. It comes from core.hazard_conditioning.is_climate_driven, the one
predicate over (haz_type, ISO3 set) that cell resolution also calls, fed by
the scenario/scenarios columns riding the hazard groups (still three
reads).
admin_levels lists the GADM tiers loaded for each country, ascending
and distinct: [0] is the country outline only, [0, 1, 2] adds regions and
districts. It comes from the same statement as the country axis
(AdminBoundaryRepository.list_countries), as a correlated subquery that reads
idx_admin_boundaries_iso3_level per country; the axis itself is unchanged —
still admin-0 rows only, so a country with only region rows loaded never appears
and a region's name never replaces the country's. The grid is still three
reads, whatever the size of either axis. undetermined carries no
admin_levels — it is not a country.
Superseded datasets are excluded from every number on the grid —
dataset_count, computable_count, indicator and exposure_datasets —
matching what
GET /v1/hazards has done by default since issue #873: ADR-048 withholds a
superseded row from default selection, so counting it would overstate coverage
and promise a cell that selection will never pick. The rule is deliberately the
same on both axes: a country row carries a hazard count and an exposure count
side by side, and one of them counting re-ingests while the other did not would
leave a reader unable to tell which number to believe. A NULL/unset status
counts as active.
The same rule governs the list a cell drills into. GET /v1/exposure-datasets
excludes superseded rows by default too (issue #1205), so the total a reader
sees after clicking a country row equals the exposure_datasets number they
clicked. It did not until #1205: the grid gained the predicate and the list never
did, so a cell reading N linked to a list reporting N+1 and the client had no
parameter to reconcile them. Both surfaces now share one SQL predicate
(_not_superseded() in db/repositories/exposure_dataset_repository.py) rather
than two copies that can drift apart again. Retired rows are withheld from the
default, never lost — ?include_superseded=true lists them for an ops or audit
view, widening the page and total together. This is the exposure twin of what
GET /v1/hazards has done since issue #873.
Three limits on that equality, none of them lifecycle bugs:
- It is per country row. The
undeterminedbucket'sexposure_datasetscounts datasets whose country is not derivable, and?country=only matches an ISO3 — so that one number has no drill-down that reproduces it. Giving the list a NULL-country filter is a separate change. - It is eventual, not instantaneous. The grid is memoised per organization
for
COVERAGE_CACHE_TTL_SECONDS(60 s, per API worker), so for up to a TTL after a re-ingest marks a row superseded the list drops it while a cached grid still counts it. - The legacy
GET /v1/exposures/catalogis not aligned. It groups the point table and appliesstatusonly when the caller passes?status=, so an unqualified read there still lists retired rows. Aligning that alias is a separate contract change.
Assembled server-side in three reads, whatever the size of either axis (country axis, pre-aggregated hazard buckets, per-country exposure counts). No read is per country or per hazard type; a regression test asserts the count is flat as the country axis grows. Per phase-14 guardrail 8 the UI must render this grid, never re-derive coverage client-side.
Cached per organization for 60 seconds, in-process and keyed by org id and
nothing else. generated_at is stamped when the grid was assembled, not when
it was served, so a cached answer says how stale it is.
Upload kinds of Data: damage curves, footprints, boundaries, NetCDF (issue #1670)¶
DT-7's four upload kinds that had no backbone route (the UI is ui#876 and ui#822).
- Damage curve —
POST /v1/impact-functionstakesexposure_type(defaultbuildings) andis_default(default false). The one-default index on(org_id, haz_type, exposure_type)(DT-5, #1576) answers a second default for the same pair with 409CONFLICT; a default for another exposure type of the same peril is stored. - Event footprint, intensity —
POST /v1/hazards/uploadwithsingle_event=trueandevent_date(YYYY-MM-DD) ingests a GeoTIFF as one observed event whosevalid_atis 00:00 UTC of that day. A GeoTIFF states no date, so a single-event GeoTIFF withoutevent_dateis refused; on a CLIMADA storeevent_datewins over the file's own date.event_datewithoutsingle_eventis a 422. - Event footprint, outline —
POST /v1/hazards/footprints(multipart:file,name,haz_type,country,event_date,source,license; ANALYST+) takes a GeoJSON Feature or FeatureCollection of polygons in EPSG:4326 that join into one connected area, and answers 201 with a single-event dataset whosefootprintis the outline and which stores no intensity.GET /v1/hazards/{id}readsoutline_only: true— derived, never a column: the dataset holds afootprintand neithernpz_pathnorcog_path— andcomputable: false. Every compute route refuses it with 422E_VALIDATION,details.reason= "this footprint is an outline with no intensity, so no loss can be computed": an outline never yields a loss, an expected annual impact or a band. Parts that do not join, a non-polygon or a non-degree outline are 422. - NetCDF hazard —
POST /v1/hazards/uploadaccepts a.ncfile, sniffed on its magic bytes (classicCDF\x01/CDF\x02, or HDF5 under a.ncname), in one declared layout (ADR-051): one variable (netcdf_variable, defaultintensity) on(return_period, lat, lon)for return-period maps — the coordinate strictly ascending years, which become the job'sreturn_periods— or on(lat, lon)for an indicator or one event (single_event+event_date),latitude/longituderead too, on a regular EPSG:4326 grid. The route reads only the header (h5py or SciPy) and dispatches the job withsource_format: "netcdf"andnetcdf_layout, anupload:<variable>:<lat>:<lon>[:return_period]name the chord's NetCDF reader rebuilds the spec from; a file in any other layout (an extratimeaxis, a missing variable) is a 422 naming the expected one. - Boundaries —
POST /v1/admin-boundaries/uploads(multipartfile,level; ANALYST+) takes a GeoJSON FeatureCollection whose features are (Multi)Polygons each with anameand aniso3, and alevelof1(regions),2(regional units),3(municipalities) orcustom(another set of areas, loaded at 3). A feature without a name, a non-polygon or a file over the 20 MiB cap is 422. It stores the file in object storage and answers 202 with anawaiting_reviewpending_ingestsrow of sourceuser_uploadwhosemetadatacarries the level, the feature count, the ISO3 set and the askingorg_idandrequested_by_user_id— and loads nothing. A platform admin'sPOST /v1/admin/pending-ingests/{id}/approveloads the areas in its own transaction asadmin_boundariesof their country (sourceuser_upload, version the row id) and answers withloaded; a rejection loads none, and an organisation admin's approval is 403.GET /v1/admin-boundaries?level=3&iso3=<ISO3>lists them.
GET /v1/admin-boundaries — list and search admin units (issue #1468)¶
The scope picker (country · region · municipality) and the offline geocoding
fallback list the loaded GADM units through this read. Role floor VIEWER;
admin_boundaries is org-less reference data, so every organization reads the
same catalogue.
{
"items": [
{"boundary_id": "…", "iso3": "GRC", "admin_level": 1, "name": "Attica",
"parent_id": "…", "parent_name": "Greece",
"bbox": [22.9, 37.6, 24.2, 38.4],
"source": "GADM", "version": "4.1", "license": "…", "citation": "…",
"source_url": "…"}
],
"total": 1
}
- Filters —
level(0 country, 1 region, 2 district),iso3(exact, case-insensitive) andq, all optional and combinable;limit(1–200, default 50) andoffsetpage the result, andtotalcounts the matches before paging. qis a case-insensitive substring match onnamewith the LIKE metacharacters taken literally:q=%finds only a name containing%. Nounaccent/trigram extension — GADM names are Latin transliterations, and an extension would need a migration.- Ranking — with
q, names that start with it come first; thenadmin_levelascending, thenname. - No geometry.
bboxis the unit's extent ([min_lon, min_lat, max_lon, max_lat], EPSG:4326); the polygon itself is never served here. Outlines come from the/geometrysibling below. - Nothing loaded is an empty page —
200 {"items": [], "total": 0}, never an error, so the offline fallback can say "no boundaries loaded".
GET /v1/admin-boundaries/{boundary_id} — one admin unit by id (issue #1589)¶
Explore reads each unit of an admin:<id> or admin:<id>,<id> clip through
this route, one id per call: the unit's name for its chip, its admin_level
and iso3 for the one-level and one-country check of a pasted link, and its
bbox to fit the map over the union. Role floor VIEWER.
- The list's item shape, exactly —
boundary_id, iso3, admin_level, name, parent_id, parent_name, bbox, source, version, license, citation, source_url, under the usualdataenvelope. No polygon; outlines stay on/geometry. - One id per call. There is no
ids=filter; a clip of several units is read one id at a time, so an unknown id fails for that unit alone. - Errors — an unknown well-formed id answers 404
E_NOT_FOUND, a malformed id 422E_VALIDATION, and a call without credentials 401. - Declared after
/geometry, so/v1/admin-boundaries/geometryis never read as an id.
GET /v1/admin-boundaries/geometry — simplified outlines (issue #1469)¶
The scope picker outlines the chosen region or municipality and frames the map
on it. Role floor VIEWER; the response is a bare GeoJSON
FeatureCollection (no data envelope, like /v1/results/{id}/geojson) that a
map source takes as-is:
{
"type": "FeatureCollection",
"features": [
{"type": "Feature", "geometry": {"type": "MultiPolygon", "coordinates": ["…"]},
"properties": {"boundary_id": "…", "iso3": "GRC", "admin_level": 1,
"name": "Attica", "source": "GADM", "version": "4.1",
"license": "…", "citation": "…", "source_url": "…"}}
],
"total": 1,
"truncated": false
}
- Parameters —
level(0–2),bbox(minLon,minLat,maxLon,maxLat, EPSG:4326) andzoom(0–22) are required;limitdefaults to 200 and caps at 500. A missing or malformedbboxis 422E_VALIDATION, so a response is always bounded. - Selection — the units at
levelwhose stored polygon intersectsbbox(the same clauselist_polygons_at_leveluses), ordered byname.totalcounts every intersecting unit;truncatedis true whenlimitcut the collection short of it. - Simplification in the query only —
ST_SimplifyPreserveTopology(geom, 180 / (256 · 2^zoom)), half a pixel of the zoom in degrees. The storedadmin_boundaries.geomis never modified, and the result map's admin path keeps its full-resolution geometry. The real GADM Attica polygon (2 433 vertices) serves 533 at zoom 6. Coordinates are written to 7 decimals, and Postgres renders the feature array itself (json_agg), so no vertex is parsed on the API's event loop. - Not a vector tile — the tile surface is #1472's. Licence: these are
GADM-derived polygons handed to org members; #1464's verdict
(
gadm-ok-self-hosted-only) and the per-country geoBoundaries fallback are recorded indocs/DATA_SOURCES.html.
event_axis / computable — the catalog's compute-gate prediction (issue #920)¶
GET /v1/hazards and GET /v1/hazards/{id} both carry two fields a client
builds its "can I run this?" affordance from:
| Field | Type | Meaning |
|---|---|---|
event_axis |
"return_period" | "scenario_year" | null |
Which axis this dataset's hazard_events rows sit on (ADR-065). null means the dataset holds no events at all — an unfinished or failed ingest, which is not a posture. |
computable |
bool | What core.compute_service.validate_impact_refs would decide about this dataset, on the unit and event axes together. |
computable: false means POST /v1/compute/impact naming this dataset is
refused before anything is queued — with E_HAZARD_DATASET_NO_EVENT_FREQUENCIES
(no annual frequency to integrate, ADR-069) or
E_INDICATOR_DATASET_NO_EAI (no seeded curve consumes the intensity unit,
issue #600). eai_eligibility.reason names which. indicator is the same
posture read the other way round and can never disagree with it.
Two boundaries worth stating:
computabledoes not cover the curve/exposure-type refusal (ADR-061). That one is keyed on the impact function a job names, which no catalog row knows, so acomputabledataset can still be refused for aiming a built-in buildings curve at population exposure. The catalog reports that axis as theeai_eligibility.by_exposure_typematrix rather than folding it into a boolean it cannot honestly compute.- The axis is read from the events, never inferred from the dataset row.
return_periods IS NULLlooks like a free answer and ADR-069's alternative 5 rejected it for the compute gate: the column is a derived summary written only by the post-ADR-067 commit step and has been nullable since migration0002, so a legacy row that never recorded one reads as a cube. The catalog must reach the same verdict as the gate, so it reads the same thing the gate reads.
Cost: the list endpoint resolves the whole page's axes in one
GROUP BY dataset_id aggregate (HazardRepository.get_event_axes), not one
probe per row; the detail endpoint keeps ADR-069 §2's single bounded row
(HazardRepository.get_event_axis). See
core/dataset_computability.py.
Before #920 the catalog classified a dataset by its intensity unit alone, so a
single-scenario multi-year cube with a consumable unit (the #797 class)
advertised indicator: false and then 422'd on Run — the dead end these
fields exist to prevent.
The /v1/risk/* score card had the same defect on its own indicator field
and was closed the same way by issue #930 — same classifier, same two
arguments, so the two surfaces and the gate are one verdict rather than three
derivations. core.eai_eligibility.classify_dataset_eai and
is_indicator_dataset now require event_axis with no default, which is
what makes leaving the question unasked a TypeError rather than a silently
wrong answer.
country — the hazard catalogue's derived country (issue #1369)¶
GET /v1/hazards and GET /v1/hazards/{id} both also carry country: str |
null — the ISO3 the row's extent overlaps most, by the same rule the
/v1/catalog/coverage grid uses (largest_overlap_admin0_iso3 over
COALESCE(footprint, bbox) — see "A dataset's country is derived, not
stored" above for the full derivation, including why a hazard row reads
footprint first and an exposure row does not). null means "not
derivable", never "in no country", exactly as it does there.
?country= (GET /v1/hazards only) filters to one ISO3, matched
case-insensitively (upper-cased server-side) against the same derivation, in
SQL and ahead of limit/offset — mirroring GET /v1/exposure-datasets'
own ?country= (issue #1166). A null-country row never matches a value.
Resolved via HazardRepository.list_datasets_with_country /
get_dataset_with_country, the country-carrying siblings of list_datasets /
get_dataset_by_id — the bare methods are unchanged and keep serving every
other caller (the compute gate, ingest, tiles, results, …), which wants the
plain HazardDataset and has no use for the derived country.
search — hazard catalogue narrowing (issue #1406)¶
?search= (GET /v1/hazards only) is a case-insensitive substring match
against name or source, applied in SQL (ILIKE, with %/_ in the
term escaped so they match literally) ahead of limit/offset — mirroring
GET /v1/exposure-datasets' own ?search= (issue #1166's precedent), widened
to source too since a hazard picker's most common query is the provider
name (e.g. "JRC"), not the dataset's own display name. Composes with every
other filter (country, haz_type, status, …). Resolved via
HazardRepository.list_datasets_with_country's search parameter; the bare
list_datasets (the compute-matching path) is unchanged and carries no
search filter — narrowing a browse/picker result is not a concern the
compute gate or lookup resolver share.
cumulative — a watch's frames and its event-to-date snapshots (issue #1767)¶
GET /v1/hazards and GET /v1/hazards/{id} both carry cumulative: bool,
read from hazard_datasets.cumulative (issue #1607, migration 0105). It is
true only for a wildfire watch's cumulative snapshot — the event to date,
every cell the maximum over the overpasses released so far — and false for
every overpass frame and every catalogue dataset. A wildfire watch holds both
kinds of row at one valid_at, so GET /v1/hazards?live_event_id=<id> lists
two rows per instant and this field is what tells them apart.
?cumulative=true|false (GET /v1/hazards only) keeps one kind: with
live_event_id, false lists the frames and true the cumulative snapshots.
Omitted, it filters nothing, so every existing query returns what it did. It
is one more AND predicate, applied in SQL ahead of limit/offset, and
does not lift the default snapshot exclusion: without live_event_id,
cumulative=false is the unchanged catalogue and cumulative=true is empty.
Both HazardRepository.list_datasets and list_datasets_with_country take
the same cumulative keyword. GET /v1/live-events/{id}/snapshots is a
separate reader and stays frames-only.
GET /v1/engines — the compute engine roster (issue #1054)¶
Any authenticated caller — org-agnostic, no RLS dependence — gets the same answer:
{ "data": [
{ "name": "climate_lama", "available": true, "default": true,
"description": "...", "supported_hazards": ["river_flood", "..."],
"conformance": null },
{ "name": "climada", "available": false, "default": false,
"description": "...", "supported_hazards": ["river_flood", "..."],
"conformance": null },
{ "name": "oasis", "available": false, "default": false,
"description": "...", "supported_hazards": [],
"conformance": {
"reference_model": "OasisPiWind 2.5.7 case_0 (ground-up)",
"reference_revision": "f6b115422448fac8d998aaf50db1124f18f74f8a",
"oasislmf_version": "2.5.7", "workflow": "oasis-conformance.yml" } }
] }
availableis the registry's config probe (worker.models.registry.available_engines) — a deployment check, not a live health probe (ADR-042).climadareportsfalsewhereverCLIMADA_SIDECAR_URLis unset, andoasiswhereverOASIS_SIDECAR_URLis unset (ADR-084), with no network call.defaultisname == core.engines.DEFAULT_ENGINE.runtime(issue #1669, EN-8) is how the engine runs —in_processinside the worker (climate_lama) orhttp_sidecarbehind its sidecar's HTTP API (climada,oasis) — fromcore.engines.ENGINE_RUNTIMES.version(issue #1669, K32) is the installedclimate-lama-enginedistribution for the in-process engine (importlib.metadata). A sidecar is never asked (ADR-042): itsversionis theengine_versionits newest completed result in the caller's organisation recorded, andnullwhen it has none or is not configured here.descriptionis a short static string owned bycore/engines.py(ENGINE_DESCRIPTIONS) — presentation text for API consumers, deliberately not sourced fromModelInterface(which has nodescription) or fromEngineRegistration.description(an internal/ops note, not user-facing).supported_hazardsis read from each adapter's own declared class attribute viaworker.models.registry.engine_supported_hazards— the engine is never instantiated to answer this, so an unavailable engine's declared hazard coverage is still reported honestly without ever building (let alone connecting) it. See theclimada/climate_lamacoverage table underX-Enginebelow — same source, same values.conformanceis a static record fromcore.engines.ENGINE_CONFORMANCE(issue #1478):nullexcept foroasis, which names the reference model it is gated against, the pinnedoasislmfversion and the gate's workflow. The path-scopedoasis-conformance.ymlPR gate keeps it truthful — a sidecar change cannot merge with the reference check red — and a drift test fails when the record's pin differs fromdocker/oasis-sidecar/Dockerfile's.
X-Engine — request-time engine selection (issue #377, phase 8.15)¶
Engine plurality is a product commitment, so every compute submission —
POST /v1/compute/impact, POST /v1/compute/impact/matrix (applies to every
cell of the matrix) and POST /v1/compute/cost-benefit — accepts an optional
X-Engine header naming the engine to run on.
The selector is a header, not a body field: it picks the runtime rather than
describing the calculation, so it stays out of the request schemas and out of
anything that round-trips a saved scenario. Selectors are normalised (case and
-/_ are interchangeable), so X-Engine: Climate-Lama == climate_lama.
- Omitted → the default engine (
climate_lama, the in-process climate-lama-engine adapter). Behaviour is unchanged from before the header existed. - Unknown engine →
422withE_ENGINE_UNKNOWN; the error details list the registered engines. Nothing is queued. - Registered but not deployed here →
400withE_ENGINE_UNAVAILABLE.
The resolved name is threaded into the job params and stamped on the result
provenance as engine_name (next to engine_version), and — for non-default
engines — folded into the result-cache key so two engines never share a cache
entry. worker/models/registry.py owns the name → ModelInterface mapping; it
holds lazily-invoked factories and an availability probe per engine, so an
engine running behind a process boundary registers exactly like an in-process
one. There is no fallback: a job that names an engine the backbone cannot serve
fails rather than being rerouted (ADR-024).
Exposure selector on impact compute (issues #1008, #1586, ADR-087)¶
POST /v1/compute/impact, POST /v1/compute/impact/matrix and
POST /v1/compute/impact/compare share one request surface (RunInputs in
api/v1/compute.py) and name the exposure set with at most one selector:
| Field | Meaning |
|---|---|
exposure_dataset_ids |
The union of these exposure datasets, for example LitPop Greece plus LitPop Cyprus in one run. All must share one value_unit and one exposure_type. A one-element list is the legacy exposure_dataset_id, with the same job params and the same cache key |
exposure_dataset_id |
One exposure dataset (legacy, kept as the alias) |
portfolio_id |
The members of a portfolio (#1008), now on /compare too |
| none | Every exposure the hazard covers in the caller's org, unchanged |
Refusals, each a 422 E_VALIDATION whose details.reason is the text:
- members that differ in unit or type: "exposure datasets differ in value_unit: EUR, USD" or "exposure datasets differ in exposure_type: buildings, population" (the unit is checked first, the values sorted).
exposure_dataset_idsempty, or sent withportfolio_idorexposure_dataset_id: "name one exposure selector".- a member outside the caller's org: "exposure_dataset_id
not found in your organization.", as for the single field.
The selector is written to the job params once (ExposureSelector.job_params
in core/eai_alignment.py: a union as the sorted exposure_dataset_ids) and
read back by exposure_selector_for_result on every path that re-runs the
compute order: /per-exposure, the result tiles' assets layer, the GeoJSON
rollups, reports, the surface writer and cost-benefit. A union result is an
ordinary ImpactResult stamped with its first compute-order row's dataset, and
its provenance carries exposure_dataset_ids, sorted. Its cache key appends the
sorted ids and the #1568 membership digest over the members' dataset_sha256,
so a re-ingest of any member is a new key, and a request without the field keeps
its key byte for byte.
Scope clip on impact compute (issue #1588, EX-15, EX-23, BC-1)¶
The same three endpoints take an optional scope that clips the exposure set
the selector names (with no selector, today's org-wide set) to a place. It names
exactly one of three keys:
scope key |
Meaning |
|---|---|
admin_boundary_ids |
One or more admin units of one admin_level and one country (iso3, as GET /v1/admin-boundaries serves them). The clip is their union, each row counted once |
admin_boundary_id |
One unit, the one-element alias of admin_boundary_ids; read as a one-element list at the request boundary, so both forms store, key and compute alike |
bbox |
[w, s, e, n] in EPSG:4326 |
Refusals, each a 422 E_VALIDATION whose details.reason is the text:
- an id with no loaded boundary, alone or beside known ones: "unknown admin
boundary
". - units of mixed levels: "admin boundaries differ in admin_level: 1, 2".
- units of mixed countries: "admin boundaries differ in country: CYP, GRC".
- a malformed box: "bbox must be [w, s, e, n] with w < e and s < n inside ±180 and ±90".
- an empty
admin_boundary_ids, or ascopenaming two keys, is refused by the request model (details.errors).
A clip that leaves no rows has nothing to compute: the job fails with the worker's empty-load refusal, worded for the clip ("No exposure row of this run lies inside its scope clip; there is nothing to compute.").
The clip rides the job params in its stored form (ExposureSelector.scope_record):
a boundary clip as {"admin_boundary_ids": [...]} with its ids sorted, whichever
key or order the request used, and a bbox as sent. The worker applies it inside
compute_order_sql (clip_predicate_sql, an EXISTS over the units'
admin_boundary_parts, or ST_MakeEnvelope), narrows scope_total with the same
predicate, and persists it as result.metadata.scope and provenance.scope.
A matrix batch also holds it, as params.scope on the batch row (#1806), in the
same stored form and only when set.
Every path that re-runs the compute order (/per-exposure, the result tiles'
assets layer, the GeoJSON rollups, reports, the surface writer) reads it back
through exposure_selector_for_result, so a clipped result's figures, rows and
tiles describe the same subset. Cost-benefit inherits the base result's clip
(K20). The cache key appends the stored form only when set, so two orders of one
list share a key, two clips mint two keys, and a request without scope keeps its
key byte for byte.
Exposure growth on impact compute (issue #1593, EX-15, K18)¶
/impact, /impact/matrix and /impact/compare take an optional
exposure_growth_rate g, with −0.5 ≤ g ≤ 0.5. Outside the bound the request is
a 422 E_VALIDATION whose details.reason is "exposure_growth_rate must lie in
[-0.5, 0.5]". A rate of 0 is the same run as no rate: same job params, same cache
key, no provenance block.
The worker applies it in step (3) of the run model's worker order, after the
selector load and the scope clip and before vulnerability: each row's value,
and the run's deductible and cover for that row, are scaled by
(1+g)^(horizon_year − reference_year). reference_year is the row's own
dataset's value epoch, resolved by core.exposure_growth.resolve_reference_years
in this order: properties.reference_year on the dataset's rows (the LitPop pack
stamps 2018), else exposure_datasets.temporal_coverage_end, else the run year.
A union of datasets with different epochs grows each row by its own factor, so
the insurance terms reach the engine as per-point arrays. An engine that takes
only one flat term (the sidecars) reports insured_loss null with
metadata.insured_loss_note "not supported by this engine" when the factors
differ. A Today cell (no horizon year) has exponent 0. Growth scales existing
locations only: no new location enters a hazard zone.
The run records provenance.exposure_growth = {rate, horizon_year,
reference_years: {<dataset_id>: {year, source}}}, source being
properties.reference_year, temporal_coverage_end or run_year. The cache key
appends exposure_growth_rate=<g> to its labelled tail only when g is set and not
0, followed by growth_run_year=<year>, the year the run is submitted in: a
dataset with no epoch compounds from it, so a grown result never answers a later
calendar year. GET /v1/exposure-datasets and GET /v1/exposure-datasets/{id} serve the same
resolver's answer as reference_year = {year, source}, with {year: null,
source: "run_year"} for a dataset that records no epoch.
Adaptation measures on impact compute (issue #1594, AD-1, AD-5)¶
/impact, /impact/matrix and /impact/compare take optional measure_ids,
measures of the caller's library, and adapted_from_result_id. The run is an
ordinary adapted result: same table, same GET /v1/results/{id}, its own
eai_exp and its own vector tiles. Refusals are 422 E_VALIDATION with
details.reason, on these three routes and on /cost-benefit:
- an id outside the caller's org:
unknown measure <id>; - a measure for another hazard:
measure <id> is for coastal_flood, the run is river_flood(AD-5). A measure is never re-typed to the run's hazard, and the cost-benefit worker sends each measure's own hazard to the engine; - a depth shift on a hazard dataset whose
intensity_unitis notm:measure <id> shifts the depth in m, the hazard intensity is in cm(AD-15, #1622), withdetails.shift_unitanddetails.intensity_unit. The shift is never converted to the hazard's unit./impact/matrixchecks its anchor cell at submission, and the worker checks every cell and the cost-benefit future hazard.
The job params carry the ids sorted, and the worker applies them in that
order in step (5) of the run model's worker order, after growth and
vulnerability and before ImpactCalc: worker.tasks._apply_measures hands each
measure to the default engine's adapter (EngineAdapter.apply_measures, the only
code that builds an engine Measure), which first drops the event rows its
protection standard holds back (rp <= rp_protect, through
core/protection_standard.py) and then applies the hazard-intensity factor, the
depth shift, the frequency cutoff and the MDD/PAA transforms. A measure's
hazard_inten_shift of h metres ("raise the asset or floor by h") is the
engine's additive term haz_inten_b = -h with haz_inten_a = 1, so each cell
reads max(depth - h, 0); a barrier or levee is a protection standard
(rp_protect) and never a depth subtraction (K30). The adapted hazard and curve come
back as arrays, so any selected engine computes the adapted run. An adapted run
writes no answer-plane surface, so it never overwrites the unadapted answer. The
same holds for a grown, insured (deductible or cover) or clipped run: only an
unmodified run files its answer under the hazard cell's surface key.
adapted_from_result_id must name a result of the caller's org (404
otherwise). It is recorded as provenance.adapted_from_result_id, beside
provenance.measure_ids, and on a saved Scenario's inputs. The worker never
reads it. The cache key closes its labelled tail with measure_ids=, each sorted
id with the SHA-256 of the canonical JSON of the measure columns the transform
reads (core.measure_transform.measures_cache_component), so patching a
measure's transform is a new key and renaming it is not.
Hazard options for an exposure selector (issue #1661, EX-19, BC-3, EX-18, EX-23)¶
GET /v1/compute/hazard-options tells the hazard picker, Scope's head and the
assistant what each hazard type can do for an exposure set before any run. It
takes the selector as exposure_dataset_ids (repeated) or portfolio_id, and
the optional clip as admin_boundary_ids (repeated) or bbox=w,s,e,n. Any
role may call it (VIEWER+); it queues no job and writes no result.
{"scope_total": 5,
"hazards": [{"haz_type": "river_flood", "climate_driven": true, "state": "covered",
"reason": null, "money": true, "money_reason": null,
"cells": [{"scenario_label": "baseline", "horizon_year": null,
"hazard_dataset_id": "…"}]}]}
| Field | Meaning |
|---|---|
scope_total |
The selector's exposure rows inside the clip, or all of them without one. A dataset or a union counts what a run of the same scope records as scope_total; a portfolio counts every member inside the clip, since no hazard footprint narrows a read |
hazards[] |
One entry per hazard type, in HazardType order |
state |
covered, no_single_dataset or not_covered, from core.compute_service.resolve_hazard_option (#1590), the function the compute refusal calls |
reason |
null for covered, otherwise the details.reason /impact refuses that hazard with |
climate_driven |
core.hazard_conditioning.is_climate_driven (#1592) over the set's countries |
cells |
For a covered hazard, every (scenario_label, horizon_year) its datasets declare that resolves, each with the hazard_dataset_id a /impact/matrix of that cell stages; empty otherwise |
money |
true when the hazard is covered, not an indicator hazard, and the vulnerability resolution /impact runs (core.impact_function_service.resolve_impact_function_id, #1591) raises no refusal for the selector |
money_reason |
That refusal's details.reason verbatim ("no money loss for value unit m3", "no vulnerability curve for exposure type population"), "indicator: a score, no money" for an indicator hazard, and null when money is true or the hazard is not covered |
The selector and the clip are refused as /impact refuses them, a 422
E_VALIDATION with the same details.reason: "name one exposure selector"
(none, or two), "exposure datasets differ in value_unit: EUR, USD", the clip
texts of "Scope clip on impact compute" above, and "send admin_boundary_ids or
bbox, not both". Another organisation's portfolio is a 404, and its dataset the
422 /impact gives, so the read never names that organisation's datasets. Core:
core/hazard_options.py::hazard_options_for_selector.
/v1/compute/impact/compare — multi-engine agreement view (issue #379, phase 8.15)¶
Runs one scenario through N registered engines and puts the answers side by side. Multi-engine agreement is a validation signal: engines that independently land on the same EAD corroborate the number; a wide divergence flags something worth investigating. It is not a ranking, and there is no blending or weighted consensus.
POST fans the inputs out to one ordinary compute_impact job per engine and
returns 202 with a comparison_id; GET .../compare/{comparison_id} serves
the group. Group tracking reuses the compute_batches row that the scenario
matrix (#380) uses — params.kind discriminates engine_comparison from
scenario_matrix, and a group is never served through the other's route.
Contract points that callers depend on:
enginesis optional. Omitted → every engine available on this deployment, further narrowed to engines whose declaredsupported_hazardscovers this hazard dataset's peril; an engine skipped either way is reported inparams.excluded_enginesas{"engine": ..., "reason": ...}with reason"unavailable"(registered but not deployed here) or"unsupported_hazard"(registered, deployed, but never measured as computing this peril — issue #1055). Named explicitly → an unregistered engine is422(E_ENGINE_UNKNOWN) and one that cannot run here is400(E_ENGINE_UNAVAILABLE), both before anything is queued; hazard support is not checked on this path — asking for "whatever you have" never fails over an engine the caller never named, and asking for a specific engine never silently drops it, for either exclusion reason.- A group of one is valid. Until a second engine is deployed, a comparison reports the single engine's metrics with zero divergence. Degenerate, not an error.
- Partial views beat failures. An engine whose job failed is named in
divergence.failed_engines(with itserror_messageon its entry) and excluded from the maths; engines still running are inpending_engines. The group still rolls up tocompletedonce every job is terminal, and no engine failure turns the view into a 500. An engine skipped before dispatch (seeexcluded_enginesabove) never reaches either of these lists — it never had a job to fail or run. - Divergence is relative to one reference — the default engine when it is in
the group, else the first engine submitted — as a signed percentage,
(value - reference) / |reference| * 100, per scalar metric (ead,aai,insured_loss) and per frequency-curve return period. A divergence isnullwhere it is undefined (zero reference against a non-zero value); two zeros agree exactly. - Each engine view carries
engine_version(issue #1055, D12) — astr | Nonecopied from that job'sImpactResult.metadata["engine_version"](stamped by whichever adapter ran it;Nonewhen the field was never written, never a fabricated placeholder). Additive: existing consumers readingfailed_engines/pending_enginesare unaffected. - No cache short-circuit. Unlike
POST /v1/compute/impact, a comparison always dispatches a job per engine so the group has a cell per engine. queued_aheadtells a long queue from a slow engine (issue #1640, EN-4). While any cell ispending, the view carries one count per group: how many messages on the impact queue (the queuetask_routesnames forcompute_impact) a worker consumes before the group's oldest pending cell. It counts every organization's queued work — the broker decides the order, and the API's row-level-security bound role cannot see other orgs' jobs — and is read bycore/queue_position.py, the only code that reads the broker's queue list, from the consuming end and no further thanQUEUE_POSITION_SCAN_LIMITmessages (default 1000). It isnullonce no cell ispending(acancelledcell does not count), when the cell sits further back than the scan window, or when the broker cannot be read withinCELERY_BEST_EFFORT_PUBLISH_TIMEOUT_SECONDS— "not known", never a guess. No ETA and no per-cell position.
Per-engine supported_hazards (ModelInterface.supported_hazards), what the
default group in engines is narrowed against:
| Engine | supported_hazards |
|---|---|
climate_lama (default, in-process) |
river_flood, tropical_cyclone, wildfire, storm_europe, earthquake, coastal_flood |
climada (sidecar) |
river_flood, tropical_cyclone, wildfire, storm_europe |
oasis (sidecar, ADR-084) |
none yet, #1480 measures the first peril |
climada's list is a claim that a real sidecar run landed within 1% EAD of
climate_lama for that peril (tests/parity/), not a claim about what the
sidecar can be made to accept — CLIMADA's haz_type is a free-form tag it
never dispatches on. earthquake and coastal_flood have parity test files
(tests/parity/test_parity_earthquake.py, test_parity_coastal_flood.py)
but no measured result yet; see tests/parity/README.md for status and how
to run one. oasis follows the same rule and declares nothing until a peril is
measured, so the default group always excludes it as unsupported_hazard. Name
it in engines to include it. The Oasis sidecar speaks the CLIMADA sidecar's
/v1/impact wire format through the shared client
worker/models/sidecar_protocol.py.
GET /v1/risk/lookup — point score card (issue #382, phase 8.8)¶
The answer plane's read path: no job, no engine, no compute. For each hazard it
reads the org-scoped risk cell covering the coordinate (falling back to the
org-less reference plane), reads the hazard COG's intensity at the point through
titiler /cog/point, and bands the result via the versioned score schemes
(ADR-043). Lat/lon only — address resolution is the geocoder's job.
Contract points that callers depend on:
- Versioned payload.
schema_version(currently1.0) is bumped on any breaking change; additive fields do not bump it. - The card says where the pin is (issue #1320). Beside
query, the payload carrieslocation: {"country_iso3": str | null, "country_name": str | null}— the admin-0 country the coordinate falls in, resolved by one boundedST_Intersectsprobe against the loaded GADMadmin_boundaries— two scalar columns andLIMIT 1, no geometry projected into Python, reported aselapsed_ms.location, and issued once per card, not once per peril. Both fields are null together when no loaded boundary covers the point, which is an absent answer — "we have loaded no boundary here" — and never a claim that the coordinate is in no country. This is not a geocoder call and adds no provider dependency (ADR-044 stands: nothing reverse-geocodes); the polygons are already loaded and the question is a spatial predicate on them. It is also distinct from theregionquery parameter, which is a filter on surfaces and datasets, echoed unchanged underquery.regionand never derived — the reason a header that renderedquery.regionnamed nothing for every pin, since the UI sends none. Additive, soschema_versionis unchanged. - The card names the admin units that contain the pin (issue #1668, run
model § 8). The same
locationblock carriescontaining_units: [{"admin_level": int, "boundary_id": uuid, "name": str}]: one entry per loaded admin level whose unit contains the pin, ordered byadmin_levelascending and starting with the countrycountry_iso3names. Each entry'sboundary_idandnameare the onesGET /v1/admin-boundariesserves for that unit, so a client can clip a run to the deepest entry (the pin's municipality where level 2 is loaded) without a second read. It is[]exactly when the country fields are null. The probe above becameAdminBoundaryRepository.units_at_pointand is still one statement per card: four scalar columns, at most one row per level. The country is decided first (iso3, thenname), and every deeper level answers for that country only, first byname, so a pin on the edge two units share names the same one every time. No route and no geocoder call is added. Additive, soschema_versionis unchanged. - A hazard the scenario does not change answers a future stop as today
(issue #1629, run model § 3.4). Each entry carries
provenance.hazard_conditioning, which is"not_climate_driven"ornull. It is set when all of these hold: the request names a scenario other thanbaseline;location.country_iso3is not null; no dataset of the hazard in that country, and none covering the pin, declares a scenario other thanbaselineorhistorical; and the hazard holds a dataset today. Earthquake is the case. The entry is then the one the unqualified lookup returns: the same dataset, read at no horizon, with the same band. Before this, abaseline-labelled earthquake dataset dropped out of anssp2-4.5question and the entry readno_hazard_dataset. The verdict iscore.hazard_conditioning.is_climate_driven, the predicate behindclimate_drivenonGET /v1/catalog/coverageand the compute resolver, and the key is the one a cell-addressed run records. It isnullon a projection, on every entry of abaselineor unqualified request, when the pin's country is unknown, and on a peril pinned withdataset_id. A climate-driven hazard with no dataset at the asked year is unchanged: it answers from an undated dataset or reads unavailable, and carriesnull. The rule costs up to two catalog reads per card (today's datasets at the pin, then the scenario labels of the hazards that hold one, asked once for all of them), reported aselapsed_ms.conditioning; on a scenario request those and the boundary probe run ahead of the COG stage. Additive, soschema_versionis unchanged. - No silent omission (RFC amendment A3.6). A hazard with no surface and no
readable COG is returned with
available: false, a machine-readableunavailable_reason, andprovenance.surface.surface == "not_built_here". The response is200— "nothing was built here" is an answer, not an error. - Band precedence (RFC amendment A3.3, narrowed by ADR-073).
Every wired peril bands from the COG point intensity — at the design return
period (
RISK_LOOKUP_BAND_RETURN_PERIOD, default 100 y) on a return-period raster, or at the resolved scenario × year slice on a cube. Windstorm, tropical cyclone and heatwave were classified to the cell metric until #924: they are indicator perils with no seeded damage curve, so no impact run and therefore no cellmetricsare ever written for them, and every one of those ladders was dead. There is no cross-source fallback either way. - Dataset selection is stated, and overridable (issue #590, ADR-056). With
several datasets catalogued per peril, each peril's dataset is chosen by
quality score → local-over-global → recency, not by insertion order, and
provenance.dataset.selectionreports the score, the locality rank and the rule that decided. An optionaldataset_id(accepted on all four/v1/risk/*rungs) pins one dataset for its own peril only; every other peril still resolves by precedence. Adataset_idthat is not visible to the caller's org is a404; one whose peril is not amonghazardsis a422. - Coverage before answerability — rule 2b (issue #1047, ADR-079). Before
precedence ever ranks a peril's candidates, the catalogue query is narrowed
to datasets whose ingested footprint covers the pin (
ST_Intersectsagainst the GiST-indexedhazard_datasets.footprint, migration 0075); a dataset with no recorded footprint (every dataset ingested before #596, never backfilled) still competes — an unrecorded extent is not a claim the dataset does not cover this point. This is what stops a smaller-extent dataset of the same peril elsewhere on the map — e.g. an ingested Cyprus raster — from shadowing a Greek pin under the extent tie-break above: the non-covering dataset never reaches precedence. An explicitdataset_idpin is unaffected: it is resolved directly by id, not through this filtered pool, so it still answers (or states why it could not) even outside its own footprint. - Feature-flag aware. With
SCORE_BANDS_ENABLEDoff the structure is unchanged and every continuous metric is still served; onlybandisnull, withband_unavailable_reason: "score_bands_disabled". - Bounded band read (issue #838). A titiler point read costs one internal
tile fetch per band, so the read asks for only the bands the card can use, via
bidx. A return-period vector no deeper thanRISK_LOOKUP_MAX_POINT_BANDS(default 16; the deepest catalogued raster has 9) is read whole and publishes its fullby_return_periodcurve; a deeper one narrows to the design return period's band; a dataset with no return-period vector (ADR-067's scenario × year slice axis) reads the slicehorizonaddresses, or the first band when it addresses none. Labels follow the band index read, never the position in the response. - Every reading names the raster band it came from (issue #964). Each
intensity.by_return_period[…]entry carriesband_index, the 1-based band the value was read under, andintensity.band_indexnames the band behind the card's ownband_value(nullexactly whenband_valueis). That is thebidxofGET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png, so a client can overlay the raster for the return period a card describes instead of inferring the band from a position — an inference that is correct on a whole-stack read and wrong on every narrowed one, and whose failure mode is a map showing a different return period under the right label. It generalisesslice.band, which answers the same question for a scenario × year cube only. Additive, soschema_versionis unchanged. Abidxpast the COG's real band count is now422 E_VALIDATIONwithdetails.band_countrather than a502 UPSTREAM_TILE_ERROR; the count is read from the raster header via titiler/cog/info, never from the dataset's declaredreturn_periods(drift between the two is tolerated elsewhere and must not become fatal here), band 1 is never probed, and a count that cannot be established fails open to the previous 502 behaviour. - A raster overlay can carry a real legend (issue #965). The tile route
additionally accepts
rescale=min,maxandcolormap_name=<ramp>as validated pass-throughs, andGET /v1/hazards/{dataset_id}/layerspublishes alegendblock on the rasterLayerSpec—domain,colormap_name,unitand the per-band statistics behind them — so the two fields a client passes back are the two fields it was given, and no domain is invented client-side. Absentrescale, titiler stretches each tile to its own pixels, so no single legend can describe the map. The numbers are measured from the COG through titiler/cog/statisticsand memoised per COG, never recorded at ingest: the seeded datasets already exist, the COG is re-tiled from the source so an ingest-time min/max can disagree with what renders, and a stored column would need an invalidation hook a TTL does not. The read fails open (legend: null) and is never on a tile-render or point-lookup path. Every rendering parameter also joins the tile cache key, or one caller's stretch would be served to the next — see Raster Serving. - A tile the raster does not cover is empty, not an error (issue #1155).
GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.pngoutside a dataset's footprint answers200 image/pngwith a fully transparent 256 × 256 tile instead of502 UPSTREAM_TILE_ERROR. A map viewport is wider than a footprint, so panning around a small event snapshot asks for such tiles constantly, and a 502 made MapLibre retry each one and counted ordinary interaction as an API error. Only titiler'sTileOutsideBounds404 qualifies — a 404 that means "no such route" (#831's shape) and every other upstream error still map to502— and the empty tile is never written to the tile cache. See Raster Serving. - A hazard raster can speak the score card's scale (issue #1372, Phase 16
D10).
GET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.png?classed=scorerenders the band in the five classes of the dataset's active score scheme instead of a continuous ramp: the scheme's tenlower/upperbands merged in pairs (scores 1-2, 3-4, 5-6, 7-8, 9-10). With no ramp named, the classes take the UI'sRISK_LEVELSpalette, Ember's five steps from sand to plum and colour-blind-safe (#f6e7b0,#d9a13d,#d26533,#a33338,#4d1442, restated incore/score_classes.py::SCORE_CLASS_COLORS, issue #1584). The backbone mirrors the UI: a change toRISK_LEVELSor Ember's stops there is copied here. The scheme is selected exactly as the Lookup card selects it for a resolved dataset — peril, the definition-keyedmetricfromdata_type, and the dataset'sintensity_unit.scoreis the only accepted value; any other is422 E_VALIDATION, and so isclassedcombined withrescale(titiler stretches before applying a colormap, which would move values off the unit the class edges are written in). The caller's value never reaches titiler: the proxy mints titiler's intervalcolormap,[[[lower, upper], [r, g, b, 255]], …], whose open tails are written as∓3.4028234663852886e38(the finite float32 range — hazard COGs are float32, andInfinityis not JSON). The interval form was verified against a local titiler 0.19.2 with rio-tiler 7.0.0 and 7.9.6: each class paints its colour over[lower, upper)and nodata stays transparent. The minted map is part of the upstream query and therefore of the cache token, so a scheme recalibration is a new cache entry rather than a stale tile. A dataset whose peril has no scheme for its unit, or a deployment withscore_bands_enabledoff, answers exactly as withoutclassed— same upstream URL, same cache key.GET /v1/hazards/{dataset_id}/layerspublishes the same list aslegend.classes:[{"lower", "upper", "label", "color"}], lowest first, the firstlowerand the lastuppernull(open tails, as in the scheme), labels built from the scheme's band labels ("Negligible to Very low (scores 1-2)"). It isnullin exactly the cases?classed=scorerenders unclassed, and it rides a measured legend, so a failed statistics read (legend: null) hides it too. Namedclassesbecauselegend.bandsalready carries the per-band statistics. - Classed score tiles follow the user's palette (issue #1475, Phase 17
D13).
?classed=score&colormap_name=<ramp>keeps the class breaks exactly as above and recolours only the classes: the ramp's 256-entry rio-tiler table is read from titiler'sGET /colorMaps/{ramp}(memoised for an hour) and sampled at the class count — the first class takes index 0, the last index 255, the rest evenly between (0, 64, 128, 191, 255for five;core/tile_render.py::ramp_positions). OnlyCOLORMAP_NAMESentries (and their_rvariants) are accepted; an unknown ramp is422 E_VALIDATION, andrescalewithclassedstays422. The ramp name is not forwarded upstream on a classed tile (titiler would prefer it over the mintedcolormapand drop the classes); it is folded into the cache token instead, so two ramps never share a cached tile. A ramp titiler cannot serve is502 UPSTREAM_TILE_ERROR. With no scheme the request is the plain?colormap_name=<ramp>tile, key included.GET /v1/hazards/{dataset_id}/layers?colormap_name=<ramp>colourslegend.classesby the same call (core/score_classes.py::ramp_score_classes), so swatch and pixel match colour for colour;legendisnullwhen the ramp cannot be read. - A cube states which slice answered (issue #844). On a scenario × year
dataset,
horizonis resolved to a band through that dataset's ownhazard_eventsrows —event_index + 1is the raster band carrying that event's year, so the mapping is a join, not an inference fromsupported_years. The answering entry carries asliceblock namingyear,scenario(the event's label, else the dataset's — ADR-071 puts it on the dataset row), the event'sevent_nameand thebandread, and the sample is labelled with thatevent_namerather than a bare band number. A horizon the events do not carry is refused, not substituted:unavailable_reason: "horizon_not_in_dataset"(or"horizon_ambiguous"when two events claim one year) withslice.available_yearsnaming what the dataset can answer. Before this,horizon=2050against the 1986-2085 heatwave cube returned the 1986 band's value with nothing saying so. Every other dataset shape is untouched — a return-period raster's events carry noyear, so it is never addressed and itssliceisnull. - A single unlabelled field bands as its only slice (issue #1630,
ADR-068). A dataset whose source declared no non-spatial axis
holds one
hazard_eventsrow with no return period, no scenario and no year (the EWDS historical fire-weather index). It has no year forhorizonto address and no other band, so band 1 is its whole answer: the card bands that value for every request, unqualified or at a future stop, withband_unavailable_reason: nullandslice: null. The fact is read from the events (HazardRepository.single_field_dataset_ids): the dataset must hold that one row and no other, so a scenario × year cube read without a horizon still reportsno_bandable_readingand is never banded from band 1. One query for the whole card, over the chosen slice-axis datasets only, inside the COG stage's window and counted inelapsed_ms.axes; a card of return-period rasters issues none. A field that declares nointensity_unitis served but not banded: with no unit to narrow by, the scheme tie-break would score a fire-weather index on the brightness-temperature ladder. A one-event field bands whatever its year label (issue #1854): prod's EWDS fire-weather datasets hold one event withyear 2023(event_name "2023", the year of the reading), no frequency and no scenario, and that event is still the dataset's whole answer. A scenario request that answers the field as today (hazard_conditioning: "not_climate_driven") bands it unaddressed; an unqualified request may address it at its year instead (slice.year 2023). An event that names a scenario is never a field, and a dataset of two or more events keeps #844's rule. - A timeout is its own reason. A read that exhausts
RISK_LOOKUP_COG_TIMEOUT_SECONDSreportsunavailable_reason: "cog_read_timeout", distinct fromcog_read_failed— one means the read is too big, the other that the raster or tile server is broken. indicatoris the compute gate's verdict, event axis included (issue #930). Each entry'sindicatoris decided the waycore.compute_service.validate_impact_refsdecides it — the dataset's intensity unit and its event axis — so a card can never advertise a runnable peril the Run button is then refused for. Before this the card read the unit alone, which is the same defect #920 fixed onGET /v1/hazards: a frequency-less cube with a consumable unit (the #797 class) showedindicator: falseand then422 E_HAZARD_DATASET_NO_EVENT_FREQUENCIES. An entry with no dataset still falls back to the peril-level question ("does the platform hold any curve for this peril?"), which is all it can answer. Cost: at most oneGROUP BY dataset_idaggregate for the whole card (never one probe per peril), skipped entirely when no chosen dataset's verdict the axis could move, and issued inside the COG stage's window rather than added to it — reported aselapsed_ms.axesso that stays checkable.- An absent band says which kind of absent (issue #933). When
bandisnull,band_unavailable_reasonis one ofscore_bands_disabled,no_catalogued_dataset,no_bandable_reading,no_scheme_for_hazard,band_metric_unavailableorvalue_unbandable. The first two of those matter most to branch on, and used to be one string:no_catalogued_datasetmeans nothing is catalogued for the peril, so no coordinate, return period or horizon can ever band it and a retry is futile;no_bandable_readingmeans a dataset answered and this address has no bandable value in it — a return-period raster that does not publishRISK_LOOKUP_BAND_RETURN_PERIOD, or a scenario × year cube whose slicehorizondid not address — so another query may well succeed. Reporting both asband_metric_unavailabletold a caller "missing data for your query" about a peril that has no data at all.tropical_cycloneandstorm_europeare the live examples: both carry seeded ladders and neither has a catalogued dataset, so both reportno_catalogued_datasettoday. That is read off whether this request resolved a dataset, never off a table of which perils have data — either peril can be catalogued throughPOST /v1/hazards/ingestor/v1/hazards/uploadwithout a seed pack, and its reason changes on the next request when it is. Adding a reason is additive and does not bumpschema_version; treat an unrecognised code as opaque rather than failing on it. - A count indicator's absent reading is the value zero (issue #935,
ADR-074). The platform stores only the cells worth storing, so
a heat-wave COG carries
nodata = 0.0and titiler answersnullat a pixel that recorded no heat-wave day. For a count index that null is a number — "no heat-wave day here that year" is0 days— so on the perils listed incore/scoring.py's_ABSENT_READS_AS_ZERO(heatwave today) a resolved slice with no stored value bands as the ladder's Negligible rung instead of reportingno_bandable_reading. Without it, band 1 of every count ladder was unreachable by construction and a reference-period year answered "band unavailable" over ~95% of the ECDE grid. Two consequences a client must read correctly: intensity.band_value_sourcesays"read"or"absent_as_zero"(nullwhen there was no band input at all), and it is the authoritative marker.band.metric_sourcenames the input class the scheme bands from and still reportspoint_intensityhere — it is a property of the ladder, not a claim that a pixel answered.- The raw reading is not rewritten:
intensity.band_valueis0.0whileintensity.by_return_period[…].valuestaysnull. That asymmetry is deliberate — the convention supplies a band input, it does not invent a measurement.
Continuous-intensity perils are not in the table: an absent flood depth
is dry land outside the footprint, and the stated non-answer is still the
honest card there. (The retired /v1/risk/assets/{id} rung projected the card
verbatim and so inherited the band, an aggregate_value of 0.0 and a
matching provenance.band_value_source; since ADR-080 it answers 410 and
the convention reaches a client only through /v1/risk/lookup.) The portfolio
and admin-unit rungs read cell aggregates and are unaffected.
- The climatological heat-wave ladder bands at version 2 (issue #925,
migration 0086). heatwave/heatwave_days_climatological uses edges
[3, 4, 5, 7, 10, 14, 20, 28, 40]; the ECDE index counts days in a run of at
least three consecutive days, so values of 1 and 2 cannot exist and band 2
([3, 4)) is exactly one minimum-length event, while band 1 is the Negligible
rung a genuine zero-day reading lands in under the convention above. The
health ladder (heatwave_days_health) stays at version 1 with edges
[2, 5, 10, 15, 20, 30, 45, 60, 90] — a June-August window caps that index at
92 days, so its >= 90 band is an intentional physical-cap sentinel. Both
sets are self-set: no published banding exists for either index, and each
row's citation says so and names what the edges were derived from. Clients
must read band.scheme_version (surfaced as scheme_ref, e.g.
heatwave/heatwave_days_climatological@v2) rather than assume a fixed ladder
— a stored band keeps the version it was computed under and is not re-banded
retroactively.
- Stated latency budget. 100 ms cells (indexed) + 1200 ms COG (one round
trip per hazard, issued concurrently) + 50 ms assembly, 1500 ms p95 end to
end. Echoed as budget_ms, measured per request as elapsed_ms (conditioning,
slices, axes, location, cells, cog, assembly — axes, location
and cells run inside the COG stage's window rather than adding to it, except
that a scenario request runs location and conditioning ahead of it, issue
#1629), and exported as the
climate_lama_risk_lookup_seconds histogram. Asserted in
tests/test_api/test_risk_lookup.py.
GET /v1/risk/admin-units/{boundary_id}/impact — a unit's impact per hazard, from saved runs (issue #1684, LK-15)¶
Lookup's "This area" shows a municipality's expected annual impact per hazard beside the score
rollup of GET /v1/risk/admin-units/{boundary_id}, which this route leaves unchanged. It is a
read of what saved runs already store (result_admin_cells, one total per GADM unit per result,
summed by each asset's own coordinates): nothing is computed, no job is queued, and no entry or
field adds two hazards or two exposures (K26). It takes no query parameter.
{
"boundary_id": "…",
"admin_level": 2,
"hazards": [
{
"haz_type": "river_flood",
"result_id": "…",
"scenario": "baseline",
"points": 7,
"eai": 110.0,
"value_unit": "EUR",
"exposure_dataset_id": "…",
"exposure_name": "Attica buildings",
"exposure_type": "buildings",
"computed_at": "2026-10-01T00:10:00+00:00"
}
],
"buildings_in_data": 7
}
- One stored result stands for each hazard type: a Today run (cell
baseline@today, nomeasure_ids, noexposure_growth_rate) of the caller's organisation that covers the whole unit and holds aresult_admin_cellsrow for it. A run covers the whole unit when it is unclipped, or itsscope.admin_boundary_idsnames the unit or one of its ancestors. An ancestor is a unit reached throughadmin_boundaries.parent_id, or a coarser unit of the same country whose polygon covers a point on the unit's surface (AdminBoundaryRepository.lineage_ids): the boundary loader's single-feature mode writes noparent_id, so the link alone would miss the seed's units. A run clipped to other units only, and any run clipped by abbox, is never read. Of the results that qualify, a one-dataset run over the country's default exposure layer (default_exposure_dataset, EX-18) stands first, then the newest bycreated_at. The results of several exposures are never summed, because a portfolio and an exposure layer over the same ground would count the same buildings twice. eai,pointsare the chosen result'sresult_admin_cellsrow for the unit at itsadmin_level.value_unitis the result's unit (K27).scenariois alwaysbaseline.computed_atis the result'screated_at.exposure_dataset_idnames the dataset of a one-dataset run and is null for a portfolio run and for a run over several datasets.exposure_nameis the dataset's or the portfolio's name (the names joined by,for several datasets).exposure_typeis the exposure type of the run's locations, null when they hold several: the keys of the result's stampedimpact_by_exposure_type, and for a result written before that stamp the declared type of the datasets its run named.buildings_in_datais thepointsof the entry over the country's default exposure layer, else of the newest entry, and null whenhazardsis empty. It is one entry's count, never a sum.- A hazard with no qualifying result has no entry. A unit no stored result covers, and a unit at
a GADM level the result cells are not written at (
result_cell_admin_levels), answershazards: []withbuildings_in_data: nullon a200. Aboundary_idthat is not loaded → 404E_NOT_FOUND, as the rollup route answers.
The choice is one statement (ResultCellRepository.today_results_of_unit), each run's inputs
read off its job's params through impact_results.job_id; the rule lives in
core/admin_unit_impact.py. Asserted in tests/test_api/test_admin_unit_impact.py.
Rollup reason codes — /v1/risk/{portfolios,admin-units}/{id}¶
The rollup rungs report an unaggregated hazard the way the point card
does: available: false plus a machine-readable unavailable_reason, on a
200. (There were three rungs; /v1/risk/assets/{id} is retired and answers
410 — see the endpoint table above and ADR-080. Nothing in this section
applies to it, because it no longer answers anything.) This is the rollup's own
reason set, distinct from the card's
band_unavailable_reason above — that one explains a missing band on a
hazard that did answer. Five codes, and they are not interchangeable:
no_members_to_aggregate— nothing was in scope to aggregate over.no_cells_in_scope— members (or a boundary) were considered, and no answer-plane cell covers any of them, and no readable surface exists for this key at all. Nothing was built here; run the scenario.no_cell_at_resolution— a readable surface is built for this hazard, scenario and horizon, and no member's cell is in it, at the lookup resolution or at the parent the fallback also tried (issue #1154). Until #1154 the portfolio rung answeredno_cells_in_scopehere too, which is what made the UI say "No risk surface in scope" over a surface that was demonstrably present — a day of read-only forensics on issue #1048.
Read this code for exactly what it says. The probe behind it
(RiskSurfaceRepository.has_ready_surface_for_key) is region-agnostic
unless the caller pinned region. Pinned, it is precise: a surface covers
your area and not these coordinates, so the fix is an exposure that reaches
the members. Unpinned — the default for GET /v1/risk/portfolios/{id} — it
says only "something readable answers for this peril and scenario, and it is
not over you", and that something may have been built over another country
entirely. Check region before concluding the exposure is at fault rather
than the surface never having been built here.
- metric_absent_from_cells — cells were found, and none of them carries a
value for this scheme's metric. A cell that exists but carries no value for
this metric is a different operator signal from no cell at all: the surface
was built, just not with this scheme's metric on it. The portfolio rung
states this exactly as the admin-unit rung does (issue #937); before that it
answered no_cells_in_scope while sitting on a full set of cells, which
points at building a surface that already exists.
- no_scheme_for_hazard — no versioned score scheme names a metric for this
peril, so there is nothing to aggregate.
Rollup payload — what aggregate_value and worst_value are¶
Both numbers, and the bands derived from them, come from one place: the
metrics blob of the answer-plane cells in scope, read under the active
scheme's metric key. Until issue #932 no production writer produced any
scheme's key, so every rollup reported metric_absent_from_cells for every
peril. ADR-075 closes that on the run path:
- What the cell carries. A run-backed surface now stores, per cell, the
mean design-return-period hazard intensity over the exposure points inside
that cell, plus
<metric>_points— how many valued points that mean was taken over. Both are written bycore/surface_writer.build_cells; a cell none of whose points had an intensity gets neither key, sometric_absent_from_cellskeeps meaning exactly what it says. - It is the hazard where the book is. A surface covers exposure locations, not a grid, so this is a book-conditioned aggregate — the same conditioning the cell's EAI already carries. An unconditioned hazard field for a district is a different question and a different (deferred) build.
- Portfolio rung —
aggregate_valueis the exposed-value-weighted mean of the members' cell values,worst_valuethe largest. Unchanged by #932, and still free of COG reads. - Admin-unit rung —
aggregate_valueis the points-weighted mean over the cells whose centroid falls inside the boundary, weighting each cell by its<metric>_points(a cell with no such count weighs 1, which is right for a legacy or hand-written cell).worst_valueis the largest cell average, not the worst point.provenance.aggregationsayspoints_weighted_cell_mean. - One surface per hex. The boundary aggregate pins neither
regionnor the engine's identity, so several READY surfaces can satisfy one lookup key — production holds six for one river-flood identity. The aggregate now picks the newest surface perh3_indexbefore aggregating, reported asprovenance.surface_selection = "newest_surface_per_h3_index". - Which perils this reaches. Only perils an impact run can be computed for,
whose dataset publishes an event at that peril's design return period and
whose
intensity_unitmatches the scheme'smetric_unit. River flood bands at the globalRISK_LOOKUP_BAND_RETURN_PERIOD(default 100). Earthquake bands at 476, not the global default:core/scoring.py'sdesign_return_period_foris the single resolver both the address-level card and this writer call, and its per-hazard override table namesHazardType.EARTHQUAKE: 476.0— the catalogued ESHM20 dataset publishes nominal return periods[50, 476, 976, 2500, 5000]and no 100-year event, so the global default could never match (issue #942). 476 is ESHM20's own nominal label for the ~475-year (10%-in-50-years) reference seismic design return period, matched exactly, not within a tolerance. Consequence for clients:intensity.band_return_periodis hazard-varying (100 on a flood card, 476 on an earthquake card,nullon a cube slice) — it is provenance about which slice answered, and a client should not assume one fixed value across perils. Heat-wave and wildfire/FWI are indicator perils with no run at all, and stay unaggregated pending issue #944. - Surfaces built before #932 carry no metric. Cells are replaced in place, so re-running a scenario is what fills them in; a bulk backfill is issue #943.
/v1/risk/lookup'scell.metricsgains the same two keys, additively, for surfaces written after this lands.eai,eai_density_km2,pointsand any*_pointskey are a reserved namespace no score scheme may claim.
Portfolio members — which cell answered, and why one did not (issue #1154)¶
GET /v1/risk/portfolios/{id} reports its membership through two lists, and
every entry in either is one MemberScore, so both carry the same keys.
worst_members— the concentration list, members that scored, ordered by combined score then exposed value thenmember_id.unavailable_members— the members that produced no score, ordered by exposed value descending so the largest unanswered exposure reads first. Capped atROLLUP_WORST_Nlike the list above;subject.unscored_member_countis the count it samples from, so a trimmed list is never mistaken for the whole of what is dark. Before #1154 this list did not exist andworst_members— being a ranking by score — could not hold an unscored member, so a book that scored nothing rendered as an empty list with no per-member explanation anywhere in the payload.
unscored_member_count counts the members this request weighed and could
not score — exactly coverage.considered - coverage.contributing. On a book
larger than ROLLUP_MAX_PORTFOLIO_MEMBERS it is therefore not
member_count - scored_member_count: member_count is the book's true size,
and the members past the cap were never loaded, so this request says nothing
about them. truncated: true is what states that.
Two fields on every member say which cell the score came from:
cell_source—exact_cellwhen the member's own H3 cell atRISK_LOOKUP_RESOLUTION(r8) answered,parent_cellwhen it was absent and the immediate parent answered instead.nullon an unscored member.cell_resolution— the H3 resolution that cell was read at (8 or 7 under the defaultSURFACE_WRITE_RESOLUTIONS).nullon an unscored member.
Both describe the driving hazard's cell — the one behind combined_score —
so a member banded off its own cell for one peril and a parent cell for another
is described by whichever produced the headline. At hazard level,
provenance.members_scored_from_parent_cell counts how much of the aggregate
rests on the coarser answer; it is always present, 0 included, so a client
never has to tell "none" from "this build cannot say".
Why a parent cell is consulted at all. A surface carries a cell only where
the exposure dataset that built it had a point, so a member a few hundred metres
from the nearest of those points has no r8 cell of its own — even where the
surface's r7 parent holds a perfectly good value for the scheme's metric. That
is a resolution artefact, not an absence of risk, and until #1154 it was reported
as one: on production a Cyprus LitPop member scored against the two LitPop-built
surfaces and against nothing GHSL built, while its r7 parent
872da449affffff carried flood_depth_m 0.249 in every GHSL-built surface.
The rollup therefore asks twice, in order: the member's own cell first — always
preferred, nothing coarser consulted while it answers — then, only for the
cells that missed, the member's immediate H3 parent. That is the finest
resolution below the lookup one that the writer actually builds
(SURFACE_WRITE_RESOLUTIONS) — r7 under the default [7, 8], and still r7
under [6, 7, 8], because the least coarsening that can answer is the one to
take. One step only: a surface built at r6 and not r7 stays a miss, and a writer
list with no entry below the lookup resolution means no fallback and no wasted
query at all.
The second batch is bounded by the misses, so it costs at most one extra
round trip per plane per hazard and the rung's fixed-query-count guarantee is
unchanged — asserted by
tests/test_api/test_risk_rollups.py::test_the_fallback_does_not_scale_the_query_count_with_the_book.
Its payload is a different matter: in the case #1154 was filed for, where every
member is outside the surface, misses equal membership and the parent batch is
membership-sized too. What is flat is the number of queries, never the number of
cells asked about — which was already true of the exact batch.
This is the rollup's rule, not a change to the read path.
core/surface_service still answers at exactly the resolution it was asked for
and still refuses to coarsen silently; the fallback is a second, explicit
question the rollup asks at a resolution it names. The address-level card
(GET /v1/risk/lookup) and the admin-unit rung are unaffected.
Indicator hazards — a typical score, outside the portfolio band (issue #1665)¶
An indicator dataset (an FWI wildfire pack, a heatwave-days cube) writes no
cell, so the cell-reading rollup reports its hazard unavailable. Each hazard
entry of GET /v1/risk/portfolios/{id} therefore also carries
indicator_score:
{band, label, metric_unit, scheme_ref, hazard_dataset_id, band_index, members_sampled, in_portfolio_band}
- What it reads. Per hazard, the indicator dataset (a COG and a unit no seeded curve consumes) whose bbox holds a member is picked by the Lookup's rule under the request's scenario and horizon (an exact scenario label first, then ADR-056 precedence). A cube answers only from the slice the horizon resolves to, never its first band (#844).
- How.
core/indicator_sampling.pyopens that raster once and samples every capped member inside the open, inasyncio.to_thread, never one titiler round trip per member. The function takes bare(lon, lat, weight)points and the dataset, so an exposure selector's rows (#1664) score through it too. - The figure. Each reading is banded through the scheme of the dataset's
own
intensity_unit(FWI,days), andbandis the weighted mean of those bands rounded to the nearest band: members weigh their exposed value under one unit, 1 each when the book mixes units (asband_distributiondoes).members_sampledcounts the members that read a value. - The band it was read from (issue #1833).
band_indexis the 1-based raster band of the dataset the readings came from: 1 for a single-band dataset, the slice's band for a scenario × year cube, the design return period's band for a return-period raster. Pass it asbidxtoGET /v1/tiles/cog/{dataset_id}/{z}/{x}/{y}.pngto draw the slice that was scored rather than the first band. The stored result'sindicator_scoreonGET /v1/resultscarries it too. - What it never touches.
in_portfolio_bandis alwaysfalse. The headline,value_weighted.band_distributionandvalue_weighted.share_high_or_severestay over run-written cells only (K25), and no cross-hazard figure or EAI is derived from it (K26). - Null for a run-banded hazard (river flood keeps its
band), with score bands disabled, and where no indicator dataset covers the members for the requested scenario and horizon.elapsed_ms.indicatorstimes the stage.
GET /v1/jobs — the org's run history (issue #1184)¶
The list side of a resource that until now only answered per id, so the Home
dashboard's Recent runs card and the Runs hub have something to read. Role
floor is VIEWER, matching GET /v1/jobs/{id}.
| Param | Meaning |
|---|---|
status |
Optional JobStatus (pending/running/completed/failed/cancelled), repeatable (issue #1662): ?status=pending&status=running lists the jobs in either state and total counts both, so the top bar's pill reads every job in flight in one call; one status answers as it always has. An unrecognised value is a 422, never a silently-empty page |
job_type |
Optional JobType (impact_calc/data_ingest/cost_benefit_calc/report_render/score_delta). Combines with status; both unset lists the whole org |
include_automated |
Include feed-driven jobs (origin='feed'), default false (issue #1287). A near-real-time watch enqueues one compute per snapshot, so leaving them in would bury a person's own runs on the very card this endpoint fills. The exclusion applies to total as well as to the page, so a pager sized from the first request agrees with the pages it walks |
limit |
Page size, 1-200, default 50 |
offset |
Rows skipped, default 0 |
Response is {"items": [...], "total": n, "impact_workers": k} inside the
standard envelope, where total counts the whole filtered set, not the
page, so a client can size a pager from the first request. impact_workers
(int or null, issue #1662) is the impact worker slots the deployment runs,
read from the setting impact_worker_concurrency (env
IMPACT_WORKER_CONCURRENCY), the same value prod's compose feeds the
impact-worker's --concurrency; it is null when the variable is unset (dev
and smoke workers run Celery's default concurrency, which the API cannot know),
so History draws "created_at descending with
id descending as a tiebreak — jobs.created_at defaults to now(), which in
PostgreSQL is transaction start time, so every cell of a scenario matrix
staged by POST /v1/compute/impact/matrix shares one timestamp and would
otherwise page non-deterministically.
Each row is {id, status, job_type, origin, error_code, error_message,
result_id, portfolio_id, exposure_dataset_ids, hazard_dataset_id, haz_type,
scenario_label, horizon_year, country_iso3, created_by_user_id,
created_by_email, ingest_job_id, cancellable, step, created_at, started_at,
completed_at, title} — the per-id shape
minus celery_task_id (a poller's handle on one job, not dashboard-card
material), plus title.
cancellable and ingest_job_id (issue #1654, SH-16) say whether the row
offers Cancel and through which route, on this list and on GET /v1/jobs/{id}
alike (rule: core.job_dispatch.job_read_state). A pending impact_calc job
is cancellable: true through POST /v1/jobs/{id}/cancel. A hazard ingest's
data_ingest job serves the ingest_job_id its params carry and is
cancellable: true while that ingest_jobs row is not terminal; it is
cancelled through POST /v1/hazards/ingest-jobs/{ingest_job_id}/cancel, which
writes the ingest lane only (#1237). Once that row reads cancelled the job is
served status: "cancelled" — derived at read time, never written to
jobs.status — and the status filter matches the served value, so
?status=running stops listing it and ?status=cancelled lists it. Every other
row (the centroid-assignment data_ingest job included, which has no
ingest_job_id) is cancellable: false with ingest_job_id: null.
created_by_user_id and created_by_email (nullable, issue #1662) name
who started the job, on this list and on GET /v1/jobs/{id} alike. The id is
read off the job's params — user_id, or owner_user_id for an upload ingest —
and is null for a feed job (even when its params name the watch's creator) and
for a job whose params name no user, so the pill tells the viewer's own jobs
from the organisation's. The e-mail is that user's in the caller's organisation
(users has no name column), null when the id is or the user is gone. origin is "user" or "feed"; it
is present on every row, including the default page, so a client that opted
the feed rows back in with include_automated=true can label them without a
second call.
portfolio_id (nullable, issue #1407) is the portfolio a compute job was
addressed at, when it was — the source climate-lama-ui#377 needed for its
Recent-runs fallback link: to /assess?result=<id> when the job has a
result, else to its portfolio. Since issue #1901 the per-id
GET /v1/jobs/{id} serves it too. It is read off Job.params["portfolio_id"]
(the same key POST /v1/compute/impact and the live-events compute route
write) via core.eai_alignment.exposure_selector_from_job_params, which
tolerates a missing key, a None value, and an unparseable one — all three
read back as null, never a 500.
Run inputs (nullable, issue #1901) — exposure_dataset_ids,
hazard_dataset_id, haz_type, scenario_label, horizon_year and
country_iso3 — say what an impact_calc job ran over, on this list and on
GET /v1/jobs/{id} alike, so History's failed card (climate-lama-ui#845)
can offer "Open this run" and "Re-run" for a run that has no result to read
them from. With portfolio_id they are the run's exposure selector:
exposure_dataset_ids (one id, or several for a union) for a dataset run,
portfolio_id for a portfolio run. hazard_dataset_id and the cell's
scenario_label + horizon_year (null for a run addressed by dataset alone)
are read off Job.params. haz_type is that hazard dataset's type, and
country_iso3 the ISO3 country GET /v1/exposure-datasets derives for the
run's exposure dataset (for a union, only when every member derives the same
one) — the country whose default_dataset_id "Open this run" opens. Both are
read for the whole page in one query each, and are null once the dataset is
deleted, for a portfolio run or a mixed-country union (country_iso3), and
when the country is not derivable. Every run input is null when the params do not
name it and on every job that is not impact_calc — a hazard ingest's
haz_type or an exposure upload's country is what it ingests, not a run
input. Rule: core.eai_alignment.run_inputs_from_job_params.
title is derived server-side from the row's own params — no joins, no
id resolution, no extra query per page — so the UI never re-derives it
(guardrail 8, dependencies flow downward). The ladder, most specific rung
first: params['action'] where it names a narrower operation than the type
does (assign_centroids → "Centroid assignment"); params['name'], the
dataset name both hazard ingest lanes stamp ("Data ingest — Attica flood");
a matrix cell's scenario_label/horizon_year ("Impact calculation —
ssp2-4.5, 2030"); a report's template; a cost-benefit run's
present_year–future_year window; and finally the job type's own label,
bare. It is never null and never empty — the compute lanes stamp only
opaque ids for the hazard, exposure and portfolio, so "Impact calculation" is
the honest answer for many rows, and a blank line on a card is worse than a
generic one. Titles are clipped to 200 characters.
POST /v1/users/me/notifications/read-all — clearing the bell (issue #1187)¶
Notifications could be read only one at a time, so an inbox never got back to zero: production held 69 unread rows for one user on 2026-09-03. This route clears them all.
count is how many rows this call actually flipped — the caller's own rows
that were not read and not dismissed — so a client can reconcile its badge
without a follow-up list request. The predicate is the badge's own
(NotificationRepository.unread_count), and it is applied in a single
UPDATE, never a loop over the per-row route.
Scoping is exactly the per-row read route's: the caller's user_id inside
the caller's org, enforced by RLS on the outside and the repository predicate
on the inside. There is no role floor beyond being authenticated — clearing
your own bell is not a privileged act — and no way to clear anyone else's.
Idempotent by predicate, not by convention. Already-read rows are excluded
rather than re-stamped, so a second immediate call returns {"count": 0} and
every original read_at timestamp survives it.
Notification payloads carry a title (issue #1187)¶
The bell renders payload.message ?? payload.title, but the payloads the
worker emitted carried only ids — a job_complete was two UUIDs and a
dataset_available was one. Every type a user is told about by name now
carries a human-readable title, resolved at emit time from
core/job_titles.py and never derived client-side:
| Type | title |
|---|---|
job_complete |
The same string GET /v1/jobs shows for that row. The worker is handed the very dict persisted to jobs.params, so the two surfaces cannot disagree. Where the ladder bottoms out on the bare type label — the compute lanes stamp only opaque ids — the name of the dataset the job read is appended: "Impact calculation — Attica flood 2026". A resolved rung is never overridden, so the twelve cells of a scenario matrix keep the twelve distinct "Impact calculation — ssp2-4.5, 2030" titles that tell them apart |
dataset_available |
The dataset's own name ("GADM Greece admin-1") |
job_failed |
The same name again, plus what became of the job: "Impact calculation failed", and "Impact calculation — ssp2-4.5, 2030 failed" where the row's params name the specific run |
title is never null and never empty on any of the three; resolution failure
degrades to the job type's label rather than dropping the key. Emission stays
best-effort throughout (ADR-037) — a title that cannot be resolved must never
cost a user their notification, let alone the job that earned it.
One resolver, three surfaces (issue #1200). job_failed (issue #1182)
originally built its own title inline in
core.notifications.job_failed_payload, from
job_type.replace('_', ' ').capitalize() — so the bell said "Impact calc job
failed" about the very job GET /v1/jobs and its own job_complete
notification called "Impact calculation". It now derives the name from
core/job_titles.py like the other two, through derive_failure_title, and
appends failed. So a notification and the GET /v1/jobs row for the same
job do not disagree about what that job was called, on the failure path as
much as the success one.
The one place the two strings are not simply name and name + " failed" is
the 200-character cap. Both surfaces are capped, and the failure title reserves
the suffix's length before clipping the name, so a title long enough to hit the
cap — only reachable from a params['name'] of a few hundred characters —
truncates a few characters earlier than the jobs row's does and keeps failed
visible. Losing the verdict to make the truncation point match would be the
worse trade: the two still agree on everything up to the ellipsis.
The name is derived from job_type and the row's params, which every
jobs-backed failure lane already holds — JobRepository.mark_failed from the
row it loaded, mark_job_failed_sync from the SELECT that resolves the type,
the watchdog's bulk reap from its UPDATE ... RETURNING — so no lane pays a
query for it. IngestJobRepository.mark_failed is the one exception: it holds
an ingest_jobs row and no jobs.params, so its title stays at the type label
("Data ingest failed"). An unrecognised job_type degrades to a de-snake-cased
rendering of the string, and never raises: this code runs while a job is
already failing, where a second exception would mask the first.
GET /v1/orgs/me/branding — defaults instead of 404 (issue #1190)¶
Branding lives in the org's org_settings JSONB blob under a branding key. An
org that never configured any — every org on a fresh install — had no such
sub-dict, and both branding routes answered 404, so the first request a
booting client made failed by design and the client fell through to defaults it
had to guess at.
The two routes now differ on the unset case, and only on it:
| Route | Org has branding | Org has none | Another org's id |
|---|---|---|---|
GET /v1/orgs/me/branding |
200, the stored sub-dict | 200, the platform defaults | n/a — resolves to the caller's own org |
GET /v1/orgs/{org_id}/branding |
200, the stored sub-dict | 404 E_NOT_FOUND |
404 E_NOT_FOUND |
"No branding configured" is not a missing resource on the me alias; it is a
resource whose values are the platform's own. Asking about a named org that
has none is still a genuine miss, so that route keeps its 404 — and a caller
asking about an org that is not theirs keeps getting 404 rather than 403, so the
endpoint never confirms an org they cannot see.
The defaults are not merged into a partially-configured blob: an org that set
only primary_color reads back only primary_color, exactly as before. The
fallback replaces an absent blob whole. Folding the platform accents in
underneath a partial blob would re-theme a configured tenant with a colour it
never picked, and every BrandConfig field is optional, so a client already
resolves an absent key itself — the defect here was the 404, not the gaps.
Cascading org values over a complete default set is a product decision in its own
right and is not this change.
The values come from Settings (originally BRAND_DEFAULT_PRIMARY_COLOR,
BRAND_DEFAULT_SECONDARY_COLOR, BRAND_DEFAULT_LOGO_URL,
BRAND_DEFAULT_FAVICON_URL, BRAND_DEFAULT_ORG_DISPLAY_NAME; issue #1321 added
four more, listed below) and are assembled by core/branding.py, so a deploy
white-labels the platform without a code change. The two colours ship as the
platform's own report accents; the logo, favicon and display name ship unset,
because those identify an organization and the platform has no default for a
specific one — a client falls back to the org's real name. All nine env keys
behind BrandConfig are documented as a commented "Branding defaults" block in
both .env.example and .env.example.prod — see
Branding defaults
in docs/DEPLOYMENT.md (issue #1329, closing a gap that predated this change
and was never closed for the original five either).
BrandConfig — the full UI contract (issue #1321)¶
The Phase 14 F-10 owner walk hit a browser tab reading "undefined" and
bg-brand-primary buttons rendering transparent on an org that had
configured no branding of its own — the never-configured case GET
/v1/orgs/me/branding answers with the platform defaults. Its BrandConfig
served five keys while the UI's BrandConfig
(climate-lama-ui/src/branding/types.ts) has always read eight, reading
app_name for document.title and theme_mode to pick the CSS-variable
theme. BrandConfig (schemas/preferences.py) now declares all eight the UI
names, and the platform default payload fills all eight with a real value:
| Key | Default | Source |
|---|---|---|
primary_color |
#0b6e4f |
Settings.brand_default_primary_color |
secondary_color |
#b45309 |
Settings.brand_default_secondary_color |
logo_url |
None |
Settings.brand_default_logo_url |
favicon_url |
None |
Settings.brand_default_favicon_url |
app_name |
"Climate LAMA" — matches the UI's own hardcoded fallback (types.ts:15) verbatim, unless a display name is configured (see fallback rule below) |
Settings.brand_default_app_name |
theme_mode |
"light" |
Settings.brand_default_theme_mode |
default_locale |
"en" |
Settings.brand_default_locale |
support_url |
None |
Settings.brand_default_support_url |
theme_mode is a three-member Literal["light", "dark", "high-contrast"],
matching the UI union exactly — a narrower literal would let the backbone
reject a mode the UI can legitimately pick, recreating the same contract lie
in the other direction. app_name falls back to a configured, non-blank
org_display_name instead of the table's default above: a white-labeled
deploy wants its own name in the browser tab, not the generic platform one
(a whitespace-only display name is stripped and does not win the fallback,
to avoid trading one blank tab title for another).
org_display_name is a ninth field that stays backbone-only — it is not part
of the UI's BrandConfig contract (the UI has no such key) and identifies an
organization for other consumers instead.
All four new fields are optional on BrandConfig, like every other field —
including on the write side (PUT /v1/orgs/{org_id}/settings validates the
same schema), so an existing stored draft that carries an explicit null for
one of them still saves. theme_mode still rejects a value outside its three
members; only the null/absent case is unaffected.
Org overrides stored in the org_settings JSONB pass through unchanged, as
before — an org that configured only primary_color still reads back only
primary_color; the new keys only widen the default payload for a
never-configured org. An org that configured some branding but not
app_name/theme_mode reads those back as absent, exactly as before this
change — closing that gap for a partially-configured org is the UI's own
merge-over-defaults, tracked separately from this backbone contract fix.
Logo, report footer and formats — branded reports (issue #1689, SH-24)¶
| Field or route | Who | What |
|---|---|---|
POST /v1/orgs/{org_id}/branding/logo |
org admin of that org | multipart file: an SVG or a PNG at least 256 px wide (an SVG's width or viewBox width), at most ORG_LOGO_MAX_BYTES (1 MiB). Stored as sent at storage_keys.org_logo_key(org_id) = branding/{org_id}/logo with its content type; branding.logo_url becomes /v1/orgs/{org_id}/branding/logo; answers the updated branding. A JPEG, a narrower file, a larger one, or an SVG holding a <script>, an event attribute (onload) or a reference to anything but a # fragment answers 422 E_VALIDATION and stores nothing — refused, never cleaned. An analyst gets 403; another org's id 404 |
GET /v1/orgs/{org_id}/branding/logo |
any member of that org | the stored bytes with their type, Content-Security-Policy: default-src 'none' and X-Content-Type-Options: nosniff; 404 E_NOT_FOUND when none is stored or for another org's id |
branding.report_footer |
PUT /v1/orgs/{org_id}/settings (org admin) |
text of 200 characters at most (201 answer 422); default_branding() serves report_footer: null |
UserPreferences.formats |
PUT /v1/users/me/preferences |
currency_display (symbol or code), money (compact or full), number (en, el or iso), area (km2 or ha) and date (short, dmy or iso), the snapshot's option values (climate-lama-ui src/i18n/format.ts); an unknown key or value answers 422, an absent key reads as its first option |
render_report prints every report for the organization and the user who asked
for it: the cover carries a band in branding.primary_color (a plain hex, else
the stylesheet accent), the stored logo — read by the worker from the object
store by its key and inlined as a data: URI, never fetched through
logo_url — and the organization's org_display_name, else its legal name;
the footer runs at the foot of every page, the page counter beside it after the
cover. Money, counts and dates follow the requester's formats, as on screen: under
money: compact a headline figure (a summary metric) reads compact and a table
cell in full. A headline reads €4.9M and the date
26 Sep 2026 with none stored, EUR 4.900.000 and 26/09/2026 for
{currency_display: code, money: full, number: el, date: dmy}, and a unit that
is not money prints after the number (16.8M m3). The renderer's URL fetcher
loads data: URIs only (core/reports/renderer.py::ALLOWED_URL_SCHEMES), so no
markup in a report can make the worker open a network or local URL.
Role lens — persona, default_persona and effective_persona (issue #1466)¶
The role lens (Insurer · Scientist · Decision-maker · Citizen) shapes presentation
only — nav order, Home and wording. No authorization check reads it:
require_role and every other guard decide on Role, never on the lens.
| Key | Blob | Values | Written by |
|---|---|---|---|
persona |
user prefs (PUT /v1/users/me/preferences) |
insurer | scientist | decision_maker | citizen | null |
the user |
default_persona |
org settings (PUT /v1/orgs/{org_id}/settings) |
same four | null |
an org admin |
effective_persona |
derived on read, never stored | same four | null |
nobody |
Any other string on either write is refused with 422 E_VALIDATION (the
prototype's decision id included). GET /v1/users/me/preferences — and the
PUT response, which has the same shape — carries effective_persona, resolved
by core/persona.py as user persona → org default_persona → null. A stored
value outside the four lenses resolves as unset. Serving the org default through
this derived key keeps GET /v1/orgs/me/settings and GET
/v1/orgs/{org_id}/settings ORG_ADMIN-only while every member still sees it
applied. An effective_persona sent on PUT is dropped before validation, not
refused, so a client that echoes the GET body back still saves.
The same change declares three display preferences on UserPreferences, read
by the UI's Settings screen and map chrome: density (comfortable |
compact), default_palette (a palette id, free string) and basemap (the UI
basemap registry's ids, climate-lama-ui src/map/basemaps.ts: auto |
light | dark | greyscale | streets | satellite | terrain, plus
the legacy mono, still accepted so stored values validate). All five keys are
optional and schema-only — both
blobs are extra="allow" JSONB, so no migration was needed.
/v1/scenarios — open on write, typed on read (issue #1186)¶
scenarios.inputs is a free dict on the way in and stays one: any dict the
client sends round-trips unchanged, and no shape ever fails a create.
ScenarioCreate.inputs is still a bare dict[str, Any], and ScenarioUpdate
still excludes inputs from PATCH (to revise inputs, save a new scenario —
result_id must keep meaning "the result produced for these inputs").
What #1186 adds is the way back out. schemas.scenario.ScenarioInputs is a
read-side lens over the same blob — every field optional, extra="allow", so
parsing it is lossless — declaring the keys the backbone actually acts on:
| Key | Used for |
|---|---|
hazard_dataset_id, exposure_dataset_id, impact_function_id |
The replay's references |
scenario_label, horizon_year, engine |
The replay's cell and runtime |
measure_ids, discount_rate, growth_rate |
Cost-benefit inputs |
batch_id, comparison_id |
Back-links to the group the scenario was saved out of |
portfolio_id or exposure_dataset_ids, scope, haz_type, cells, adapted_from_result_id |
A saved run's keys (run model § 6, #1663). cells is the matrix request's [{scenario_label, horizon_year}] with horizon_year null for Today, completed cells only |
compare |
A saved comparison's sides, {a, b, engines}, each side in the cell grammar (baseline@today, ssp5-8.5@2080); an engine comparison stores {a, engines: true} beside its comparison_id |
live_event_id, event_key |
A saved event replay's watch and the one event its figures sit inside (EV-19) |
Anything else is preserved and reported, never interpreted. GET /v1/scenarios
takes two optional filters, live_event_id=<id> (the rows whose
inputs.live_event_id names that watch) and mine=true (the rows the caller
created). They combine, total counts the filtered set, and with neither the
list answers as before.
Nothing is ever rejected — only described. core/scenario_service.py turns
each discrepancy into a warnings[] entry with a machine-readable code, the
field it concerns and a message:
| Code | Meaning | Blocks computable? |
|---|---|---|
unknown_input_key |
A key the backbone does not interpret; stored and returned unchanged | no |
invalid_input_type |
A declared key holding an uncoercible value. It is dropped from the parsed view only and the blob is re-parsed without it, so one bad horizon_year cannot hide a good hazard_dataset_id |
yes |
unparseable_inputs |
inputs is not an object at all |
yes |
missing_hazard_dataset |
Neither a hazard_dataset_id nor a haz_type with an exposure selector (exposure_dataset_id, exposure_dataset_ids or portfolio_id). A saved event replay keeps it and reads computable: false |
yes |
incomplete_scenario_cell |
A horizon_year with no scenario_label |
yes |
label_without_horizon |
A scenario_label with no horizon_year |
no |
unresolved_reference |
A referenced hazard/exposure/impact-function row no longer exists in the org | yes |
unresolved_measure |
A referenced measure is gone; impact replay is unaffected | no |
The two halves of the pair are not symmetric, and that is the point. The
pair is applied to a replay together or not at all (#1185's rule), so a lone
horizon_year is simply dropped — and nothing else in the blob carries a
horizon, so replaying it would silently compute a different year than the one
that was saved. A lone scenario_label is recoverable: the replay runs
dataset-addressed and worker.tasks keys the run surface with
params.get("scenario_label") or dataset.scenario, so the hazard dataset's own
scenario stands in. It is still reported, because that substitution is only a
no-op while the two agree — but calling such a scenario uncomputable would be a
false negative on the most ordinary shape a client saves.
POST /v1/scenarios returns the shape half of that list in its 201 body — no
reference lookup, so the write path costs no extra query. GET
/v1/scenarios/{id} returns the raw inputs unchanged, plus:
inputs_resolved—hazard_dataset/exposure_dataset/impact_function/measuresby name (the hazard descriptor is the same{id, name, scenario, supported_years}shape an unresolved matrix cell lists its candidates with), the normalizedscenario_label/horizon_year/engine,unresolved_ids, andcomputable;warnings[]— the full list, shape and references.
computable means "nothing in the saved blob is missing, malformed or
dangling". It is derived from the warning list (not any(code in
REPLAY_BLOCKING_WARNINGS)) rather than computed alongside it, so the flag and
the list cannot disagree.
It is a verdict on the scenario as saved — neither necessary nor sufficient for a particular replay to be accepted. Both directions are real and neither is a defect:
| Case | Why |
|---|---|
computable: true, replay still 422s |
The replay runs the ordinary compute gates (impact-function resolution, EAI eligibility, curve applicability) that a read endpoint has no business executing. The exposure reference is also read from two different tables on purpose: the read side resolves the exposure_datasets catalog (that is where the name lives), while validate_impact_refs requires actual exposures rows, so a catalog row whose assets were all ungrouped reads as resolved and is refused on compute. |
computable: false, replay still succeeds |
The request overrode the broken field — {"scenario_id": S, "exposure_dataset_id": <a good one>} — so the dangling id never reaches a gate. That is precedence rule 2 working, not a hole. |
So computable: false means "replaying this scenario as saved would be
refused", never "this scenario would not compute today" and never "any request
naming it is refused".
Every lookup is org-scoped, so a reference belonging to another organization reads as unresolved rather than resolved — the description must never become a way to confirm that another org's id exists.
GET /v1/compute/impact/matrix — one index over both group kinds (issue #1186)¶
The batch list now returns engine-comparison groups alongside scenario
matrices, and tags every row with kind ("scenario_matrix" or
"engine_comparison", via core.engine_comparison_service.batch_kind, so rows
predating the discriminator read as matrices). Comparisons were excluded because
the two kinds share one table and the route is named for one of them — but the
Runs hub this list feeds is precisely the surface a comparison is supposed to be
recoverable from, so hiding it there left a comparison as unrecallable as a batch
had been before #1009.
kind is what tells a client which detail route to open a row with:
GET /v1/compute/impact/matrix/{id} for a matrix, GET
/v1/compute/impact/compare/{id} for a comparison. The detail endpoints are
unchanged — each still answers 404 for a group of the other's kind, because
their cells key on different things (scenario_label/horizon_year vs engine)
and their views mean different things. The list is a cross-kind index, not a
claim that the two are interchangeable.
A comparison carries no scenario axis, so its scenario_labels and
horizon_years come back empty — the honest reading of a group that fans one
scenario across engines. Status rolls up identically for both kinds, since every
cell of either carries a job_id.
Naming a group's inputs, and discarding a group (issue #1270)¶
1186 landed asymmetrically: the scenario detail route resolved its inputs¶
while this list and the scenario list did not. Both list rows now carry
inputs_resolved — hazard_dataset, exposure_dataset, impact_function
(each null when the reference is gone) plus unresolved_ids — under the same
key names on both surfaces, so one client component renders a scenario row and a
group row. A group's projection carries no computable: that flag is a
verdict on replaying a saved scenario as saved, and a compute group is a record
of a run that already happened. A group row also carries exposure_datasets
(issue #1727): one descriptor per exposure the group ran against, the members of
a union in stored order, or the single exposure_dataset for a one-exposure
group; a member that is gone is named in unresolved_ids. A matrix built from
explicit cells lists its scenario_labels and horizon_years as the distinct
labels and non-null years of its cells, in cell order.
The keys are purely additive — hazard_dataset_id and every other
pre-existing field stay exactly where they were, because climate-lama-ui's
tests/e2e/smoke/contract.spec.ts reads them by name.
Both lists resolve their whole page in at most four reads, not one per row
(core.catalog_resolution.load_catalog_names): the point was to delete a
client-side N+1, and doing that by opening a server-side one would be no fix.
DELETE /v1/compute/impact/matrix/{id} and DELETE
/v1/compute/impact/compare/{id} discard a group (ANALYST+), kind-scoped exactly
like the detail routes so a mistyped path cannot destroy the group the caller did
not name. The delete is hard, unlike a scenario's soft delete: nothing
references compute_batches by foreign key, and the Jobs and ImpactResults the
group tracked are independent rows — so what is discarded is the grouping,
never the compute history. One exception, comparisons only (issue #1639, EN-4):
a deleted comparison's cells that are still pending end cancelled, with
completed_at set, in the same commit as the delete, and each one's Celery task
is then revoked (best effort, never terminate=True), so they free the queue
instead of running for a group nobody can open. A running cell finishes and
keeps its result. The worker also refuses a job whose row already reads
cancelled, because a revoke lives only in worker memory and a restart forgets
it. A compute_group.deleted audit entry
records that it happened. A cross-org id is a 404, indistinguishable from an
absent one, because a 403 would confirm that a foreign id is real.
Status codes¶
200 (GET/PUT), 201 (POST create), 202 (async job submitted), 400, 401, 403, 404, 422, 500.
Naming¶
Resources: plural nouns (/exposures, /hazards). Actions: POST + verb (/compute/impact). URLs: kebab-case. JSON bodies: snake_case.
5. Data Models¶
HazardDataset¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| name | str | Dataset name (dedup key with haz_type) |
| haz_type | HazardType enum | river_flood, tropical_cyclone, etc. |
| source | str | Data source (e.g., "JRC") |
| region | str | Geographic region |
| scenario | str | Climate scenario label |
| intensity_unit | str | "m", "m/s" |
| frequency_type | str | "marginal" or "occurrence" |
| npz_path | str | MinIO path to intensity matrix NPZ |
| return_periods | JSON list or None | Original RPs (e.g., [10, 50, 100, 200, 500]) |
| crs | str | Coordinate reference system (default "EPSG:4326") |
| bbox | JSONB or None | Upstream extent before any ingest-time clip |
| grid_transform, grid_width, grid_height | float[6], int, int | Regular-grid definition — the primary centroid representation (ADR-045 topic 2) |
| footprint | Polygon (PostGIS) or None | EPSG:4326 envelope of the ingested grid; GiST-indexed dataset extent (#596) |
HazardEvent¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| dataset_id | UUID | FK -> HazardDataset, ON DELETE CASCADE (hazard_events_dataset_id_fkey) |
| event_name | str | "RP-10", "RP-100", "ssp5-8.5@2050", etc. |
| event_index | int | 0-based row of the intensity matrix; unique per dataset (uq_hazard_events_dataset_idx) |
| rp | float or None | Return period value (e.g., 10, 100, 500); null on a scenario × year event |
| exceedance_freq | float or None | Exceedance frequency (1/RP); null on a scenario × year event |
| marginal_freq | float or None | Computed marginal frequency (used by engine); null on a scenario × year event |
| scenario | str or None | Scenario label of a scenario × year event, verbatim from the source (ADR-068) |
| year | int or None | Projection year of a scenario × year event (ADR-068) |
Event order is event_index. Each event row stores the intensity-matrix row it describes in
event_index, written at ingest from the event's position on the dataset's event axis, and
(dataset_id, event_index) is unique. Readers that align events with the NPZ matrix sort on it
(ORDER BY event_index), never on insertion or primary-key order.
Deleting a dataset removes its events and centroids. HazardRepository.delete_dataset
deletes the hazard_datasets row only; hazard_events and hazard_centroids follow through
their ON DELETE CASCADE foreign keys. Migration 0002 created both dataset_id columns without
the key, so until migrations 0112 and 0113 (#1816) a deleted dataset left its rows behind. 0112
adds the two keys NOT VALID, which takes no row scan under its lock; 0113, in its own
transaction, removes the rows whose dataset_id matches no dataset and validates the keys while
reads and writes go on.
HazardCentroid — legacy only since #596 (ADR-057)¶
No ingest path writes these rows any more. A hazard dataset's centroid geometry is the
grid_transform/grid_width/grid_height triple on HazardDataset plus the pixel_idx
array in its v2 intensity.npz — eight numbers and one array instead of one row per pixel
(the table was 4294 MB of a 4.32 GB prod database). Rows written before #596 are still read:
get_centroids, admin aggregation, and the membership assignment join all keep a fallback for
datasets that have not been backfilled onto a v2 artifact.
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| dataset_id | UUID | FK -> HazardDataset, ON DELETE CASCADE (hazard_centroids_dataset_id_fkey) |
| array_index | int | Column index in intensity matrix |
| pixel_index | bigint or None | Row-major cell index in the dataset grid |
| lat, lon | float | Coordinates |
| geometry | Point (PostGIS) | Spatial geometry with GiST index |
Exposure¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| name | str | Dataset name (dedup key) |
| dataset_id | UUID | Optional grouping FK |
| geometry | Point (PostGIS) | Location |
| value | float | Asset value |
| value_unit | str | Currency |
| exposure_type | ExposureType | buildings, population, etc. |
An asset's hazard cell is not a column of exposures: it is one ExposureCentroidAssignment
row per (asset, hazard dataset) pair, so one asset takes part in several hazards at once.
ExposureCentroidAssignment¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| exposure_id | UUID | FK -> Exposure, ON DELETE CASCADE |
| hazard_dataset_id | UUID | FK -> HazardDataset, ON DELETE CASCADE |
| array_index | int or None | 0-based column of the hazard's intensity matrix; null is the no-cell sentinel (#442), an asset inside the grid on a pixel with no stored cell, which computes to zero |
(exposure_id, hazard_dataset_id) is unique (uq_exposure_centroid_assignments_pair). The rows are
written by the centroid assignment (core/centroid_assignment.py), explicitly or at compute time.
ImpactFunction¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| name | str | "JRC Flood - Europe - Residential" |
| haz_type | HazardType | river_flood, coastal_flood, tropical_cyclone, wildfire, storm_europe, earthquake |
| intensity_unit | str | "m", "m/s", "K", "g" |
| mdd_x, mdd_y | list[float] | MDD curve sample points (x = intensity, y ∈ [0, 1]) |
| paa_x, paa_y | list[float] | PAA curve sample points (x = intensity, y ∈ [0, 1]) |
| source | str | "builtin" (seeded, protected from deletion) or "user" (user-created) |
User-defined curves (Phase 4+, #69): The data model already supports arbitrary user
curves via the source = "user" flag. The POST /v1/impact-functions endpoint (Phase 4+)
will accept any piecewise curve with monotonically increasing x and y ∈ [0, 1]. Builtin
functions cannot be deleted. In Phase 2+ (multi-tenancy), a org_id column will scope
user-defined curves per organisation. The compute endpoint requires no changes — it already
accepts any impact_function_id UUID without distinguishing source.
Vulnerability resolution (issue #1591, PF-4, K19)¶
/impact, /impact/matrix and /impact/compare resolve the damage curve on
(haz_type, exposure_type) (run model § 3.5, § 6), in
core.impact_function_service.resolve_impact_function_id. The exposure set's
exposure_type and value_unit come from its members' catalogue rows: the
named datasets, or a portfolio's member datasets. Members that differ in either
are refused first ("exposure datasets differ in value_unit: USD, m3"), a
portfolio included. With no impact_function_id, the curve is the
is_default row of impact_functions for the hazard's haz_type and the set's
exposure_type (the (org_id, haz_type, exposure_type) WHERE is_default index
of #1576). Two refusals name their cause, both 422 E_VALIDATION at
submission, before any job is queued:
| Refusal | When | details |
|---|---|---|
| "no vulnerability curve for exposure type population" | no curve for the hazard type and the set's type (a curve-less, non-EAI-eligible dataset keeps #918's typed refusal; buildings on a curve-less peril keep the seeding-gap message) | reason, exposure_type |
| "no money loss for value unit m3" | the set's unit is not money (core.value_units.is_money_unit: m3, persons, "") |
reason, value_unit |
A caller-supplied impact_function_id passes the first refusal (ADR-061's
built-in applicability check still runs on it) and never the second. An
org-wide run (no exposure selector) keeps the buildings default and neither
refusal runs. GET /v1/impact-functions?haz_type=river_flood&exposure_type=buildings
lists the options for one pair, each with its version; a pair with no curve
answers an empty list. A live watch asks for the first refusal's text when
auto_compute is switched on, before any snapshot exists ("The curve gate"
under Live events, issue #1615).
Every result of those routes records provenance.vulnerability =
{impact_function_id, version, resolved_on}, with resolved_on
"haz_type+exposure_type" for an auto-resolved curve and "explicit" for the
caller's. The job carries it as params.vulnerability_resolved_on.
A hazard that is not climate-driven runs on its baseline (issue #1592, run
model § 3.4). When no dataset in a cell's resolution pool declares a
non-baseline scenario (climate_driven: false on the coverage grid, e.g.
earthquake), resolve_cells_within_scope answers a cell (s, y) with
s ≠ baseline from the baseline group, on /impact, /impact/matrix and
/impact/compare. The job carries params.hazard_conditioning =
"not_climate_driven", and the result records
provenance.hazard_conditioning = "not_climate_driven" beside the cell's own
scenario_label/horizon_year, so it reads as "same as today", never as a
projection. A climate-driven cell records no hazard_conditioning key, and a
climate-driven hazard missing the asked SSP still fails with #1185's reason
and candidates. Exposure growth is unchanged by this rule.
Job¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| job_type | JobType | data_ingest (covers hazard ingest + centroid assignment), impact_calc |
| status | JobStatus | pending, running, completed, failed, cancelled (terminal; a queued cell of a deleted comparison, migration 0116, issue #1639) |
| celery_task_id | str | Celery task ID |
| params | JSON | Input parameters |
| result_id | UUID or None | FK -> ImpactResult (set on completion) |
| result_payload | JSON or None | Small result body of a job that stores no result row — the score_delta appraisal (migration 0111, issue #1619); null on every other job |
| error_message | str or None | Set on failure — free text, capped at 1024 chars |
| error_code | str or None | Set on failure — a core.errors.Code value (E_JOB_FAILED when nothing more specific fits). Clients branch on this, never on error_message (migration 0089, issue #1182) |
| created_at, started_at, completed_at | datetime | Timestamps |
JobRepository.mark_failed(job_id, *, code, message, user_id=None) is the single
chokepoint that writes the failure: it sets status/error_code/error_message/
completed_at and emits a job_failed in-app notification (the sixth
ADR-037 type, payload {job_id, job_type, error_code, error_message,
title}). mark_job_failed_sync is its Celery-side twin and
IngestJobRepository.mark_failed the ingest-lane equivalent. The recipient is always
passed in — jobs has no user column — and a None user_id fails the row and skips
the emit. set_failed remains only as a deprecated shim delegating to mark_failed.
ImpactResult¶
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| job_id | UUID | FK -> Job |
| exposure_dataset_id | UUID | FK -> ExposureDataset |
| hazard_dataset_id | UUID | FK -> HazardDataset |
| impact_function_id | UUID | FK -> ImpactFunction |
| ead | float | Expected Annual Damage |
| aai | float | Average Annual Impact |
| impact_per_event | JSON | {event_name: value} |
| eai_exp_path | str | MinIO path to per-exposure NPZ: eai_exp and, since issue #1685, band_exp beside it (each location's score band, 0 for none) |
| freq_curve | JSON | [{return_period, loss}] exceedance curve, or null / [] when the engine produced none |
| curve_basis | str (nullable) | Exceedance basis of freq_curve: OEP or AEP (CHECK ck_impact_results_curve_basis, migration 0096, #1476). NULL on a curveless result |
| metadata | JSON | ORM attribute metadata_. Engine version, valid/excluded points, plus the scope-coverage pair rows_computed/scope_total (#1180), the rows' shared value_unit (#1583), the exposed value, band shares and by-type split (#1595), the cells stamp of the result's stored aggregates (#1470, below) and checksum (#1636, below) |
Result checksum (issue #1636, SH-10)¶
compute_impact stamps metadata.checksum when it writes the result (no migration): the
SHA-256 hex digest (64 lowercase characters) of the result's stored arrays in one canonical byte
form, built by core/result_checksum.py::result_checksum. In order:
eai_exp, the per-location array written toeai_exp_path(sentinel rows included): its length, then each value in array (compute) order.impact_per_event: its entry count, then for each entry, sorted by the event name's UTF-8 bytes, the name's byte length, the name's bytes and the loss.freq_curve: its point count (0 for a curveless result), then for each point in stored order itsreturn_periodandloss.
Every count and length is an unsigned 64-bit little-endian integer and every value an IEEE-754
binary64 little-endian float. Sorting the events makes the digest recomputable from the stored
JSONB, which does not keep key order. The same inputs computed on the same engine version give
the same checksum. GET /v1/results/{id} serves it as checksum, null for a result written
before it; History's Reproduce dialog shows it (ui#848).
ReportDocument (report documents)¶
A saved, listable report (migration 0097, issue #1487, ADR-085). One org-scoped
table, ENABLE + FORCE ROW LEVEL SECURITY with the standard org_isolation policy.
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| org_id | UUID | FK organizations, ON DELETE CASCADE |
| created_by_user_id | UUID (nullable) | FK users, ON DELETE SET NULL — the author |
| name | str | Display name |
| template | str | A key of core/reports/composer.TEMPLATES, checked on write (no CHECK constraint) |
| scope | JSON | {label?, portfolio_id?, admin_boundary_id?} — what the report covers, the studio's scope chip: one portfolio (PF-6) or one municipality, a place report's unit (RP-2, never a run's clip list), not both |
| widgets | JSON | Ordered list, one of result{result_id \| haz_type, scenario, horizon_year?; view: summary\|curve\|top_exposures\|by_hazard\|horizons\|top_areas\|map}, batch{batch_id, view: horizons\|matrix}, analysis{scenario_id}, event_result{live_event_id, job_id}, place{lat, lon, scenario?, horizon_year?}, portfolio_rollup{portfolio_id, scenario?, horizon_year?}, adaptation_plan{plan_id}, text{title, body, section?}, note{body, include_in_export=false} |
Scoped documents and run-address widgets (#1638, PF-6, RP-2). A result widget names either
a result_id or a run address {haz_type, scenario, horizon_year} (horizon_year absent is
Today), never both. A run address is read at render against the document's scope through
core/stored_results.py::list_stored_results on the default engine: the scope's exposure is its
portfolio_id, or for a place the default exposure layer of the unit's country
(default_exposure_dataset) clipped to admin_boundary_id. With no stored result the section reads
"Not computed yet. No stored result for map credits the portfolio template's map, top_areas its top
areas). A portfolio_rollup widget re-reads rollup_portfolio and the per-hazard EAD of
PortfolioRepository.list_metrics at render: the headline band, each hazard's band, the band
distribution with the share in high or severe, the "value-weighted average band (ordinal,
screening)" and one expected annual impact per hazard, never a cross-hazard total (K26). A
mixed-unit portfolio prints "Mixed units: summary nor its hazards. On write, another
organization's portfolio in scope.portfolio_id or a widget is a 404 and an unknown
scope.admin_boundary_id a 422.
Adaptation-plan widget (#1642, AD-8). An adaptation_plan{plan_id} widget stores the plan id
only. At render core/reports/gather.py::read_adaptation_plan re-reads the plan and its
last_appraisal_job_id job's cost_benefit_results rows, the rows
GET /v1/results/cost-benefit/{job_id} serves, through the same cost_benefit_service readers
(plan_block, job_figures). The section prints the plan's name, hazard, scope (portfolio or
exposure layers, and the clip) and cell, the plan totals (NPV benefit and cost, benefit-cost ratio,
payback), the avoided loss and residual risk of the plan row (or a one-measure job's one row,
never a sum of per-measure rows), and the measures ranked by benefit-cost ratio at their cumulative
cost. A plan with no linked job reads "Not appraised yet." with no figure; a deleted plan reads
"not available". On write, another organization's plan is a 422 reference_not_found, as for
results, batches, analyses and event results. The coverage checklist credits nothing to this
widget.
Widgets hold ids, never copied figures: every render re-reads the stored rows, and a reference
deleted since the document was saved renders "not available" (#1489). On write,
core/report_document_service.py refuses a template the composer does not register, more than 50
widgets, and any referenced id the caller's organization does not own (a soft-deleted scenario
counts as absent). PATCH (#1488) runs the same checks on the fields it is sent, and replaces
widgets wholesale. DELETE removes the row only.
Every TEMPLATES key carries a TEMPLATE_INFO entry beside it in core/reports/composer.py (a
guard test enforces it): a display name, the subject it reports on (result, batch or
document) and an availability of available or wording_to_verify (a framework template
whose wording is not yet verified against the primary source — the three framework templates and
orsa, whose checklist items are all EIOPA paraphrases still to_verify; a guard test fails
while a template whose items are all to_verify is listed available, #1637). The names are the
prototype's eight: "Impact analysis report", "EIOPA ORSA climate scenario pack", the three
framework names, "Portfolio risk report", "Place risk report" and "Post-event report"; the
portfolio, place and event covers print them.
GET /v1/report-documents/templates serves it to the report studio.
Rendering (#1489). POST /v1/report-documents/{id}/render queues a REPORT_RENDER job; the
worker renders through WeasyPrint (ADR-039) and stores the PDF, then its manifest, under
reports/{org}/{document_id}/{template}/ (core/reports/storage.py, keyed by exactly one of
compute job, batch or document id). The portfolio template (subject document) composes a
DocumentReportContext: one section per exported widget, in widget order, each reusing the
gather_report_context / gather_matrix_report_context output of the row it references
(core/reports/gather.py::gather_document_report_context). A result or batch that no longer
exists renders the studio's sentence, "This result is no longer stored. The analysis it came from
was deleted, so there is no figure to show." (a gone batch its own "not available" line), with no
figure; a note
is left out unless include_in_export; an analysis widget renders a "not available" section.
The place and event templates (#1490, subject document) share that path and its section
markup (templates/_macros.html::document_section), so every document template renders every
widget kind the same way. A place widget reads the point's score card through lookup_point
and prints the D6/D7 headline — the worst banded hazard, an "indicator" badge when an indicator
drives it, its hazard group and the four group levels (flood, earthquake, fire & heat, storm) —
plus "n hazards not covered", counted from get_coverage's none cells for the country the card
resolved; with score_bands_enabled false there is no headline, only the bands-disabled notice.
An event_result widget reads its stored event_impact_results row: event name, source,
valid_at, the loss labelled "Event loss" (never an annual figure, ADR-078) and the exposed count
and value; a missing row renders the same gone-result sentence. Beside the event name it prints
the event's key (RP-2, #1637): the event_key of the snapshot the result scored (EV-11), else the
live event's id, read at render and never stored on the widget (ADR-085); an event document's
cover prints the key of its first stored event result, and a gone result prints none. The
impact and orsa templates
render a document through their own context when it holds exactly one result (impact) or batch
(orsa) widget, otherwise the render request is 422. GET /v1/report-documents/{id}/pdf serves
the last complete render of the document's current template.
Coverage checklist (#1491). GET /v1/report-documents/{id}/checklist returns what the document
covers for its template: every template has an item set in core/reports/checklist.py::ITEM_SETS
(a guard test enforces it), and each item is ok, partial or missing with an evidence line.
It is computed on each read from the stored widgets and the rows they reference (a batch's
scenario and horizon axes, read as the ORSA pack reads them, a result's hazard type and scenario, an event result's loss
and exposed count, read through ReportDocumentRepository.referenced_rows), never stored, so it
always matches the current content. A deleted reference counts as absent. There is no score, grade
or "compliant" verdict (compliance scoring is Phase 18). Each item carries a wording:
internal for the platform's own section names (impact, portfolio, place, event),
to_verify for a paraphrase of framework guidance, or verified with the source from
orsa.EIOPA_SOURCES it is quoted from. The orsa items paraphrase EIOPA guidance, so every one is
to_verify. A guard test fails if an ORSA item claims verified without such a source.
Framework templates (#1492). Three document templates arrange a saved report document under
a reporting framework's physical-risk headings: esrs_e1 (CSRD / ESRS E1), eu_taxonomy_cra (the
EU Taxonomy climate risk and vulnerability assessment) and ifrs_s2. All three render through
one file, templates/framework_report.html, on the document path above (subject document), and
print an outline of the framework's requirement headings
and then every exported widget in widget order. Everything they say about a framework — headings,
descriptions and checklist items — lives in the pure-data core/reports/frameworks.py::FRAMEWORKS,
on the orsa.py pattern. No published text has been read against it yet, so every heading and item
is a paraphrase: the templates are listed with availability: wording_to_verify (computed from the
module, so verifying the last paraphrase flips it), each document prints a "Wording to verify"
banner and suffixes every unverified heading "(to verify)", and every checklist item is
to_verify. Verifying one is a later step that sets verified=True together with the
orsa.Source it is quoted from. Two guard tests (tests/test_core/test_report_frameworks.py)
fail the build on verified wording without a Source, and on "must", "shall" or a numbered
requirement reference (an article, paragraph or coded disclosure number,
frameworks.REQUIREMENT_LANGUAGE) in any unsourced framework string. No framework document
states that the report satisfies the framework, and none carries a compliance score.
AdaptationPlan (saved adaptation plans)¶
A saved, listable adaptation plan (migration 0114, issue #1632, AD-8, AD-17; domain model § 4.3
"A saved plan"). One org-scoped table, ENABLE + FORCE ROW LEVEL SECURITY with the standard
org_isolation policy. The field names are the compute request's (guardrail 11).
| Column | Type | Description |
|---|---|---|
| id | UUID | PK |
| org_id | UUID | FK organizations, ON DELETE RESTRICT |
| created_by_user_id | UUID (nullable) | FK users, ON DELETE SET NULL — the author |
| name | str | Display name |
| portfolio_id | UUID (nullable) | The portfolio selector; no FK — a plan stores ids |
| exposure_dataset_ids | UUID[] (nullable) | The dataset-union selector; exactly one of the two (ck_adaptation_plans_one_selector) |
| scope | JSON (nullable) | The run's clip as sent: {admin_boundary_ids}, {admin_boundary_id} or {bbox} |
| haz_type | hazard_type_enum |
The plan's one hazard; every measure is of it |
| scenario, horizon_year | str, int (nullable) | The cell appraised, both or neither (ck_adaptation_plans_cell_pair) |
| measure_ids | UUID[] | The measures in plan order |
| custom_order | bool | true once the user reordered the plan away from the canonical order (AD-13) |
| discount_rate, value_growth_rate, time_horizon_years | float, float, int | The appraisal settings, as on POST /v1/compute/cost-benefit |
| last_appraisal_job_id | UUID (nullable) | FK jobs, ON DELETE SET NULL — the plan's last combined appraisal; written by the cost-benefit path only (#1641), never by a client |
A plan holds ids and settings, never figures (ADR-085). Every read serves
appraisal_status (the linked job's status, null with no job), avoided_loss and
avoided_loss_unit, read from that job's cost_benefit_results (core/adaptation_plan_service.py
plan_appraisal): the plan row's risk_baseline_future − risk_with_measure_future — the plan
block's avoided_loss_future (#1624) — or, for a one-measure job, its one measure row's; never a
sum of the per-measure rows (AD-13). Both figures are null until the job completed. A PATCH that
changes anything the appraisal read clears the link, so a listed figure is always the plan's own run.
ResultH3Cell and ResultAdminCell (result-keyed aggregates)¶
Written by compute_impact right after the result commit, so a result map can be served as
zoom-driven vector tiles (ADR-083). Both tables are keyed by the result,
unlike the answer-plane org_risk_cells. Both are org-scoped with FORCE RLS, and both cascade
when their impact_results row is deleted (migration 0095).
| Table | Columns | Notes |
|---|---|---|
result_h3_cells |
org_id, result_id, h3_index, resolution, metrics {eai, eai_density_km2, points, band_max?, band_value_weighted?}, geom (POLYGON) |
CHECK (resolution BETWEEN 0 AND 8), unique (result_id, h3_index), GiST on geom; built by core/surface_writer.build_cells at result_cell_resolutions (default r4–r8), capped at result_cell_max_cells (default 250 000, highest-EAI cells kept). The two band keys (issue #1685) are present only on a cell holding a banded point, see "Score bands of a result's locations" below |
result_admin_cells |
org_id, result_id, admin_boundary_id, admin_level, eai, points |
unique (result_id, admin_boundary_id); each asset is summed into the GADM units its own coordinates fall in (ST_Intersects against admin_boundary_parts, below), at result_cell_admin_levels (default 0, 1, 2). An asset in no loaded unit is in no row, so a result whose every asset lies outside them (a book at sea, issue #1791) stores none, stamps metadata.cells.admin_cells = 0 and serves an empty admin layer |
The point-in-unit join reads admin_boundary_parts (migration 0100, #1570), not the whole
GADM polygons: each admin_boundaries row cut by ST_Subdivide(geom, 256) into polygons of at
most 256 vertices, columns admin_boundary_id (cascade), admin_level, geom (POLYGON, GiST).
It is reference data like admin_boundaries (no org_id, no RLS), and no application code
writes it: the trigger trg_admin_boundary_parts rewrites a boundary's parts on insert or on an
update of geom / admin_level and drops them on delete, so both loader paths of
scripts/load_admin_boundaries.py keep it in step. A geometry ST_Subdivide cannot cut is
stored whole. The parts of a boundary union to the boundary, so the per-unit totals are those of
the whole-polygon join; a point on a seam between two parts is counted once per unit.
core/result_cells.py decides what is stored: EAI is paired with assets by compute-order
position, never by centroid index (#370). The write is best-effort and one transaction. Its
outcome is stamped on impact_results.metadata.cells =
{status: "built" | "failed", resolutions, h3_cells, h3_dropped, admin_cells, eai_range} and
served by GET /v1/results/{id}. eai_range is measured once at write time so the UI can colour
tiles by value: {admin: {"<level>": [min, max]}, h3: {"<resolution>": [min, max]},
assets: [min, max] | null}, with no key for a tier that stored no cells (a failed stamp holds
{admin: {}, h3: {}, assets: null}). Stamps written before it carry no eai_range and are not
backfilled. A result stamped failed, or with no stamp at all, is drawn from the
GeoJSON fallback (/v1/results/{id}/geojson).
Ranked admin units of a result — GET /v1/results/{id}/admin-cells (issue #1636, SH-16)¶
Home's top municipalities and the run drawer's treemap read one result's areas highest-impact
first from the stored result_admin_cells, never from the request-time join of
/geojson?admin_level= (which Home must not call, ui#539). core/result_admin_ranking.py
answers {cells_status, items: [{admin_boundary_id, name, admin_level, eai, points}]}: the rows at
admin_level ordered by eai descending (then id), at most limit (default 10, capped at 50),
each named from admin_boundaries by id (null when a re-load dropped the unit). It is one
statement through the (org_id, result_id, admin_level) index whatever the result's size; nothing
is summed and no point is joined to a polygon. admin_boundary_id is what a client writes into
clip=admin:<id>.
cells_statusis the result'smetadata.cells.status, or"absent"with no stamp. Rows are served only when it is"built": afailedor unstamped result answers its status anditems: [].geometry=simplifiedaddsgeometry, each unit's polygon as GeoJSON,ST_Simplify'd with a tolerance of one pixel of a 200 px wide drawing of the served units' extent (longitude span ÷ 200). A ring the simplification collapses (a small or thin island) is dropped, and coordinates keep the digits a tenth of the pixel needs. Greece's outline cut into its 14 level-2 units answers about 13 KB (History card thumbnails, #1673 item 48a).- Another organization's result is 404
E_NOT_FOUND; anadmin_leveloutside 0 to 2 is 422.
Result vector tiles — GET /v1/tiles/results/{id}/{z}/{x}/{y}.mvt (issues #1472, #1473)¶
Serves those rows as Mapbox Vector Tiles (application/vnd.mapbox-vector-tile), encoded by
PostGIS (ST_TileEnvelope → ST_AsMVTGeom → ST_AsMVT; no tile server). z/x/y are XYZ
(row 0 is north). One layer per tile; with no layer the zoom picks it:
| Zoom | Layer served |
|---|---|
| 0–3 | admin, GADM level 0 |
| 4–6 | admin, level 1 |
| 7–8 | admin, level 2 |
| 9–10 | h3, resolution min(z − 2, finest stored) |
| ≥ 11 | assets, one point per asset |
layer=admin|h3|assets with an optional level (admin 0–2, H3 0–8; assets takes none)
overrides the table — the UI's level switch. level alone past z 10 selects H3, as it did
before the assets tier. An H3 level is clamped to the result's stored resolutions
(metadata.cells.resolutions): the largest at or below the target, else the coarsest.
Features: admin carries boundary_id, iso3, name, admin_level, eai, points (polygons from
admin_boundaries, simplified to half a pixel at z); h3 carries
h3_index, resolution, eai, points; assets carries exposure_id, eai, value on a point.
Where the result holds a band, an assets feature also carries band and an h3 feature
band_max and band_value_weighted (issue #1685, next section).
The tile guard (issue #1804, EX-26). A slow tile must not take the API down: on 2026-10-12
fifteen tile statements ran for minutes, held every pooled connection, and every other route
answered 503 E_SERVICE_BUSY. Every vector tile route — this one and the exposure and portfolio
tiles below, all of which encode through VectorTileRepository.encode — therefore runs under two
bounds, set by three settings:
| Setting (env) | Default | What it bounds |
|---|---|---|
vector_tile_statement_timeout_ms (VECTOR_TILE_STATEMENT_TIMEOUT_MS) |
5000 | One tile statement. encode sets statement_timeout for the request's transaction (set_config(…, true), as the org GUC is set), so PostgreSQL cancels a statement that runs past it and the next request on the same pooled connection starts from the server's default. |
vector_tile_max_concurrency (VECTOR_TILE_MAX_CONCURRENCY) |
8 | Tile requests in flight per API process (core/tile_guard.py). The slot is a route-level dependency, taken before the auth dependency checks out a connection and released after the connection is back in the pool, so a waiting or refused tile holds none. |
vector_tile_slot_wait_ms (VECTOR_TILE_SLOT_WAIT_MS) |
2000 | How long a tile request past the cap waits for a slot before it is refused; 0 refuses it at once. One map view asks for more tiles at once than the cap (12 a source against 8), so without the wait an ordinary view lost its tiles past the eighth. |
Either refusal is 503 E_SERVICE_BUSY with Retry-After, told apart by details.reason:
tile_query_timeout (the statement was cancelled; the driver's message is never served) or
tile_concurrency_cap (every slot was still taken after vector_tile_slot_wait_ms). The map does
not ask again for a tile answered 503: it draws that tile on the next pan or zoom, which is why
a tile past the cap waits rather than being refused at once. No new error code: both reuse
E_SERVICE_BUSY, whose other raiser, a starved pool checkout, carries no reason. The pool
itself is db_pool_size (DB_POOL_SIZE, 5)
plus db_max_overflow (DB_MAX_OVERFLOW, 10), waited on for db_pool_timeout_seconds
(DB_POOL_TIMEOUT_SECONDS, 30), and the API refuses to start with a cap at or above
db_pool_size + db_max_overflow: at the defaults tiles hold at most 8 of 15 connections and 7
stay free for every other route. The guard bounds a slow tile; it does not make one fast. The COG
proxy (/v1/tiles/cog/…) holds no connection while it waits on titiler and is outside it.
The 2 s bound of a whole-country result (issue #1805, EX-26). An admin tile at level 1 and
at level 2, an h3 tile and an assets tile of a whole-country result each answer in under 2 s
as the application role. What holds it: nothing is aggregated on the request path. The admin
and h3 layers read the rows write_result_cells stored, by (org_id, result_id); a GADM
outline is clipped to the tile before it is simplified, so its cost follows the piece drawn, not
the unit's vertex count; and the assets layer loads the result's rows once and windows them in
memory. On prod the four statements took at most 0.43 s as climate_lama_app (#1803), so this is
a regression bound, not a repair. tests/test_api/test_result_tile_bound.py (slow) holds it on
a 7,896-asset result over Greece with 1, 8 and 14 units at levels 0, 1 and 2 of about 32,000
points a level, the slowest tile at about 0.3 s locally.
Tiles are planned as the application role. The tables a tile reads are
FORCE ROW LEVEL SECURITY, and for a NOBYPASSRLS role PostgreSQL runs the policy before any
operator that is not leakproof. No PostGIS operator is, so on those tables geometry && … and
ST_Intersects are filters for the API and never index conditions, while the owner role and the
test suite's superuser bypass the policy and get a GiST scan. A tile statement must therefore be
fast without a spatial index on exposures or the cell tables (admin_boundaries and
admin_boundary_parts carry no policy and keep theirs), and a timing taken as a superuser says
nothing about the API. The two bound tests run as a throwaway NOSUPERUSER NOBYPASSRLS role with
climate_lama_app's grants.
Exposed value on the h3 layer (issue #1783, EX-14). An h3 feature also carries value:
the summed exposed value of the points the cell aggregates, a number in the result's
value_unit. It is stored as value in result_h3_cells.metrics when the result is written
(core/surface_writer.py::VALUE_KEY, no migration), summed over every point the cell's points
counts, banded or not: the value the engine computed over (after growth) for a computed row,
and the row's own value for a sentinel row (#442). So the cells of one resolution sum to the
result's exposed_value plus its sentinel rows' value, unless the cell cap shed some. A result whose rows mix value units
stores and serves no value (K27, the rule that withholds value_sum on a portfolio's admin
tiles), and neither does a result written before the key was stored: nothing is recomputed or
backfilled (backfill_result_cells writes no value), and no other property of the feature
changes.
Nearest admin level (issue #1783, EX-10). The admin layer serves, per country (iso3),
the units of the asked level — the zoom table's or an explicit level — or, for a country with
no unit loaded at that level in admin_boundaries, the units of its deepest loaded level below
it. A country loaded at levels 0 and 1 therefore draws its level-1 units at z 7–8 rather than an
empty layer, and its level-0 unit if only that is loaded; a country that holds the asked level is
unchanged, and no level above the asked one is ever drawn. Each feature's admin_level names the
level it was drawn from. The same rule serves the exposure and portfolio tiles' admin layer
below (db/repositories/vector_tile_repository.py::_nearest_level).
Avoided-loss difference, ?minus= (issue #1628, AD-7). On an adapted result,
minus=<base result id> adds eai_avoided = base.eai − eai to every feature of whichever layer
the tile serves and keeps eai, so the map can show where a plan saves loss and not only what is
left. The base's figure is found by key, never by row position: boundary_id on admin
(result_admin_cells.admin_boundary_id), h3_index on h3, exposure_id on assets (the two
stored eai_exp arrays, each aligned to its assets as above, then matched on the exposure id).
A feature the base holds no row for carries eai and no eai_avoided: a saving is never made up.
The layer, the zoom table and the caps are the adapted result's own, so a difference tile draws
the features the plain tile draws; on assets the vector_tile_max_points cap keeps the points
with the highest adapted EAI, so in a tile past the cap the assets a measure cut hardest are the
first left out. Both results' stored rows are read as built; nothing is aggregated on the request
path.
minus is accepted only when it is the tile result's own provenance.adapted_from_result_id
(run model § 6), checked in this order:
minus names |
Answer |
|---|---|
| no result of the caller's organization (another organization's id, or none) | 404 E_NOT_FOUND |
| any other result of the organization, the result itself included, or any result when the tile result was adapted from nothing | 422 E_VALIDATION, details.reason = "minus must be this result's adapted_from_result_id" |
the base, with metadata.cells.status not built |
404 E_NOT_FOUND with details.cells_status (the base's) |
A difference has no GeoJSON fallback. Without minus the tile is the one served before, byte
for byte: the three plain statements are unchanged and the difference ones sit beside them
(db/repositories/vector_tile_repository.py, core/vector_tiles.py::check_difference_base).
Score bands of a result's locations (issue #1685, EX-14)¶
A result stores the score band of each location beside its EAI, and serves it at three tiers. No migration: the array sits in the result's archive and the cell keys in JSONB.
| Field | Where | Meaning |
|---|---|---|
band_exp |
the result's NPZ archive (eai_exp_path), beside eai_exp |
One integer per location in the same compute order: 1 to 10, and 0 where the location has no band (a sentinel row, #442, or no intensity at the design return period). It is the band band_distribution counts for that location, from one banding pass (worker/tasks.py::_band_rows) |
band |
each assets tile feature |
The asset's band_exp entry, paired with the asset exactly as eai is (core/result_asset_tiles.py). Absent for an entry of 0 |
band_max |
result_h3_cells.metrics and each h3 tile feature |
The highest band among the cell's banded points |
band_value_weighted |
result_h3_cells.metrics and each h3 tile feature |
The mean band of the cell's banded points weighted by each point's value: Σ band × value over Σ value, by core.scoring.value_weighted_mean with the weights of the result's value_weighted_average_band (the value the engine computed over, or 1 per point when the rows' units mix), rounded to 4 places |
A cell with no banded point has neither key. A result written before the worker stored
band_exp serves no band and its cells neither key: nothing is backfilled
(backfill_result_cells writes no band) and no band is guessed. An adapted run bands nothing
(its band_exp is all 0). The answer-plane surfaces (org_risk_cells) never hold either key.
result_admin_cells holds no band, so an admin unit has no result band (K61).
One asset's and one admin unit's figures per hazard (issue #1685, EX-14, K26)¶
Two reads list one location's stored figures across hazards. Each takes the query of
GET /v1/results less haz_type (the selector exposure_dataset_ids or portfolio_id, the clip
admin_boundary_ids or bbox, cells, exposure_growth_rate, measure_ids,
impact_function_id, engine), finds the newest stored result per hazard type and cell exactly
as that read does (core/stored_results.py), and answers one item per hazard type and cell. A
hazard with no stored result has no item, and neither has an indicator hazard, since no stored
result holds one. Neither read computes, queues a job, or adds two hazards. VIEWER+.
GET /v1/results/assets/{exposure_id}→{exposure_id, items: [{haz_type, scenario_label, horizon_year, result_id, eai, band, value_unit, created_at}]}.eaiis the asset's entry of that result'seai_expandbandits entry ofband_exp, null for 0 and for an archive with noband_exp. The pairing is the asset tiles' (the compute-order join, the same per-process cache), so a figure here equals the one on the tile. A result with no stored array, one whose exposure set drifted from its array, and one that does not hold the asset give no item rather than an error.GET /v1/results/admin-units/{boundary_id}→{boundary_id, admin_level, items: [{haz_type, scenario_label, horizon_year, result_id, eai, points, value_unit, created_at}]}.eaiandpointsare the unit's storedresult_admin_cellsrow of that result, read through the(result_id, admin_boundary_id)unique index with no join onadmin_boundaries. A result holding no row for the unit gives no item.
Refusals reuse the codes of GET /v1/results: no selector or two selectors → 422
E_VALIDATION "name one exposure selector"; the clip refusals of /impact; another
organisation's portfolio → 404, another organisation's dataset → the /impact 422. An
exposure_id that is not one of the selector's rows inside its clip, and a boundary_id that
is not loaded, → 404 E_NOT_FOUND.
The assets layer (issue #1473) has no table behind it: the result's stored eai_exp array is
paired with its assets by the compute-order join (core/eai_alignment.py, issue #370) — the
same pairing GET /v1/results/{id}/per-exposure serves, so a point's eai is that route's
value for the same exposure_id. The aligned arrays are cached per process per
(result_id, eai_exp ETag) (result_asset_cache_size, default 8) and single-flighted, so the
tiles of one result share one MinIO read and one alignment query; the pairing and each tile's
window run in a worker thread (core/result_asset_tiles.py). A tile carries at most
vector_tile_max_points points (default 20 000), the highest-EAI ones. layer=assets is
served at any zoom under the same cap, so a small portfolio can show its points early. A
result storing no eai_exp array → 404; an exposure set that drifted from it →
409 E_RESULT_STALE, as on /per-exposure.
Answers: another org's result → 404; a result whose metadata.cells.status is not built
→ 404 E_NOT_FOUND with details.cells_status (null when never stamped) — the UI's cue for
the GeoJSON fallback; out-of-range z/x/y, an unknown layer or level → 422 E_VALIDATION;
a tile nothing reaches → 200 with a zero-length body (a valid empty tile). Nothing is
aggregated on the request path. core/vector_tiles.py owns the zoom table, coordinate validation and the
response helper; the SQL lives in db/repositories/vector_tile_repository.py.
Exposure vector tiles — GET /v1/tiles/exposures/{dataset_id}/{z}/{x}/{y}.mvt (issue #1474)¶
Draws an uploaded exposure dataset of any size before any compute, from its exposures rows
(GiST on geometry, idx_exposures_dataset_id), through the same PostGIS pipeline. One layer
per tile; with no layer, z ≥ 11 serves points and anything coarser bins. layer=points|bins
overrides that at any zoom.
points— one point per exposure row, propertiesexposure_id, value. At mostvector_tile_max_pointsper tile (default 20 000), the largestvaluefirst.bins—ST_HexagonGridhexagons computed at query time in EPSG:3857, about 16 px across at the tile's zoom, propertiescount, value_sum. The grid is anchored at the EPSG:3857 origin and sized fromzalone, so every tile of one zoom shares it. Each hexagon counts the points inside the whole hexagon, not only its part in the tile, so a hexagon straddling a tile edge carries the same totals in both tiles. Its MVT featureidpacks the grid's(i, j)and is the same in every tile of that zoom, which lets a client dedupe the straddlers. A hexagon is clipped to the mercator world before it counts, so one crossing the antimeridian never wraps onto the far side of the map. Bound: 2 s (issue #1805, EX-26), for a whole-country book as the application role. What holds it: the statement readsexposuresonce a tile, and each row finds its own hexagons on the grid, so the cost follows the scope's rows and not rows times hexagons. Until #1805 the scan sat under the per-hexagon loop; as the application role it could not use the GiST index (see "Tiles are planned as the application role" above), so each of a tile's 449 hexagons read every row of the scope again and a portfolio tile over a 7,896-row book ran past the 30 s timeout on prod while the owner role answered in 0.7 s. The hexagons, theirid,countandvalue_sumare unchanged.tests/test_api/test_bins_tile_bound.pyholds it twice: a plan test, in CI's default job, that no scan ofexposuresis repeated per hexagon, and aslowtest serving bothbinstiles over that book (about 0.1 s locally).admin(issue #1686, EX-14) —layer=adminwith an optionallevel0–2: the GADM units of that level reaching the tile, one feature per unit holding at least one row, propertiesboundary_id, iso3, name, admin_level, value_sum, count. The totals cover the rows inside the whole unit, summed at query time over the subdividedadmin_boundary_parts(#1570) with a row on a seam between two parts counted once, so a unit reaching several tiles carries the same totals in each; its MVT featureid(the unit's uuid cut to 60 bits) is the same in every tile. An absentlevelfollows the result tiles' zoom table (z 0–3 level 0, z 4–6 level 1, z ≥ 7 level 2). A country with no unit loaded at the level draws its deepest loaded level below it (the result tiles' nearest-level rule, issue #1783), so each feature'sadmin_levelis the level drawn, not always the one asked.adminis never zoom-picked: the UI asks for it below z 9,binsat z 9–10 andpointsfrom z 11. A z 5level=1tile over 50 000 points in Attica answers in about 0.75 s locally (bound 2 s).
No personal-data field (ADR-041) is a tile property. Answers: another org's or an unknown
dataset → 404 E_NOT_FOUND; out-of-range z/x/y, an unknown layer, a level outside 0–2 or a
level on points/bins (the zoom-picked layer included) → 422 E_VALIDATION; a tile no row
reaches → 200 with a zero-length body. The zoom rule and bin size live in
core/exposure_tiles.py; no migration and no write-time aggregate back it.
Portfolio vector tiles — GET /v1/tiles/portfolios/{portfolio_id}/{z}/{x}/{y}.mvt (issue #1598)¶
Draws a portfolio before any compute (EX-14). A portfolio is a selection over exposure datasets,
so its tile is not any dataset's: the rows are the portfolio's members, resolved by the shared
membership predicate (core/portfolio_membership.py). An all attachment draws its whole
dataset, a bbox attachment only the rows inside its box, a rows attachment only the rows it
lists; an unattached dataset's rows never appear. The layers, zoom rule, bin grid and properties
are the exposure tiles' above: points (exposure_id, value) from z 11, bins
(count, value_sum) below, layer=points|bins overriding the zoom, and layer=admin with an
optional level serving the members' totals per GADM unit (issue #1686). A portfolio whose
members hold more than one value_unit serves its admin units with count and no
value_sum (K27). No personal-data field (ADR-041) is a tile property and there is no cluster
layer. Answers: another org's or an unknown portfolio → 404 E_NOT_FOUND; out-of-range z/x/y,
an unknown layer, a level outside 0–2 or a level on points/bins → 422 E_VALIDATION
(details.layers names points, bins and admin); a tile no member reaches → 200 with a
zero-length body. The bins statement is the exposure tiles' with the membership predicate in
place of the dataset, so the same 2 s bound and the same single read of exposures hold it
(issue #1805): the whole-country portfolio that timed out on prod, one attachment selecting 216
rows of a 7,896-row dataset, is the slow test's portfolio. The predicate is evaluated once per
row of the attached dataset inside the box the tile's hexagons reach, never once per hexagon.
Scope score vector tiles — GET /v1/tiles/scope-scores/{hazard_dataset_id}/{z}/{x}/{y}.mvt (issue #1833)¶
Draws a scope's score on an indicator hazard (an FWI wildfire pack, a heatwave-days cube) by
region, where no money result exists for the result tiles to draw. The scope is an exposure
selector (exposure_dataset_ids, repeated, or portfolio_id) and its clip (admin_boundary_ids,
repeated, or bbox=w,s,e,n), refused as /impact refuses them; the cell is scenario and
horizon_year (absent: the baseline, Today), and a scenario × year cube answers only from the
slice the cell resolves to. Layer admin holds one feature per GADM unit of level (1 or 2, the
result tiles' tiers below the country; absent, z 0-6 draws level 1 and z >= 7 level 2; a country
with no unit at the level draws its deepest loaded level below it) holding a scored member:
boundary_id, iso3, name, admin_level, band (1 to 10), label, score (the unrounded
value-weighted mean band that band rounds half up) and rows_sampled, with an MVT feature id
stable across tiles. The members are the scope's rollup_max_portfolio_members largest rows
inside the clip, the set POST /v1/compute/score-delta scores; each reads the dataset's raster
once per tile through core/indicator_sampling.indicator_reads and each unit's readings are
reduced by score_indicator_sample, the stored indicator_score's own reduction. Members that
mix value units weigh 1 each (K27), decided over the whole member set, so a unit reads the same in
every tile. Nothing is banded in the browser and no money figure is served. The member read runs
under the vector tile guard (statement timeout, concurrency cap, issue #1804). Answers: an
unknown or another org's hazard dataset, one without a COG and a money hazard → 404 E_NOT_FOUND;
no selector, a refused clip, out-of-range z/x/y or a level other than 1 or 2 → 422
E_VALIDATION; a tile no scored member reaches, and every tile while score bands are disabled →
200 with a zero-length body.
6. Key Implementation Patterns¶
Hazard Ingest Flow¶
- API receives
POST /v1/hazards/ingest-> creates Job (PENDING) -> dispatches Celery task - Worker reads GeoTIFF(s) via rasterio, extracts intensity bands
- Converts return periods to marginal frequencies (ADR 018)
- Stores sparse intensity matrix as NPZ in MinIO
- Bulk inserts centroids via SQL
unnest()(ADR 022 -- 630K in ~1 sec vs 30+ min with ORM) - Creates HazardDataset + HazardEvent rows
- Marks Job COMPLETED
Hazard Ingest Input Modes¶
POST /v1/hazards/ingest admits exactly one of three input modes. Naming
two is a 422, never a resolution in either direction.
| Mode | Body | Event axis |
|---|---|---|
| 1 -- multi-band pack | file_path + return_periods |
Return periods; one hazard_events row per band |
| 2 -- per-RP files (JRC layout) | files: [[path, rp], ...] |
Return periods; one row per file |
| 3 -- single observed event | file_path + event |
One row, no frequencies |
A file_path ending .h5/.hdf5 dispatches with
params["source_format"] = "climada_hdf5" (issue #1174) in modes 1 and 3, so
the chord resolves the CLIMADA adapter rather than the GeoTIFF one; mode 2 is
the JRC one-file-per-RP layout and is never a CLIMADA store. The decision is
made on the suffix here — unlike POST /v1/hazards/upload, which sniffs
the magic bytes — because file_path names a path on the worker's
filesystem that the API process is not guaranteed to be able to open. A
mis-named file is still refused, in the worker, with
E_HAZARD_INVALID_SOURCE.
Mode 3 -- single-event snapshot (issue #1020)¶
One observed raster -- a ShakeMap PGA grid, a hotspot grid -- rather than a return-period pack:
{
"name": "M6.8 Aegean ShakeMap",
"haz_type": "earthquake",
"file_path": "/data/shakemap_pga.tif",
"event": {"label": "us7000abcd", "valid_at": "2026-08-30T04:12:00Z"},
"live_event_id": "…"
}
event.labelis the event's identity and reacheshazard_events.event_name. It never reachesscenario, which means an RCP/SSP forcing label to every reader of it (the same rule #816 established for storm-catalogue ids).labelis optional: omitting it commits ADR-068's fully unlabelled row, named"field".event.valid_atis stamped ontohazard_datasets.valid_at. Phase 13 puts time on the dataset, not on a new surface shape -- a snapshot is an ordinaryhazard_datasetsrow keyed by its dataset.live_event_idis optional and must name alive_eventsrow visible to the caller's organization; anything else is a404(absent and another tenant's are deliberately indistinguishable). It is stamped ontohazard_datasets.live_event_id.- The source must be single-band. A multi-band file in this mode means the
caller meant mode 1 and dropped its
return_periods, and is refused.
The NULL-frequency contract. A single-event dataset is a deterministic realization, so it carries no annual exceedance frequency:
- the one
hazard_eventsrow hasrp,exceedance_freqandmarginal_freqall NULL -- ADR-065's second shape, widened by ADR-068 to admit a row with noscenarioand noyeareither. There is no sentinelrp;ck_hazard_events_event_shapemakes writing one impossible, anduq_hazard_events_dataset_single_fieldcaps such rows at one per dataset; hazard_datasets.return_periodsis NULL, not[]and not[0.0]-- the 0-year return period issue #791 removed;- the dataset's event axis is therefore
SCENARIO_YEAR, so it is structurally excluded from expected annual impact:POST /v1/compute/impactrefuses it with422 E_HAZARD_DATASET_NO_EVENT_FREQUENCIES, and the catalog reports it as non-computable through the same classifier. Read it through the score card (GET /v1/risk/lookup) instead.
Everything else about the ingest is unchanged: the same chord runs, the COG is
built from the committed intensity matrix and cog_path is populated exactly
as on the return-period lane. Internally the mode is declared onto the
ingest_jobs.staged_source descriptor by validate_source, so a resumed chord
re-plans and re-derives the identical axis from persisted state.
intensity_unit override (issue #1021)¶
IngestRequest.intensity_unit is an optional string, orthogonal to the three
input modes above and to indicator — it works with any mode and neither
selects nor restricts either axis. Left unset (the default), the ingest is
labelled with the registered ingestor's own unit exactly as before — a
wildfire raster still stamps "FWI". When set, it overrides that default and
must be one of seeded_units_for(haz_type) (core/impact_function_seeder.py)
for the peril, e.g. "K" for wildfire: assert_ingest_unit_compatible
refuses anything else with 422 E_VALIDATION before a job is even queued. A
wildfire raster genuinely measured in Kelvin can therefore be declared
intensity_unit: "K" and — with indicator: false — ingests as
EAI-eligible against the seeded Lüthi curve instead of being forced through
the FWI indicator-only path; the override is stamped straight onto
hazard_datasets.intensity_unit and flows through aggregate_and_commit
unchanged. No unit conversion happens on this path — the caller is asserting
the file's values are already in the target unit — and no new ingestor class
or source_format is involved.
Hazard Upload -- CLIMADA Hazard HDF5 (issue #1174)¶
POST /v1/hazards/upload accepts a CLIMADA Hazard store alongside the
GeoTIFF it has always taken, and POST /v1/hazards/ingest routes a pre-staged
one to the same adapter. The full format matrix and refusal table are W1.4c;
what follows is the request contract.
The format is chosen on the file's magic bytes. .h5/.hdf5 join the
accepted suffixes, but the suffix stays a request pre-filter: a store uploaded
as flood.tif is still read as HDF5, and a GeoTIFF named flood.h5 is refused
as malformed (400 INVALID_FILE_TYPE) rather than handed to the GeoTIFF
adapter under a name that says otherwise. Same rule as POST /v1/exposures
(issue #1172). A recognised store sets params["source_format"] =
"climada_hdf5", which is what makes worker/tasks.py resolve the CLIMADA
adapter instead of falling through the peril chain.
Three new multipart fields, all optional:
| Field | Meaning | Default |
|---|---|---|
crs |
The frame the grid is in. Only EPSG:4326 is accepted (case-insensitively); anything else is a 422 — nothing on this path reprojects and a CLIMADA store states its CRS only in a pickled attribute the reader will not decode. |
EPSG:4326 |
single_event |
Declare that the store holds one observed event rather than a return-period set. Mutually exclusive with return_periods. |
false |
event_label |
The operator's identity for that event, e.g. us7000abcd. Wins over the label the file carries. |
the file's own |
return_periods is now optional — for a CLIMADA store only. A GeoTIFF
states no frequencies of its own, so omitting the field there is still a 400
naming the format. For a CLIMADA return-period set the axis is the unrounded
1/frequency the file implies; supplying return_periods anyway is a
cross-check, never an override, and a rounded 50 against the file's
49.0196… is refused.
The two admitted shapes are declared, not guessed. A one-event store is
vacuously both (a single frequency is trivially monotonic), so
single_event is what disambiguates it. A shape-(b) store dispatches on the
existing mode-3 seam — params["event"] = {"label", "valid_at"}, with
valid_at decoded from CLIMADA's proleptic Gregorian ordinal — and carries no
return_periods key at all. A store whose date is absent, or is CLIMADA's
unset zeros, is refused: dating an undated observation from this server's
clock would fabricate the one fact the row exists to record.
Guards run after staging, on metadata reads only — intensity/shape,
frequency, the centroid vectors — never a densification, through the same
core/ingest/climada_hazard.py::inspect_climada_hazard the worker's adapter
uses, so the endpoint and the adapter cannot drift on what they admit.
Refusals split by fault class (phase-14 guardrail 5):
| Fault | Status | Code |
|---|---|---|
Suffix outside the accept-list; .h5 name with no HDF5 signature |
400 | INVALID_FILE_TYPE |
Irregular grid, non-identity fraction, neither admitted shape (e.g. a dated multi-event catalogue — every event sharing one frequency, so it fails the strict-monotonicity test — refused toward the Phase 15 event-set track), undated single event, haz_type or unit that is not the peril being ingested, contradicted return_periods |
400 | INVALID_FILE_TYPE, carrying the reader's own reason |
return_periods omitted for a GeoTIFF |
400 | E_VALIDATION |
Over hazard_h5_max_events (default 100) on n_events, or over hazard_ingest_max_grid_cells on n_centroids — the file's grid, the same quantity that ceiling bounds for a GeoTIFF or NetCDF. One ceiling per axis (#1225); their product is bounded by hazard_h5_max_densified_bytes (#1239) |
413 | FILE_TOO_LARGE |
| Densified size would exceed the uploader's quota | 413 | QUOTA_EXCEEDED |
crs ≠ EPSG:4326; single_event with return_periods |
422 | E_VALIDATION |
The quota is asked twice, and neither answer is the ledger. A CLIMADA store is a compressed CSR, so the bytes on the wire say nothing about the matrix inside it — the committed 3-event × 12-centroid fixture is 14,512 bytes on the wire and 144 densified, and a sparse 40-event store over a 900-cell grid is 35,008 on the wire against 144,000 densified. The two halves of the question therefore use different numbers:
- Can this body reach the disk?
core/upload_streaming.pyanswers on the wire bytes, mid-body, which is what bounds the disk write (#1170). - Should this dataset be admitted? Answered on
n_events × n_centroids × 4, an upper bound on the matrix.
Both compare against the same used_bytes, read once before staging, so
nothing stacks.
The ledger is hazard_datasets.file_size_bytes, and it holds the uploaded
source's byte count — for every format, written once by
worker.tasks.ingest_hazard when it materialises the placeholder row, and
never rewritten (issue #1228). The column's own comment in
models/hazard.py carries the decision; the settings pair, migration 0027 and
the exposure sibling exposure_datasets.file_size_bytes all denominate in the
same unit.
That the densified bound is not the ledger is what makes the lifecycle
sound. The row is stamped before the chord is dispatched, so any number written
there must stay true whether the ingest commits or fails. #1174 briefly wrote
the request's estimate and had aggregate_and_commit._finalize_dataset
overwrite it with the intensity NPZ's size on commit — but only on commit, so
an .h5 ingest that failed after dispatch (a worker-side re-guard
disagreement, enforce_valid_cell_floor, a MinIO blip) kept the estimate
forever, and with the two ceilings bounding one axis each and not their product
(#1225) the only thing bounding that estimate is the quota itself: a
few-kilobyte CSR could stand as a permanent multi-GiB charge, refusing every
later upload with 413 QUOTA_EXCEEDED until the user noticed and deleted the
failed dataset. Uploaded bytes need no correction on either path. The stored
artifact's size is recorded on the ingest manifest, where a question about the
artifact belongs.
The staged file is reclaimed api-side, and the rule is "the creator
reclaims" (issue #1237).
An ingress-denominated quota bounds /uploads only if the bytes it counts are
released once the source is in object storage, and until #1237 nothing on
POST /v1/hazards/upload released them: the staged file survived every ingest,
so the volume grew monotonically while the quota re-charged the same 5 GiB
budget forever.
The obvious fix is the wrong one, and #1228 shipped and reverted it. Handing
stage_source this request's <uuid4> staging leaf through cleanup_local_dir
does not work: the api container runs as root (docker/core.Dockerfile
declares no USER), so the leaf is created root:root 0755, while the worker
has dropped to the unprivileged celery user and
docker/worker-entrypoint.sh chowns /uploads non-recursively on purpose —
"the api's staged upload dirs stay root-owned 0755, which the worker only ever
reads". shutil.rmtree raises PermissionError there and _cleanup_local_dir
swallows OSError, so the reclaim looks wired and does nothing. The same
argument rules out a Celery beat task: the beat service in
docker-compose.yml declares no volumes: at all, and a beat schedule is
executed by the worker — the one container with the mount and the wrong uid.
worker/dataset_ingest.py may set cleanup_local_dir precisely because the
worker downloaded the directory it names.
So the reclaim runs in the process that created the leaves and holds a writable
mount — the api. core/hazard_upload_reclaim.py sweeps
settings.hazard_upload_dir every hazard_upload_reclaim_sweep_minutes
(started by main.lifespan, first pass immediately on boot so a restart does
not wait out an interval), and decides each leaf on the state of the ingest job
that owns it, read per organization through the RLS-scoped jobs and
ingest_jobs tables:
ingest_jobs state |
Leaf |
|---|---|
succeeded / failed / cancelled (IngestJobStatus.is_terminal) |
reclaimed — stage_source has copied the source into MinIO on every one of those branches, and the failure path keeps its object-store staging for forensic re-runs (#296) |
pending / running / chunking / writing / aggregating / committed |
kept — a worker is about to read it, or is reading it. committed is not terminal; it precedes succeeded |
| no job references it | kept until it is older than hazard_upload_orphan_ttl_seconds (default 1 h) — the window between minting the leaf and committing the jobs row, plus true orphans from a failed dispatch |
The lane matters, and it is IngestJobStatus — not JobStatus. The upload
drives two rows: an ingest_jobs row (GET /v1/hazards/ingest-jobs/{id},
states above) and the generic jobs row (GET /v1/jobs/{id},
completed/failed). Waiting on the wrong one is silent: there is no
JobStatus.SUCCEEDED, so a sweep keyed on it reclaims nothing — the same inert
no-op #1228 shipped. The ingest lane decides because it closes first and is the
only one that records a cancellation: POST /ingest-jobs/{id}/cancel writes
ingest_jobs and nothing else, so a reclaim reading only the generic row would
hold a cancelled upload's bytes until the 24-hour stale-job backstop fired. The
generic row is kept as the backstop in the other direction — an ingest_jobs
row wedged in a non-terminal state is still released once
worker.job_watchdog reaps its jobs row, and that watchdog is floored at
ingest_chord_timeout_seconds so it cannot outrun a live ingest. Either lane
going terminal is enough.
Two invariants make the sweep safe to run against a shared volume. Only a
direct child of the staging root whose name parses as a UUID is a candidate
— the exact shape core/upload_streaming.py::staged_upload_path mints — so
settings.dataset_source_download_dir (/uploads/dataset-ingest, the worker's)
is invisible to it rather than merely spared by the age check. And the root
itself is never passed to a delete: removal goes through
discard_staged_upload, the same primitive the route's rejection path uses,
and _remove_leaf raises rather than act on anything whose parent is not the
root.
A third invariant is easy to get wrong and is worth stating: the orphan TTL is measured from the newest mtime among the leaf and its files, never from the directory alone. A directory's mtime advances only when an entry is created, renamed or removed inside it — not while an existing file is being written — so a leaf's own clock freezes at the first staged byte. Anchoring on it would let a slow or very large upload age past the TTL while it was still streaming, and the sweep would delete a leaf the route was about to hand the worker.
cleanup_local_dir therefore stays unset on this route — deliberately,
and the pinning test in tests/test_api/test_hazard_h5_upload.py now asserts
the api-side reclaim in its place.
Two consequences follow from the ledger's meaning, and both are accepted rather than overlooked. Neither is changed by the reclaim above, which releases the staged bytes and touches neither the column nor the quota.
Admission is stricter than accounting. The densified bound and the staged size differ, so an upload the endpoint refuses may well have fitted, and a user's recorded total is smaller than the sum of the bounds they were admitted against. That is the conservative direction on purpose: nothing at request time can tell a dense CLIMADA source from a sparse one without reading the whole matrix, which is exactly the read the upload path exists to avoid, and the alternative to a pessimistic bound is discovering the overflow part-way through a long-running ingest.
Accounting does not bound storage. An ingress quota bounds what a user pushes
into the platform, not what the platform ends up holding, and the two are
not ordered — the committed intensity NPZ can be larger than the compressed
GeoTIFF or CLIMADA CSR it came from, so a user with dense rasters can occupy
more object storage than their recorded total says. This is the price of a
ledger the enforcement point can actually weigh: stream_upload_to_path sees
wire bytes mid-body and nothing else, and bounding ingress is what the cap was
added for (#1170). Bounding storage is a different mechanism — a sweep over
npz_path sizes, or a post-commit reconciliation — not this column. #1174's
attempt to get storage accounting out of it instead produced a column that
meant the uploaded figure before a commit and the stored figure after one, with
no correction at all on the failure path.
Hazard Upload -- declaring horizon years (issue #1189)¶
POST /v1/hazards/upload is the browser-facing direct-upload form (the
counterpart of POST /v1/hazards/ingest, which takes a pre-staged server
path). Two of its multipart fields say which horizon years the uploaded raster
represents:
| Field | Value | Stored as |
|---|---|---|
supported_years |
Comma-separated years, e.g. "2030,2050" |
hazard_datasets.supported_years = {2030,2050} |
reference_year |
A single year, e.g. "2050" — alias for the above |
{2050} |
| neither | — | NULL |
Both are parsed by core/hazard_years.py::parse_supported_years, which
normalises to an ascending, de-duplicated list[int] — the exact column shape
HazardRepository.list_datasets's year_range filter reads. Supplying both
fields is fine when they name the same year and a 422 when they disagree;
neither is ever preferred over the other.
Omitting the field is the wildcard, and that is deliberate. An absent
supported_years — NULL or an empty {} — is not a positive claim, so the
catalogue keeps such a dataset at every horizon (issue #877, and the same
rule the point lookup's ?horizon= applies). Declaring years makes the dataset
a positive claim instead: it answers the horizons it names and is excluded from
the ones it does not. Before this field existed, every uploaded raster landed
NULL, so a 2050-only upload was offered as an answer for 2030 — the field
closes that, and an upload that omits it behaves exactly as it always did.
Validation is at the boundary, ahead of the quota check and the staging write.
Each of these is refused with 422 E_VALIDATION (ValidationError through the
ADR-028 handler):
- a non-integer entry (
"twenty-thirty","2050.0"); - a value that names no year at all (
","); - a
reference_yearcarrying a list; supported_yearsandreference_yearnaming different years;- a year of
0, or one wider than the column'sint4limit; - a field longer than 256 characters — nothing upstream bounds a multipart form field, and this parser is synchronous on the event loop.
A malformed entry is never dropped and never defaulted. A blank field, though,
reads as absent rather than malformed: an untouched optional text input posts
"", and that means the uploader declared nothing.
Note that the sibling return_periods field on the same endpoint still answers
a malformed value with 400 (its own _parse_return_periods raises
HTTPException directly). The year fields deliberately use the 422
ValidationError convention that assert_known_region and
assert_ingest_unit_compatible already use here; reconciling return_periods
onto it is a separate change.
The parsed list rides params["supported_years"] to worker.tasks.ingest_hazard,
which stamps it onto the placeholder row at creation. It survives the commit
because aggregate_and_commit._finalize_dataset writes the scenario/year
column group only on the SCENARIO_YEAR axis, and an uploaded GeoTIFF is
always RETURN_PERIOD (ADR-067).
Async Ingest Pipeline¶
The hazard ingest path uses a Celery chord so the per-chunk NPZ writes
run in parallel. The full chord is assembled dynamically because the
number of chunks (N) is only known after plan_chunks runs.
Chord structure
chain(
stage_source ← preface 1: copy local files into MinIO
validate_source ← preface 2: metadata sanity checks
plan_chunks ← preface 3: deterministic chunk list
_dispatch_chunk_writes ← dispatcher: builds and fires the inner chord
└── chord(
group(write_chunk × N), ← parallel body
aggregate_and_commit ← callback
)
)
The linear preface runs in series on a single worker. The dispatcher is
itself a Celery task; it issues apply_async() for the inner chord and
returns immediately. aggregate_and_commit is the callback that fires
once all write_chunk tasks complete.
State machine
pending → running → chunking → writing → aggregating → committed → succeeded
↘ ↘ ↘ ↘ ↘
failed/ failed/ failed/ failed/ failed/
cancelled cancelled cancelled cancelled cancelled
failed and cancelled are reachable from every non-terminal state.
Terminal states (succeeded, failed, cancelled) have no outgoing
edges.
Resume guarantee
plan_chunks produces a deterministic chunk list from the staged
descriptor (same input → same chunk boundaries). write_chunk is
idempotent: it skips any chunk whose row already has status =
'succeeded'. Together these two properties make it safe to re-dispatch
the chord on a worker crash — already-completed chunks are no-ops and
only the remaining chunks are re-run.
Determinism is conditional on the same inputs, and the request's
bbox is one of them: since #445 the planner sizes windows, the
hazard_ingest_max_grid_cells fail-fast and the window boundaries
against the bbox-clipped extent. The clip is therefore persisted on
ingest_jobs.bbox (migration 0062) rather than living only in the
original chord's Celery message, and both planning entry points
(plan_chunks, select_chunks_for_resume) resolve it from the row —
their bbox parameter is a pure override. Without that, a resume would
re-plan against the raw file and write a different window into a chunk
slot that ingest_chunks addresses by chunk_idx alone (#453).
The request's resolution (degrees/pixel) is the same class of input, one
layer down: it does not feed the plan, but every write_chunk read derives
its GridSpec from it — grid_width / grid_height, and therefore the
pixel_idx space each chunk's NPZ slice is expressed in. It is persisted on
ingest_jobs.resolution (migration 0064) and resolved from the row by
select_chunks_for_resume, its parameter likewise a pure override. Without
that, a resume of a resampled ingest wrote native-resolution slices into a
dataset whose already-succeeded chunks were resampled, and
aggregate_and_commit unioned two different grid bases (#482). NULL on both
columns means "un-clipped" / "native resolution" — the correct value for
every historical row.
The same rule applies to plan_chunks's other post-#445 parameter, for
the same reason: a chord dispatched before a deploy carries the old
three-argument message, so an omitted job_id must not mean "there is no
legacy Job row" — a rejected plan would leave the row that
GET /v1/jobs/{id} polls stuck at running. It is instead recovered from
the jobs.params->>'ingest_job_id' back-link that both ingest producers
stamp (#454). The reverse skew (a new, longer message reaching an old
worker) is a TypeError that no amount of defaulting fixes — see
Deployment → In-flight Celery messages.
Cancel semantics
POST /v1/hazards/ingest-jobs/{id}/cancel writes status='cancelled'
on the ingest_jobs row. Cancellation is chunk-boundary only: each
write_chunk task polls the row at the start before doing any work and
exits as a no-op if the parent is cancelled. aggregate_and_commit
performs the same poll at the start of the callback. In-flight writes
are always allowed to finish; there is no mid-chunk interrupt.
Log buffer
Each ingest_jobs row carries a logs text column. append_log caps
the buffer at 256 KiB. On overflow the oldest bytes are dropped and a
truncation marker ...[truncated]... is prepended so readers can tell
the buffer was trimmed. The cap is enforced on every write in
IngestJobRepository.append_log (db/repositories/ingest_job_repository.py).
Chunk-size table
Chunk sizes are configured in Settings.hazard_ingest_chunk_size_bytes_per_haz_type
(config.py). The defaults reflect the per-event memory footprint of
each hazard type:
| Hazard type | Chunk size |
|---|---|
river_flood |
256 MiB |
tropical_cyclone |
512 MiB |
wildfire |
256 MiB |
Tropical-cyclone is larger because its per-event footprint is roughly
half that of the other types, so a 512 MiB chunk fits the same worker
RAM budget while halving chord fan-out overhead. Unlisted types fall
back to hazard_ingest_chunk_size_bytes_default (256 MiB). Chord width
is bounded by hazard_ingest_max_parallel_chunks (default 8) to keep
aggregate memory bounded under burst. The decisions behind this design
are captured in ADR-027, ADR-029, and ADR-034.
Approved-Dataset Ingest Handoff¶
The weekly DatasetSource poll (issue #321) queues new/changed external
items into pending_ingests as awaiting_review. When a platform admin
approves a row, POST /v1/admin/pending-ingests/{id}/approve dispatches the
ingest_approved_dataset Celery task, which runs the handoff into the async
ingest pipeline above: it resolves the item's ingest descriptor, downloads the
rasters, and dispatches ingest_hazard.
Ingest-descriptor contract
The queue row carries only source_id / item_id and the item's opaque
metadata — the backbone never interprets that metadata while polling or
reviewing. To make an item auto-ingestable on approve, a source publishes an
ingest object inside that item's metadata, in one of two raster layouts:
{
"id": "jrc-flood-eu",
"title": "JRC EU river flood",
"download_url": "https://.../flood_hazard/",
"ingest": {
"haz_type": "river_flood",
"files": [
{ "rp": 10, "url": "https://.../Europe_RP10_filled_depth.tif" },
{ "rp": 100, "url": "https://.../Europe_RP100_filled_depth.tif" }
]
}
}
| Field | Requirement |
|---|---|
haz_type |
A HazardType value (river_flood, tropical_cyclone, wildfire, storm_europe). Absent/unknown ⇒ not auto-ingestable. |
files |
Mode 2. Non-empty list of {rp, url} — one single-band raster per return period, return periods unique and positive. |
return_periods |
Mode 1. Non-empty list of positive years, one per band of the item's single multi-band raster at download_url. |
Exactly one of files / return_periods may be given; supplying both is
refused rather than silently ranked. Mode 2 exists because it is what real
hazard publishers serve — the JRC river-flood distributions publish one
*_RP<N>_*.tif per return period in a directory, not a stacked raster.
Modelling only mode 1 was why no source could satisfy this contract (#830).
Descriptors are derived, not hand-maintained. JrcFloodSource reads the
Apache autoindex its DCAT download URL resolves to and turns each
*RP<years>*.tif it lists into one {rp, url} pair, so the return periods come
from the provider. Derivation never guesses: an unreachable listing, a non-HTML
body, or a distribution that publishes its return periods as tiled
sub-directories (the global GLOFAS package does) yields no descriptor and the
item stays reviewable-but-not-auto-ingestable.
Because the queue row's metadata is a poll-time snapshot, a row queued
before its source learned to publish a descriptor carries none. The handoff
therefore re-resolves the descriptor against the live source
(DatasetSource.ingest_descriptor) before giving up, so an old row still
ingests without waiting for a re-poll.
The contract and its parser live in
ingest/sources/base.py (IngestDescriptor, IngestFile,
ingest_descriptor_from_metadata). Only the ingest block is read;
everything else in the item metadata stays opaque.
Every approval row names its asking organisation (issue #1669, SH-26)
Migration 0119 gives pending_ingests an asking org_id, the
requested_by_user_id, licence_accepted_at/licence_accepted_by_user_id,
withdrawn_at and dismissed_at, and a licence status. Every
GET /v1/admin/pending-ingests row carries org_id, org_name and
requested_by_user_id — all null on a row the weekly poll found, which keeps
its review flow. An organisation's request (a boundary upload, or a catalogue
request-ingest once its licence is accepted in /v1/admin/data-requests)
reaches this queue as awaiting_review; a withdrawn one leaves it, and
approve/reject answer 404 for it. Approving an organisation's request
dispatches its ingest for that organisation and its asker (a poll row keeps
the reviewing admin's), so the dataset lands in the asker's Data and its
completed job reads ready there. The upgrade copied org_id and
requested_by_user_id from metadata onto the boundary files #1670 queued
before it.
Refused runs: compute.refused (issue #1669, PF-4)
A compute refused at submission queues no job. Each of POST /v1/compute/impact,
/impact/matrix and /impact/compare therefore records a refusal once per
refused request as an audit event compute.refused (resource_type
compute): metadata holds route, code, reason (details.reason, else
the message), exposure ({selector, id, name} — portfolio,
exposure_dataset or org), haz_type, hazard_dataset_id and cells, and
never a figure. A refusal is a domain error or an HTTPException carrying a
registry code, except the auth, rate-limit and platform-failure codes; a body
FastAPI's own validation refuses never reaches the route and records nothing.
GET /v1/admin/failures lists these as run_refused.
Sources that can never be auto-ingested say so
A source whose items are structurally not hazard rasters sets a class-level
auto_ingest_unsupported_reason. WorldpopSource does (it publishes gridded
population — exposure, not hazard intensity). GET /v1/admin/pending-ingests
surfaces this per row as auto_ingestable + auto_ingest_unsupported_reason
so the review queue marks those rows review-only instead of letting a
reviewer discover it from a failed approval, and the handoff reports
handoff: auto_ingest_unsupported with that reason.
Failure is recorded, not silent
A non-ingestable approved item — no ingest block (and none derivable live), a
non-hazard descriptor, a malformed descriptor, or a failed download — flips the
row to the terminal ingest_failed state (migration 0051) and logs the reason
at ERROR. The pipeline is dispatched only for valid hazard descriptors; nothing
is guessed. Because the platform-level queue carries no org, the resulting
hazard dataset and its jobs are created in the approving admin's org
(threaded from the approve request).
Per-approval clip override (issue #1052)
The fetch-time region clip (dataset_source_clip_bbox / dataset_source_clip_region,
above) is process-wide configuration — every approval clips to the same
window and stamps the same region label. POST /v1/admin/pending-ingests/{id}/approve
accepts an optional JSON body that overrides both for that one approval:
Both fields are independently optional, and so is the body itself — an
existing caller sending no body reproduces today's settings-driven bbox and
region byte-for-byte. clip_bbox is [min_lon, min_lat, max_lon, max_lat]
in EPSG:4326, validated by the same rule the dataset_source_clip_bbox
setting's string form uses (core.ingest.clip.validate_clip_bbox: exactly 4
numbers, min_lon < max_lon, min_lat < max_lat, longitude in [-180, 180],
latitude in [-90, 90]); clip_region is validated against the loaded
admin-boundary vocabulary (core.region_vocabulary.assert_known_region,
above). An invalid override is a 422 — the row's status is left untouched and
no ingest task is dispatched, exactly as if the approval had not been
attempted. The validated overrides ride the ingest_approved_dataset
dispatch as two new optional task parameters, so a call already on the queue
from before this issue (which never sends them) is unaffected.
No decision is a dead end for the dataset (issue #870)
The poll writes a dataset_source_catalog last-seen marker for every item
it sees, not just the ones it queues. That is what keeps a declined item from
nagging admins on every poll — but it also means a rejected or ingest_failed
row is skipped by every later poll until its upstream content changes, so a
mis-click would otherwise put a dataset permanently out of the platform's reach.
POST /v1/admin/pending-ingests/{id}/re-arm is the supported way back: it
deletes that item's catalog marker so the next poll sees the item as new and
queues a fresh awaiting_review row. Platform-admin only, audited as
pending_ingest.rearmed, and idempotent — the response's last_seen_cleared
says whether a marker was actually there. The decided row is left untouched as
the record of the decision, and polling behaviour is unchanged: an item nobody
re-arms is still never re-queued. Re-arm brings an item back through review,
never around it — it dispatches no ingest. Only dead-end rows are re-armable;
awaiting_review (already reachable) and approved (its dataset exists or is
being built) are a 422.
Re-arm is self-completing and batched (issue #905)
Clearing the marker only unblocks the next poll, and the beat is weekly — a
recovery whose effect is invisible for up to seven days is close enough to no
recovery at all. So both re-arm endpoints dispatch a scoped poll after
committing: poll_dataset_source(source_id, item_ids, notify=False), one task
per affected source, narrowed to exactly the re-armed items. Scoping is what
makes it affordable — dispatching the platform-wide poll_dataset_sources was
rejected in #870 because it polls every source and notifies every admin — and
a scoped poll deliberately does not catalog items outside its scope, so it can
never mark an unrelated item as seen. The response's polls_dispatched reports
what went out.
POST /v1/admin/pending-ingests/re-arm takes {"ingest_ids": [...]} (max 500)
for recovering from a bulk sweep. Partial success is the contract: it answers
200 with rearmed / failed / counts, where an id that does not exist or a
row that is not a dead end lands in failed with a not_found /
not_rearmable code while every other id is still re-armed — so read counts,
not the status code. Each re-armed row is audited individually.
Both responses also carry source_registered, and a warning when it is
false. A source can be de-registered from the build while its rows survive in
the queue; re-arming one of those still clears the marker but no poll will ever
re-surface it. Registration is reported, never enforced — the source may be
registered again later, and refusing the re-arm would leave nothing to re-arm
when it is. docs/RUNBOOK.md has the operator procedure.
Centroid Assignment¶
- API receives
POST /v1/hazards/{id}/assign-centroids-> Job -> Celery task - Worker loads all hazard centroids for dataset, loads all exposures
- Builds KDTree, finds nearest centroid per exposure point
- Bulk inserts one
ExposureCentroidAssignmentrow per asset (exposure_id,hazard_dataset_id,array_index)
Assignment also runs automatically at compute time (#558): when a compute
job finds no junction rows for its (hazard dataset, exposure dataset) pair, the
worker assigns them in place — under a pg_advisory_xact_lock so concurrent
scenario-matrix cells on one dataset do not collide — and retries the load before
failing. The endpoint above stays the explicit, pollable path (the UI's Dashboard
preflight uses it), but no client can skip assignment by not calling it.
Impact Calculation Flow¶
- API receives
POST /v1/compute/impactwith{hazard_dataset_id, impact_function_id, save_mat: true} - Creates Job (PENDING) and commits -> dispatches
compute_impactto compute queue (see Job submission ordering) - Worker loads: hazard (from DB + MinIO NPZ), exposures (from DB), impact function (from DB).
The exposure load follows the run model's worker order: (1) the set the
selector names, then (2) the
scopeclip applied insidecompute_order_sql(core/eai_alignment.py), for a list ofadmin_boundary_idsthe union of its units with each row counted once (issue #1588, see "Scope clip on impact compute" below).scope_totalcounts the clipped members only. Step (3) is exposure growth (issue #1593,worker.tasks._grow_exposure): with anexposure_growth_rate, each row's value, deductible and cover are scaled by(1+g)^(horizon_year − reference_year)from its own dataset's epoch, before vulnerability and measures (see "Exposure growth on impact compute" below) - Converts to engine data contracts (dicts) and resolves vulnerability (step 4).
Step (5) applies
measure_ids(issue #1594): each measure, in the job's sorted order, transforms the hazard and the curve throughEngineAdapter.apply_measures, a protection standard removing the event rows withrp <= rp_protect(see "Adaptation measures on impact compute" above). The arrays then pass to the selected engine - Adapter builds
Hazard,Exposures,ImpactFuncSet-> runsImpactCalc.impact() - Stores the
eai_exparray, andband_expbeside it, as compressed NPZ in MinIO - Creates ImpactResult row with EAD, AAI, per-event impacts
- Records the cache key in
result_cache(issue #283 — see below) - Marks Job COMPLETED
Job submission ordering¶
Every API path that queues a job goes through
core/job_dispatch.py (create_and_dispatch_job, or stage_job +
commit_and_dispatch for batches). Two rules, both enforced by
tests/test_api/test_submit_dispatch_ordering.py:
- Commit before dispatch. The
jobsrow (and anything the task needs to read, e.g. the matchingingest_jobsrow or thecompute_batchesrow) is durable beforeapply_asyncis called. A worker can never be handed an id it cannot read. - The API never writes
status. The Celery task id is pre-generated and passed toapply_async(task_id=...), so it lands in the same pre-dispatch INSERT and no post-dispatch write is needed.pending -> running -> terminalbelongs to the worker alone —JobRepositorydeliberately has noset_running.
Issue #578 is what happens without them: the submit path dispatched inside its
open transaction and wrote status='running' afterwards, so a job that failed
in the worker in 8 ms had its failed overwritten by the API's commit ~150 ms
later and sat at running forever.
The one write after dispatch is the failure path: if apply_async raises, the
committed row is marked failed (in a fresh transaction) before the error
propagates, since nothing will ever run it. Committing mid-request ends the
transaction the RLS app.current_org_id GUC was set on, so the helper re-issues
it right after the commit.
Result Cache (issue #283)¶
POST /v1/compute/impact checks result_cache before dispatching the
Celery task. The cache is content-addressed by a deterministic SHA256
over: exposure_dataset_id, exposure_dataset_sha256, hazard_dataset_id,
hazard_dataset_sha256, impact_function_id, impact_function_version,
year, the canonical-JSON of the saved scenario's inputs, discount_rate,
and growth_rate. Further components join the digest only when the
request set them, so every key minted before they existed stays valid: a
non-default engine_name (issue #377), portfolio_id (issue #1008 — two
books of assets over the same hazard and curve compute different losses and
must never share an entry), the scenario/horizon_year pair (issue #1185 —
a horizon-agnostic baseline resolves several horizons to the same dataset,
so hazard_dataset_id alone would let two different cells collide on one
cached answer) and, last, a labelled tail (issue #1568):
membership=<digest>, deductible=<v> and cover=<v>. The tail is labelled
because its entries are independent, and a deductible-only and a cover-only
request with the same number would otherwise hash alike; deductible and
cover are there because an insured and a ground-up run differ. The
membership digest is core.result_cache_service.membership_digest: the
SHA-256 of the portfolio's sorted (exposure_dataset_id, dataset_sha256,
selection kind, canonical selection JSON, weight) attachment tuples, read from
portfolio_datasets (the selection is hashed in Postgres, so a large rows
list never crosses the wire). It is required for every portfolio-addressed
key: a portfolio_id names a portfolio, not what it holds, so attaching,
removing, re-weighting or re-selecting a dataset, or re-ingesting one under
the same id, is a different key and a recompute, never the previous
membership's loss. hazard_dataset_id in the digest is always the
resolved dataset, not the one the caller named. An adapted run (issue
1594) appends measure_ids=, each sorted measure id with the SHA-256 of the¶
canonical JSON of the measure columns its transform reads, so a patched measure is never served a stale adapted result.
A matching entry is served only when its result is visible to the caller's
org, carries the scope-coverage pair (issue #1319), and has its tile cells
built (issue #1838): a result whose tile cells are not built — its
metadata.cells.status is absent (a result computed before #1470) or
"failed" (a failed aggregate write) — is never served from the cache,
because its map tiles would all answer 404.
Both lookups apply these rules — try_get_cached_result in front of
POST /v1/compute/impact and lookup_cached_result_sync in the worker's reuse
step. Such a hit is a miss: the run recomputes, writes its cells, and the
worker's last-writer-wins upsert repoints the entry at the new result. The key
is unchanged, and no entry or result is deleted.
Scenario x horizon on impact compute (issue #1185)¶
POST /v1/compute/impact accepts an optional scenario + horizon_year pair
that addresses the run as a scenario-matrix cell instead of as a dataset id.
Supplied together, the backbone resolves which hazard dataset answers that cell
through the same rule POST /v1/compute/impact/matrix has used since issue
420 (core.compute_batch_service.resolve_cells_within_scope — ADR-048 provider¶
family first, ADR-069 frequency-less datasets excluded, #805 horizon-agnostic baselines resolvable at any horizon). Nothing about the matching rule is new or duplicated here; only the surface it is reachable from.
| Field | Type | Meaning |
|---|---|---|
scenario |
str |
Scenario label to resolve for (e.g. ssp5-8.5, baseline) |
horizon_year |
int |
Horizon year to resolve for (e.g. 2050) |
haz_type |
enum | Hazard type to resolve within — required when hazard_dataset_id is omitted, rejected when it is supplied |
region |
str |
Region to resolve within, on the same terms as haz_type |
scenario (a taxonomy label) is not scenario_id (a saved scenario row the
result is linked back to); the two are independent and may be sent together.
hazard_dataset_id is conditionally required. The rules, each a 422
naming the offending field:
scenarioandhorizon_yearare both-or-neither — half a pair names no cell, and silently dropping the half that was sent is exactly the defect this removed from the Assess panel, where the year slider's value never reached the request at all.- Neither
hazard_dataset_idnor the pair → refused: nothing names a dataset. - Pair present,
hazard_dataset_idsupplied → the id pins only the(haz_type, region)scope and the provider family; the dataset actually computed against is whichever one answers the cell, which may be a sibling of the id that was sent. That is the point — a caller holding a baseline and asking forssp5-8.5at 2050 wants the 2050 slice, not the baseline replayed under a label. - Pair present,
hazard_dataset_idomitted → the caller names the scope withhaz_type(required) andregion.regionmatches exactly, and omitting it means datasets carrying no region at all, not "any region" — the same semantics a matrix gets. That holds when the request names no exposure set; with one, an omittedregionis the region-less resolution below. haz_type/regionalongside an explicithazard_dataset_id→ refused as contradictory; the named dataset already carries both.
Nothing resolves the cell → 422 E_VALIDATION that names the catalog. The
envelope's details carry reason (the same sentence a matrix cell's
resolution_error renders — e.g. that the only ssp2-4.5 dataset declares 2030
and the cell asked for 2100), the scenario/horizon_year/haz_type/region
that were resolved for, and candidates: up to 25 of the datasets that were
considered, each as {id, name, scenario, supported_years}, plus
candidates_total. A caller can therefore see why nothing matched instead of
being told only that something did not.
Region-less resolution from the exposure set (issue #1590, EX-15, EX-19, BC-3)¶
When hazard_dataset_id and region are both omitted and the request names an
exposure set (exposure_dataset_id, exposure_dataset_ids or portfolio_id),
/impact, /impact/matrix and /impact/compare resolve the hazard from the
set's countries (run model § 6). One public core function owns the rule:
core.compute_service.resolve_hazard_option(session, *, org_id, haz_type,
exposure_dataset_ids=(), portfolio_id=None, scope=None) -> HazardOption. It
takes the selector, the optional clip in its stored form and one haz_type, and
the hazard options read route (#1661) and the assistant's list_hazard_options
(#1604) call it too, so they cannot disagree with the compute refusal.
- The set's countries are the #1166 derivation (
largest_overlap_admin0_iso3) over each member'sbbox; a portfolio's members are its attached datasets. Candidates are the datasets ofhaz_typewhose derived country, overCOALESCE(footprint, bbox), lies in that set. - A candidate answers only when its extent covers every member's
bbox. With ascopeclip eachbboxis first intersected with the clip (the union of its units, or its box), and a member the clip misses drops out, so narrowing the clip can satisfy the refusal. One run never stitches per-country datasets. - The covering candidates are the pool the cell then resolves within, through
the same
resolve_cells_within_scoperules as above (provider family, ADR-069, #805), so a covered hazard can still miss a cell with #1185's reason.
HazardOption.state is covered, no_single_dataset or not_covered, with
reason null for covered and otherwise the details.reason the compute
refusal gives:
state |
When | reason |
|---|---|---|
covered |
a candidate covers every member extent | null |
no_single_dataset |
candidates exist, none covers every extent | "no single hazard dataset covers CYP, GRC; narrow the exposure or the clip" |
not_covered |
no dataset of the hazard lies in the set's countries | the same sentence |
A refused request (no single hazard dataset covers <ISO3 list>; narrow the
exposure or the clip, ISO3 codes sorted) is a 422 E_VALIDATION whose details carry reason,
state, haz_type and the sorted countries. Two edges keep older behaviour:
a set none of whose members yields a country (no bbox, as an upload before
EX-18's fill, or no loaded admin-0 boundary under it) keeps the region-NULL pool
above (empty → not_covered, "no hazard dataset without a region answers an
exposure set with no derivable country; send region or hazard_dataset_id"); a clip that misses every member bbox is
not_covered with the worker's own "No exposure row of this run lies inside its
scope clip; there is nothing to compute." An explicit region keeps the exact
match, and a request with no exposure set keeps the region-NULL rule.
Provenance and cache key. A resolved run's job params carry the resolved
hazard_dataset_id and — under the matrix's own key names — scenario_label
and horizon_year, so a scenario-addressed single run describes itself exactly
as a matrix cell does. Both reach the result's provenance block (added only
when present, so older blocks stay shape-identical), and the pair joins the
result-cache key (see above). With both fields absent the path is unchanged
byte for byte: no resolution query runs, job params and cache key are what
they were before this existed.
Cells on /matrix (issue #1587, EX-15). POST /v1/compute/impact/matrix
takes an explicit cells list, [{scenario_label, horizon_year}] with 1 to 24
entries, where horizon_year: null is Today (baseline@today). Each entry is
one run and the list is never expanded, so baseline@today, ssp2-4.5@2050 is two
cells, not four. scenario_labels × horizon_years keeps its Cartesian
meaning, and its batch params are unchanged. The two forms are exclusive:
cells sent with scenario_labels/horizon_years → 422 E_VALIDATION,
details.reason "send cells or scenario_labels × horizon_years, not both". An
empty list, more than 24 cells, a blank label or a repeated cell is a 422 too.
The matrix takes haz_type (plus optional region) in place of
hazard_dataset_id, on /impact's terms: one of the two names the scope,
never both. Without a dataset the impact function and the references are
checked against the first resolved cell's dataset, and a scope where no cell
resolves is a 422 carrying reason and candidates as above. A cells batch
records params.cells instead of the two axes, and a haz_type batch records
haz_type/region with hazard_dataset_id: null.
A cell on /impact/compare (EX-11). POST /v1/compute/impact/compare
takes the /impact body's cell fields: scenario + horizon_year with
haz_type/region, or with a hazard_dataset_id pinning the scope, under the
same rules and refusals. Every engine's job then carries the resolved
hazard_dataset_id, scenario_label and horizon_year, and the comparison's
params record scenario_label/horizon_year (plus haz_type/region when no
dataset was named), so an engine comparison can run a future cell. A body
without the pair is unchanged, and the result cache stays on /impact only.
Climate-adjusted future hazard in cost-benefit (issue #1482)¶
POST /v1/compute/cost-benefit computes each measure's present risk from the
baseline result's hazard and its future risk from a future hazard. That
hazard is addressed as a scenario-matrix cell, future_scenario +
future_horizon_year (both or neither), and resolved on the server from the
baseline result's own hazard dataset through the rule above
(resolve_scenario_horizon_hazard_dataset_id), so the baseline pins the hazard
type, the region and the provider family.
- Half a pair →
422 E_VALIDATION. Nothing is queued. - No dataset answers the cell →
422 E_VALIDATIONwhosedetailslist thecandidates, exactly as a scenario-addressed impact run does. - The cell resolves to the baseline's own dataset →
422 E_VALIDATION: the future risk would replay the present hazard, which is the no-climate-change figure this exists to replace. - Pair omitted → the present hazard stands in for the future one, as it did before. The request body of an existing client stays valid unchanged.
The job params record future_scenario, future_horizon_year and the resolved
future_hazard_dataset_id (null when the pair is omitted, so a reader can tell
a climate-adjusted run from a present-as-future one).
The appraisal period (issue #1623, AD-12). The appraisal ends in the hazard's
horizon year: with a future cell the params record
future_year = future_horizon_year (2050 for ssp2-4.5@2050) whatever
time_horizon_years says, because the future risk is the risk in that year and
the ramp from the present risk must end there. Without a cell it stays
present_year + time_horizon_years
(core/cost_benefit_service.py::appraisal_end_year). The request body is
unchanged.
Each measure's lifetime_years reaches the engine as its benefit_years
(null lifetime → None, the whole period). The engine zeroes the measure's
benefit for every year index above it and leaves
n = future_year - present_year alone, so the ramp slope does not move: a
30-year measure appraised from 2026 to 2080 counts 30 benefit years on a
54-year ramp. The engine takes one benefit_years per call, so the adapter
calls it once per measure.
The plan as one combined run (issue #1624, AD-13, K29). A job of two or
more measures is appraised a second time as one plan, and the worker stores it
as one more cost_benefit_results row beside the per-measure rows:
combined = true, measure_id null, measure_name = "plan", cost the sum
of the measures' resolved costs. A one-measure job stores no plan row.
- Canonical order. The engine chains the plan's measures on one hazard and
one curve set (
calc_cost_benefit(combined=True)), so two measures that remove the same loss are counted once. Chaining is not commutative once a measure lowers the depth, so the worker orders them first (core/measure_order.py::canonical_order): hazard-side shifts (hazard_inten_shift), then hazard-side factors (hazard_inten_imp), then protection standards (rp_protect, and ahazard_freq_cutoff), then curve-side factors (the MDD and PAA transforms); measures of one class follow their id. The order holds whatever ordermeasure_idsarrived in: a 0.6 m shift and a 0.5 depth factor on a 1.0 m cell leavemax(1.0 − 0.6, 0) × 0.5 = 0.2m, never the 0 the factor first would. A protection standard later in the chain finds its events' return periods through the hazard'sevent_id, which every removal keeps aligned. - Lifetime segments. The plan's distinct lifetimes
L1 < … < Lk(a null lifetime, and one longer than the period, countn) split the appraisal (core/measure_order.py::lifetime_segments). Segmentjis one combined engine call over the measures still alive (lifetime ≥Lj) cut atbenefit_years = Lj; it addsnpv_of_avoided(…, benefit_years=Lj) − npv_of_avoided(…, benefit_years=L(j−1))of that run's avoided pair (the first segment adds its whole NPV toL1, year 0 included) and supplies the plan'sper_year_benefitfor yearsL(j−1) < i ≤ Lj. A 50-year and a 5-year measure overn = 54are two calls: years 0 to 5 from the run of both, 6 to 50 from the 50-year measure alone, zero above 50, so the short measure never cuts the long one's benefit and the plan is never below its own 50-year row. A plan whose lifetimes are all null is one combined run over the whole period. The engine does every discount (npv_of_avoided); only the adapter calls it. - The plan row's risks are those of the run of every measure: the baseline risks, and the risks with the whole plan applied.
- A masked measure (issue #1626, AD-14). A measure whose
applies_tois an admin boundary or asset ids protects part of the run's rows, and the engine'scalc_cost_benefitapplies a measure to every row. The worker computes its avoided loss itself (worker/masked_measures.py::calculate_masked_cost_benefit): on the present and on the future hazard, the masked rows with the measure plus the other rows without it (the split and combine of the impact run), minus the baseline. The storedrisk_with_measure_*is that combinedaai_agg. The adapter'snpv_of_avoideddiscounts the pair over the measure'sbenefit_yearsand returns the benefit with itsper_year_benefitseries, read as the difference of the engine's NPV cut at two consecutive years. The cost is the measure's resolved total and is never split by the mask, so the BCR falls with the loss the mask leaves out. A measure masked to every row appraises as the same measure over the whole scope. A plan that holds a masked measure reads the same combine in every lifetime segment, each row computed with the measures alive in the segment that apply to it, and never callscalc_cost_benefit(combined=True). Whole-scope measures of the same job keep the engine's own call.
The engine takes one Exposures for both hazards, so the two datasets must
share a centroid grid. The worker compares the two grids (size and every
centroid position) before it computes, and a mismatch fails the job with a
ValidationError naming both dataset ids. No cost_benefit_results row is
written and exposures are not re-assigned to the future grid.
The cost-benefit base — Today, unadapted, ungrown (issue #1600, EX-15, K20)¶
POST /v1/compute/cost-benefit appraises only from a base result
(impact_result_id) that is a Today result with no measures and no exposure
growth. The engine grows the future stream itself through value_growth_rate,
evaluates each measure against the unadapted risk and takes the climate signal
from the future hazard, so any other base would count one of the three twice.
core.compute_service.check_cost_benefit_base reads the rule off the base's
provenance, before the job is staged:
The base's provenance |
Why it is refused |
|---|---|
measure_ids not empty |
the base is an adapted result (#1594) |
exposure_growth.rate not 0 |
the base grew its exposure (#1593), on a Today cell too |
horizon_year set |
the base answers a future cell, such as ssp2-4.5@2050 |
Each is a 422 E_VALIDATION whose details.reason starts
"appraise from an unadapted, ungrown present run", then names the result and
every fault found, and details.impact_result_id is the base. Nothing is queued. A base with none
of the three keys passes, which includes every result stored before they
existed. A base that does not resolve in the caller's org is not refused here:
the worker fails that job, as before. value_growth_rate keeps its name and
meaning.
A measure's total cost and its unit (issue #1621, AD-11)¶
A benefit-cost ratio divides a benefit in the exposure's value_unit by a
measure's cost, so core/measure_cost.py holds two rules:
- Total cost.
total_cost(measure)iscostforcost_basis = lump_sumandcost × unit_countforper_building,per_kmandper_ha. The worker hands the engine that total, never the unit cost, andcost_benefit_results.coststores it: a reference copied at 42,000,000 per km over 3 km is appraised at 126,000,000. - One unit.
POST /v1/compute/cost-benefitanswers422 E_VALIDATION,details.reason"measure cost unitdiffers from the exposure unit " (with details.cost_unit,details.value_unitanddetails.measure_id), when a measure'scost_unitis not thevalue_unitof the base's exposure set. Nothing is queued and no currency is converted. Units are compared stripped, a currency code upper-cased, soeurandEURare one unit; a blank unit reads(none). A non-money set (m3,persons) answers K19's "no money loss for value unit" first.
The exposure set is the one the worker reloads: the selector of the base's
producing job, else the dataset the result names. When none of its members has
a catalogue row the submit-time check has no unit to read, and the worker
makes the same check on the rows it loads and fails the job. The engine
adapter assumes no unit: exposure_data without value_unit, or a
cost-benefit measure without cost or cost_unit, is a ValidationError
naming the field. An impact run applies a measure without pricing it.
Protection standards by return period — rp_protect (issue #1566)¶
A measure's rp_protect (migration 0099, INTEGER NULL) is the return period
in years it protects to. A protection to the 1-in-N-year standard removes every
event row with rp <= N before the engine evaluates that measure; rarer events
are not touched. The selection is by return period, never by event weight
(core/protection_standard.py::protected_event_mask): the stored weights are
marginal band widths abs(diff(1/rp, append=0)), which are not monotone in the
return period, so a frequency cutoff of 1/100 over RPs 10, 20, 50, 100, 200
(widths 0.05, 0.03, 0.01, 0.005, 0.005) removes RP10 and RP20 only.
- Where it is applied.
worker/models/engine_adapter.py::without_protected_eventstakes an engine hazard, its per-event return periods andrp_protect, and returns the hazard without those rows. Cost-benefit calls it per measure on the present and on the future hazard separately (their RP lists may differ), and the baseline keeps every event. Impact compute withmeasure_ids(#1594) calls it per measure inEngineAdapter.apply_measures, before the measure's own transform. A measure withrp_protecton a hazard that carries no event return periods fails the job. - No
freq_cutoffis derived.hazard_freq_cutoffkeeps its stored meaning for user measures and still reaches the engine asfreq_cutoff; a measure may carry both. - API.
POSTandPATCH /v1/measurestakerp_protectas a positive integer (0or a negative value is422 E_VALIDATION), and every measure body serves it. - Seeded rows. The two builtin flood protections carry
rp_protect: 100and nohazard_freq_cutoff. Migration 0099 rewrites rows seeded before it (matched onsource = 'builtin'and the name; a user measure is untouched), andPOST /v1/measures/seedwrites the same values over an older row.
Measure, plan and score-job fields — migration 0111 (issue #1619)¶
Migration 0111 is the one schema change of the Adapt wave (Phase 17b W4,
AD-4 and AD-11 to AD-16). It adds the columns below. Cost-benefit reads
cost_basis and unit_count (#1621); no compute reads the others yet.
measures — served on every measure body (GET /v1/measures,
GET /v1/measures/{id}) and taken by POST and PATCH (#1620; the write
rules are below the table):
| Column | Type | Meaning |
|---|---|---|
applies_to |
JSONB NOT NULL DEFAULT '{"kind": "scope"}' |
Where the measure applies: {"kind": "scope"} (the whole run scope), {"kind": "admin", "admin_boundary_id": …} or {"kind": "assets", "asset_ids": […]} (AD-14; masking is #1625) |
cost_basis |
VARCHAR(16) NOT NULL DEFAULT 'lump_sum' |
What cost is a price of: lump_sum, per_building, per_km or per_ha (models.enums.CostBasis, held by the CHECK ck_measures_cost_basis; AD-11) |
unit_count |
FLOAT NULL |
The count the per-unit cost multiplies by (core.measure_cost.total_cost, #1621) |
hazard_inten_shift |
FLOAT NULL |
"Raise the asset or floor by h": a depth in metres, zero or more, that lowers the depth each asset sees (AD-15; the engine's haz_inten_b = -h, #1622) |
effect |
VARCHAR(255) NULL |
The editor's one-line "what it does" |
cobenefits |
TEXT NULL |
The editor's co-benefits text |
icon |
VARCHAR(32) NULL |
The plan icon, a key from the UI's fixed list (the sketch pattern) |
A row written before the migration reads applies_to = {"kind": "scope"},
cost_basis = "lump_sum" and null for the rest, and the seeded reference rows
are written at those defaults with cost: null. cost_basis is a varchar with
a CHECK, not a Postgres ENUM, so the migration round trip creates no type.
On write (core/measure_fields.py) a malformed value is a 422 E_VALIDATION.
For applies_to, for a unit_count that does not match the cost_basis and
for a hazard_inten_shift the hazard cannot take, details.reason names the
field:
applies_tois exactly one of the three shapes above. Anadminmask names one loadedadmin_boundary_id, anassetsmask a non-emptyasset_idslist, and a key the kind does not take is refused.unit_countis required withper_building,per_kmandper_haand refused withlump_sum. APATCHnaming only one of the two is judged against the other as stored, so going back tolump_sumsendsunit_count: nullin the same request.hazard_inten_shift(#1622) is written only for the depth perilsriver_floodandcoastal_flood; on another hazard the reason ishazard_inten_shift is a depth in m; earthquake intensity is not a depth. Beside ahazard_inten_impit is refused withone hazard-side transform per measure; split it into two measures so the plan order applies. A negative shift is a 422 from the request model. APATCHis judged against the hazard type and the factor as stored.iconfollows thesketchkey pattern andeffectholds at most 255 characters.
cost_benefit_results:
| Column | Type | Meaning |
|---|---|---|
per_year_benefit |
JSONB NULL |
The per-year avoided-loss series payback and NPV are both read from (AD-12; written by #1623). Null on a row stored before #1623 |
combined |
BOOLEAN NOT NULL DEFAULT false |
True on the plan row of a combined appraisal (AD-13; written by #1624) |
measure_id |
UUID NULL (was NOT NULL) |
Null on the plan row, which names no one measure |
jobs: result_payload JSONB NULL is the small result body of a job that
stores no result row, and job_type_enum gains score_delta
(JobType.SCORE_DELTA), the queued score-delta appraisal of a measure on an
indicator hazard (AD-16; the job is #1627).
The downgrade drops the columns, deletes the cost-benefit rows with a null
measure_id before restoring NOT NULL, and leaves the score_delta enum
value in place (PostgreSQL cannot remove one, as migration 0059 documents).
The score delta of a measure on an indicator hazard (issue #1627, AD-16)¶
A heatwave-days cube or an FWI wildfire pack has no loss curve, so a measure
on it has no avoided loss and never a BCR, payback or money figure.
POST /v1/compute/score-delta appraises it as a score delta instead
(core/measure_score.py): each member's intensity is read from the covering
indicator dataset of the measure's hazard (the read behind the portfolio
rollup's indicator_score, core/indicator_sampling.indicator_reads), banded
before and after the measure's hazard-side transform by
transformed_band(scheme, value, measure), and the two sets of bands are
reduced by the rollup's own band_distribution.
- The transform. The adapted band is the band of
max(hazard_inten_imp × value − hazard_inten_shift, 0). A protection standard withrp_protectat or above the peril's design return period (design_return_period_for) returns the scheme's lowest band. No scheme is added and no edge moves: banding iscore/scoring.py's. - Request.
measure_ids(at least one), one selector (portfolio_idorexposure_dataset_ids), an optionalscopein the run contract's three forms (admin_boundary_ids, its one-element aliasadmin_boundary_id, orbbox; a member inside two units is counted once) and the cell asscenario+horizon_year(both omitted: baseline, Today). - Answer.
scenario_label,horizon_year,unit,unit_is_money,reason,weighting,members_considered,members_in_scope,truncated,bands_enabledandmeasures[]. Each measure carriesmeasure_id,haz_type,baseline_bandandadapted_band(the scope's value-weighted average band as an integer, the mean rounded half up exactly as the stored indicator score's band is, so one scope reads one baseline band on both surfaces;band_label"value-weighted average band (ordinal, screening)", K25; ADR-088),baseline_meanandadapted_mean(the unrounded weighted mean bands, so a move inside one band stays visible),baseline_distributionandadapted_distribution(ten{band, exposed_value, share}entries),baseline_share_high_or_severeandadapted_share_high_or_severe(bands 7 to 10),exposed_value(the value of the members that read a value, which each distribution sums to),members_scored,hazard_dataset_id,metric_unit,scheme_refanddesign_return_period. - One unit or none (K27). A member weighs its row
value, the weightGET /v1/resultsgives a selector's indicator score, not the rollup's value × attachment weight. Members that mix units weigh 1 each:unitand everyexposed_valueare null,reasonis "mixed units",weightingismember_countand the shares are by member count. - 200 or 202 (architecture rule 2). A portfolio is bounded to its largest
rollup_max_portfolio_membersmembers (truncatedsays so) and anexposure_dataset_idsset of at most that many rows inside its clip also answers 200. A larger set answers 202{job_id, job_type: "score_delta"}with no raster read and no band computed on the request;worker/score_delta.py::compute_score_delta(queuecompute) calls the samemeasure_scorefunction over every member, stores the answer injobs.result_payloadand completes the job with a nullresult_id.GET /v1/jobs/{id}serves it asresult. - Refusals (422,
details.reason): "name one exposure selector"; the run contract'sscoperefusals; "unknown measure"; "no loss curve for this hazard, this measure cannot be appraised; record it as a qualitative action" for a measure with no hazard-side transform (only mdd_impact_*/paa_impact_*, or a factor of 1 and a shift of 0); "measureapplies to an admin area or named assets only; a score delta is appraised over the whole scope" for a masked measure, which is refused rather than applied to every member; "no indicator dataset for answers @ "; "nothing to score" when the scope holds no member; and "this hazard's score is not read from its intensity; the measure cannot be scored" for a scheme whose band input is not the hazard intensity. A queued job that meets one fails with the same message. - Cost-benefit.
POST /v1/compute/cost-benefitgives the same curve-side-only refusal when the base result ran on an indicator hazard (validate_cost_benefit_refs).
Reading a cost-benefit result — stored risks and the future basis (issue #1486)¶
GET /v1/results/{job_id}/cost-benefit returns, beside the unchanged
job_id, npv_benefit, npv_cost, benefit_cost_ratio and per-measure
measure_id, measure_name, npv_benefit, npv_cost, ratio:
- on each
per_measurerow, the four stored risksrisk_baseline_present,risk_with_measure_present,risk_baseline_future,risk_with_measure_future, the valueunitand thecost_unit, read straight from thecost_benefit_resultscolumns; - a job-level
future_hazardblock,{"basis", "dataset_id", "scenario", "horizon_year"}.
future_hazard.basis is read from the job params #1482 records
(core/cost_benefit_service.py::future_hazard_basis), not from a deployment
flag: a flag cannot tell a row computed before #1482 from one computed after.
future_hazard_dataset_idset →"climate_adjusted"with the recorded dataset id,future_scenarioandfuture_horizon_year.- The key absent (a job from before #1482) or
null(pair omitted) →"present_as_future"with the three other fieldsnull. There is no backfill; such rows keep present-as-future risks.
The change is additive: the ratio keys keep their values. A reader decides
whether to show a ratio from basis (the UI hides it while the basis is
present_as_future).
The per-year benefit series (issue #1623, AD-12). Each per_measure row
carries per_year_benefit: a list of n + 1 floats, the engine's discounted
and value-grown benefit of each appraised year i = 0..n, zero for every
i > lifetime_years. It is the stream the engine sums into that row's
npv_benefit, stored as returned (cost_benefit_results.per_year_benefit), so
sum(per_year_benefit) == npv_benefit and the backbone never discounts it a
second time. It is null on a row stored before #1623. The job level carries
no series of its own: the plan's series is plan.per_year_benefit.
The plan block (issue #1624, AD-13). A job that stored a plan row serves
it as plan (core/cost_benefit_service.py::plan_block), and its figures
come from that row, never from a sum of the per-measure rows:
| Field | Value |
|---|---|
npv_benefit, npv_cost, benefit_cost_ratio |
The plan row's benefit, cost and bcr |
per_year_benefit |
The plan's lifetime-segmented series; it sums to npv_benefit |
payback_years |
Read from that series against the plan's cost; null for a present_as_future job |
avoided_loss_present, avoided_loss_future |
Baseline risk minus the risk with every measure applied |
residual_risk_present, residual_risk_future |
Baseline risk minus the plan's avoided loss (the stored risk with the plan) |
single_measures_avoided_loss |
The sum of the per-measure rows' avoided_loss_future, in the plan's unit: the comparison figure of "single measures add to x, overlap removed". It enters no plan figure |
haz_type |
The hazard of the baseline result; one job is one hazard (K26) |
unit, cost_unit |
The plan row's units |
The top-level npv_benefit, npv_cost, benefit_cost_ratio and
payback_years are then the plan block's. With two overlapping flood measures
the plan's avoided loss is strictly below the sum of the rows'. per_measure
keeps one row per measure for ranking (the plan row is not among them), each
now also carrying avoided_loss_present and avoided_loss_future. plan is
null for a one-measure job and for a job stored before #1624; their top-level
figures stay the per-measure sums described below.
Payback (issue #1544). Each per_measure row and the job level carry
payback_years (float, years, or null,
core/cost_benefit_service.py::plan_paybacks). For a row with a stored series
it is the first point where the cumulative per_year_benefit reaches the cost,
interpolated linearly inside the crossing year
(i + (cost - cumulative_before_i) / per_year_benefit[i]), so payback and NPV
read one series and a measure whose lifetime ends before it repays reports
null. The job-level figure is the plan block's when the job stored a plan
row; without one it walks the year-by-year sum of the rows' series against
the summed costs.
A row stored before #1623 has no series and keeps the read-time derivation
from stored values only: the stored
cost, the four stored risks and the job params discount_rate,
value_growth_rate, present_year and future_year. No engine call is
involved. With n = future_year - present_year, the avoided loss
A_p = risk_baseline_present - risk_with_measure_present and
A_f = risk_baseline_future - risk_with_measure_future, year i = 0..n
contributes
This is the yearly stream the engine sums into npv_benefit (linear time
dependence), so while the avoided loss stays non-negative a measure repays
within the horizon exactly when its benefit reaches its cost. Year i spans [i, i + 1); payback_years is the first
point where the running sum reaches the cost, interpolated linearly inside the
crossing year: i + (cost - cumulative_before_i) / stream[i]. A cost of 100
against a constant 25/yr at 0 % discount pays back in 4.0 years. The job-level
payback_years runs the same formula on the plan total (summed costs and
summed avoided losses). It is null when the cumulative avoided loss never
reaches the cost within the n + 1 appraised years, and always null for a
present_as_future job (D9: no benefit figure without the climate-adjusted
future hazard).
Replaying a saved scenario — scenario_id alone (issue #1186)¶
POST /v1/compute/impact with scenario_id and nothing that addresses a
hazard dataset re-runs the saved scenario: hazard_dataset_id,
exposure_dataset_id, impact_function_id, the scenario/horizon_year pair
and the compute engine are all populated from scenarios.inputs before the
request is judged. Every other scenario_id request keeps its original,
narrower meaning — link the produced ImpactResult back to scenarios.result_id
— and takes nothing else from the row.
Populate, then validate. The addressing rule
(core.compute_service.check_hazard_addressing) is not copied or relaxed for
this path; it is deferred. ImpactRequest's validator passes
dataset_from_scenario=True when the body carries a scenario_id, which
suppresses exactly one refusal — "nothing names a hazard dataset" — because that
body is incomplete rather than wrong. The endpoint then fills the gaps and calls
the same function again with the strict default, on the request as it will
actually be computed. model_copy does not re-run validators, so that second
call is the real gate, not a belt-and-braces one. Every other refusal (half a
scenario/horizon_year pair; haz_type/region alongside a
hazard_dataset_id) still fires at parse time, because those judge fields the
caller did send and no stored scenario can repair them.
Precedence, three rules:
- Replay is entered only for a body that addresses nothing — no
hazard_dataset_idand not both ofscenario/horizon_year. That is exactly the shape #1185 refuses withHAZARD_DATASET_ID_REQUIRED, so the feature is purely additive: no request that was ever accepted computes differently now. In particular the long-standing{hazard_dataset_id, scenario_id}body does not start resolving away from the dataset it explicitly named. - Inside a replay, explicit request fields win, field by field — the
scenario fills only gaps. "Replay this scenario over a different exposure
set" works; "replay it and ignore what I typed" cannot happen.
X-Enginebeats the scenario's savedengineon the same terms, and the saved engine is resolved through the same registry, so one that has since been retired is refused with the header's own422/400. scenario_label/horizon_yearare applied as a pair or not at all, mirroring #1185's both-or-neither rule. A scenario storing half a cell is not replayable at all, rather than replayable with the saved half silently dropped.
Refusals. 404 when the scenario_id is absent or belongs to another org
(checked before every compute gate, so a scenario the caller cannot see always
wins over a hazard-side complaint). 422 when the saved inputs cannot address a
run — no hazard_dataset_id, half a cell, a declared key holding the wrong type
— with details.warnings carrying the same list GET /v1/scenarios/{id} shows
under computable: false. A saved reference that no longer exists is refused by
the endpoint's ordinary gates in their own words; when the replay is
cell-addressed, that refusal carries the reason/candidates detail every other
unresolved cell gets.
Cache key. The key is derived from the effective request, so a replay's
populated scenario/horizon_year join the digest on exactly the terms #1185
set for an explicitly cell-addressed run — two replays that resolve to the same
horizon-agnostic baseline under different labels cannot share one entry and
answer each other's question. No pre-existing key shifts, because replay only
ever completes a body that used to be a 422.
Exposure selector on impact compute (issue #1008)¶
POST /v1/compute/impact and POST /v1/compute/impact/matrix name the exposure
set with at most one selector:
| Selector | Meaning |
|---|---|
exposure_dataset_id |
One uploaded exposure group |
portfolio_id |
The assets in a portfolio, joined through portfolio_exposures |
| neither | Every exposure in the org assigned to the hazard dataset |
Supplying both is contradictory and returns 422 E_VALIDATION; a portfolio_id
outside the caller's org returns 404 E_NOT_FOUND. A portfolio is a selector,
not an endpoint shape: the job, the ImpactResult row, GET /v1/results/{id}
and the /v1/scenarios listing are all unchanged, so a matrix over a portfolio
comes free and a single asset is the n=1 case. An empty portfolio produces the
same "nothing to compute" refusal an exposure filter matching no rows already
did. The selector is not stored on the result — serving paths recover it from
the producing job's params when they re-derive the compute order (#370).
Dual response shape on the same endpoint:
- Cache hit ->
200 OK - Cache miss ->
202 Accepted
Invalidation contract. The cache key includes
impact_functions.version, so bumping the version retires every entry
derived from the old version (the new key is different — old rows just
become unreachable). Bug fixes that change result values MUST bump the
version; otherwise the cache will serve stale numbers.
DB-level guard (issue #485). A BEFORE UPDATE trigger on
impact_functions (migration 0063) RAISE WARNINGs whenever any of
mdd_x/mdd_y/paa_x/paa_y changes without version also changing.
It is advisory — the UPDATE still succeeds and the trigger never bumps
version itself (an automatic bump would silently invalidate cached
results out from under a running system) — so bulk paths like restores,
reseeding migrations, and core/impact_function_seeder.py's
upsert-on-conflict stay unblocked. Treat the warning in the Postgres log
as a prompt to check whether the edit should have bumped version.
Cross-org safety: cache rows have no org_id; lookup re-checks RLS on
the underlying impact_results row before treating a row as a hit, so
two orgs with identical inputs share the same key but only the org that
owns the result sees the hit.
Maintenance: prewarm_result_cache (Celery beat, 03:00 UTC) reads
config/prewarm.yaml and dispatches missing entries; purge_orphan_cache_entries
(04:00 UTC) deletes rows whose result_id no longer exists. Hit/miss
counts are emitted as climate_lama_result_cache_hits_total{result=...}.
Shared result vocabulary — curve_basis, aal, pml_200 (issue #1476)¶
Every result speaks the same minimal vocabulary, whichever engine produced it
(Phase 17 D19). GET /v1/results/{id} and each engine's metrics in the
agreement view (GET /v1/compute/impact/compare/{id}) carry:
| Key | Meaning |
|---|---|
curve_basis |
"OEP" or "AEP", stored per result (impact_results.curve_basis). Both built-in engines compute a per-event exceedance curve and stamp OEP explicitly in their adapter. null when the result has no curve |
aal |
Aggregate annual loss: an alias of ead (equal to aai for monetary results). Never the per-point EAI (eai_exp_path). ead and aai keep their names |
pml_200 |
{"return_period": 200.0, "loss": <the curve's RP-200 point>, "basis": "OEP", "label": "PML 1-in-200 (OEP)"}, or null when the curve does not sample RP 200 or has no basis. Never interpolated between curve points |
aal and pml_200 are computed at read time (core/results_vocabulary.py),
so they cannot drift from ead and the stored curve.
GET /v1/results/{id}/freq-curve carries curve_basis beside its points.
The agreement view compares curves on a shared basis only. When every engine
holding a curve holds it on the same basis, divergence.freq_curve is the
per-return-period block and divergence.different_curve_basis is false.
When the bases differ (OEP beside AEP), divergence.freq_curve is [] and
different_curve_basis is true. Scalar metrics still compare.
scripts/verify_engine_agreement.py prints different curve basis in that
case.
Scope coverage on a result — rows_computed / scope_total (issue #1180)¶
A compute over a partially assigned scope completes normally: it scores the
exposure rows that carry a centroid assignment for the hazard and simply leaves
out the rest. The compute preflight fills every gap it can
(worker.tasks._load_exposures_with_preflight), but a member it cannot pair
stays out of the compute order for good, so a run can legitimately cover less
than the scope holds. That used to be entirely silent.
Two numbers now survive the run:
| Key | Meaning |
|---|---|
rows_computed |
Rows in the run's compute order — what the eai_exp array indexes, sentinel rows (#442) included |
scope_total |
Members the run's exposure scope holds, from the same statement that loaded the rows (total_scope_members, #1060). Scope-dependent population — see below |
Which members scope_total counts. Read this before rendering "N of M":
- dataset scope — every row of the exposure dataset.
- portfolio scope — members inside the hazard's
footprintwhen one is stamped, which is every dataset ingested with a grid definition (#596; footprint and grid definition are stamped all-or-nothing). A member outside a stamped footprint is therefore excluded fromscope_totaland from the rows, and the pair reads "all of it" rather than reporting a gap the hazard could never close — that exclusion is deliberate (#1066): counting such a member made the coverage check permanently short and re-ran the preflight on every compute. WhenfootprintisNULL(a pre-#596 / not-yet-backfilled dataset, #1049) the count falls back to every member, and an out-of-extent member then does show up as a real, permanent gap. - hazard scope — no independent total (
NULL);scope_totalfalls back torows_computed, so an unscoped run never reads as a phantom gap.
So the pair records how much of the counted population this result covers — it is not a "the whole portfolio is in range" assertion.
- Where they live: the existing
impact_results.metadataJSONB (ORMImpactResult.metadata_), whose documented remit is "engine metadata + calculation diagnostics". Notprovenance, which is the reproducibility block (engine name/version, input SHA-256s, impact-function id/version, seed, resolvedscenario_label/horizon_year) and is copied verbatim onto scenarios and matrix cells. The provenance shape is unchanged by #1180. No new column and no migration —core/result_coverage.pyowns the two key names for both the worker that writes them and the API that serves them. - Where they surface:
GET /v1/results/{id}lifts both to top-level keys, so a client renders "N of M assets" without knowing where they are stored. A result stamped before #1180 has neither key in its blob and serves both asnull— never aKeyError, never a 500. Themetadatablob is still returned verbatim alongside, so the pair appears twice; the top-level keys are the validated view and the ones to read. The promotion is specific to these two keys (the UI's "N of M assets" needs them), not a general rule for promoting whatever else that free-form JSONB grows.GET /v1/results/{id}/geojsonis unchanged, and so is the report export — neither carries the pair. (The admin-level GeoJSON carries a different figure,admin_coverage, with a different denominator — see "Admin coverage on admin-level GeoJSON" below.) - When the gap is real:
rows_computed < scope_totalafter the preflight is logged once at WARNING by the worker, naming the hazard dataset, the scope selector and both numbers. Every compute path warns (impact, event, and cost-benefit all share the preflight); only the impact path also stamps the pair onto a result row, so the message says nothing about where it is recorded. The job still completes — this records the outcome, it does not change it. - Reading the pair:
rows_computedis the row count this result covers, not a coverage verdict._scope_fully_covered's own numerator for portfolio scope isassigned_in_scope_members— assigned members inside the hazard footprint — while the compute-order row set has no footprint filter, so the two can differ in either direction for that scope (#1066).
Not the same as n_valid_points, which already lives in the same blob: the
engine reports that one as the rows surviving its own exclusions, it is absent
on a CLIMADA-engine result (that adapter stamps only engine_version), and the
PDF report renders its exposure count from it. rows_computed is
engine-independent and pre-exclusion. Use the scope pair for "N of M assets";
use n_valid_points for "what the engine actually scored".
Value unit on a result — value_unit (issue #1583)¶
compute_impact stamps metadata.value_unit: the unit every compute-order row
shares ("USD", "EUR", "m3", ...), or null when the rows mix units. It is
never the first row's unit, which is what the engine call is handed (the engine
takes one unit; refusing a mixed-unit scope is the run contract's job). Helpers:
core/result_coverage.py (value_unit_metadata, resolve_value_unit). No new
column, no migration.
GET /v1/results/{id} lifts it to a top-level value_unit: the stamped value,
a stamped null included. A result written before the stamp falls back at read
time to its exposure_dataset_id dataset's value_unit, so pre-17b saved
analyses still print a unit. null when neither exists. Event results and
cost-benefit rows keep their own unit fields.
Exposed value and band shares on a result (issue #1595, EX-10, SH-16)¶
compute_impact stamps seven figures into metadata, beside the coverage pair
and value_unit, and GET /v1/results/{id} lifts each to the top level the same
way. Helpers: core/result_coverage.py (RESULT_FIGURE_KEYS,
read_result_figures) and worker/tasks.py (_result_figures_metadata). No new
column, no migration.
| Field | Meaning | null when |
|---|---|---|
exposed_value |
The summed value the engine computed over: after the clip, after growth (scaled from each layer's reference year), in the result's one value_unit. Sentinel rows (#442) never reach the engine and are not counted |
The rows mix units (K27) |
band_distribution |
[{band, exposed_value, share}], one entry per band 1-10. Each computed row's intensity at the design return period is banded under the run's score scheme (the rollup's ladder behind the unit gate, core.scoring.rollup_scheme_for_run) and reduced by the rollup's own core.rollup_service.band_distribution, weighted by the row's grown value (by row count with exposed_value: null per entry when units mix) |
The dataset has no event at the design return period (earthquake RPs 50 and 975), no scheme matches its intensity unit, or score_bands_enabled is off |
share_high_or_severe |
The value share in bands 7 to 10 | As band_distribution |
count_high_or_severe |
How many locations band 7 to 10; 0 when the share is 0 | As band_distribution |
design_return_period |
The return period the compute banded at (core.scoring.design_return_period_for): 100 for river flood and 476 for earthquake under the default settings, also when the dataset has no event there |
Never on a new result |
value_weighted_average_band |
Σ band × exposed_value over Σ exposed_value of band_distribution, by core.scoring.value_weighted_mean (the rollup's value_weighted.combined_score function); ordinal, a screening figure |
As band_distribution |
impact_by_exposure_type |
{exposure_type: expected annual impact} over the result's locations, summing to its ead; one entry for a single exposure layer |
A row carries no exposure_type |
A result written before #1595 serves all seven as null.
GeoJSON Result Retrieval¶
GET /v1/results/{id}/geojsonloads ImpactResult + Exposure rows + eai_exp NPZ from MinIOeai_expholds one entry per exposure point, ordered bycore.eai_alignment.EXPOSURE_COMPUTE_ORDER_BY— array position is not a centroidarray_index. The endpoint re-runs the worker's exposure join in that order and rolls the values up onto the centroid each asset snapped to (ADR-024 era bug, fixed in #370)- Returns FeatureCollection with
[lon, lat]coordinates (GeoJSON standard) andeai_exp,array_index,point_countproperties — one Feature per centroid that carries exposure; unexposed centroids are omitted - Returns
409 E_RESULT_STALEwhen the exposure dataset has changed since the job ran, so no safe attribution exists.GET /v1/results/{id}/reportbehaves the same way - With
?admin_level=0|1|2the same rollup is grouped by GADM admin polygon instead: oneMultiPolygonfeature per polygon at that level containing at least one hazard cell, withadmin_id,admin_name,iso3,admin_level,eai_exp_sumandpoint_count(assets on the polygon's cells).bboxfilters which polygons are returned, never a polygon's own sums
Admin coverage on admin-level GeoJSON — admin_coverage (issue #1370)¶
An admin-level response also carries one collection-level figure saying how much
of the result the level's polygons cover, so a map can tell a regional view that
represents the result from one that shows a sliver of it (prod's four-row
admin_boundaries turned a Greece-wide run into "Attica only"):
"properties": {"result_id": "…", "ead": 1234.5, "aai": 1234.5, "admin_level": 1,
"admin_coverage": {"assigned_points": 7, "total_points": 10, "ratio": 0.7}}
| Key | Meaning |
|---|---|
total_points |
Every exposure point the stored result attributed to a centroid — count_by_centroid.sum() from the eai_exp rollup; sentinel rows (#442) are not attributed and not counted |
assigned_points |
Those assets that sit on a cell some polygon at the level contains — the sum of the per-feature point_count |
ratio |
assigned_points / total_points; 0.0 when total_points is 0 |
- Assets on both sides, not rows and not cells. This is a different
denominator from the scope-coverage pair on
GET /v1/results/{id}:rows_computed/scope_totalcompare the run's compute-order rows with the members its exposure scope holds (did the run score the whole book?);admin_coveragecompares assets inside the loaded boundaries with the assets the result attributed to a centroid (do the regions represent the result?). A run can score 100 % of its scope and still have 5 % regional coverage. Hence the distinct name — it is never calledcoverage. - Viewport-independent. Computed over the whole level, not the
bbox: panning the map never changes it, and abboxthat shows no polygon still answers with the full figure.core.admin_aggregation.resolve_admin_aggregation_viewaggregates the level once and marks which polygons the viewport intersects. - Disjointness is checked before summing. Every exposure-carrying cell must
lie in at most one polygon, or its assets would be counted twice. When two
polygons at one level do share such a cell (a boundary-data defect), the
features are still served — each polygon really contains its assets — but
admin_coverageis omitted and a WARNING names the polygons. A client treats an absent figure as "no coverage known"; the UI then keeps the asset view. An overlap on a cell carrying no assets cannot move either sum and is tolerated. - No polygon at the level at all. When no polygon at the requested level
contains any of the dataset's cells, the response is the empty collection it
always was, without
admin_coverageand without reading the stored array — a client reads the absence the same way ("no coverage known"). - Cost. The whole level is aggregated on every admin-level request,
bboxor not (a no-bboxrequest always paid this), plus one id-only query for the viewport's polygons when abboxis present. - Raw-point responses (no
admin_level) carry noadmin_coverage.
Per-exposure results and a portfolio's own runs (issue #1179)¶
Two read sides of data that already exists — no new compute, no schema change.
GET /v1/results/{id}/per-exposure¶
VIEWER+, org-scoped. ?limit= 1–500 (default 100), ?offset= ≥ 0, ?sort=
eai (default) or value, both descending; any other sort is a 422.
{"items": [{"exposure_id": "…", "name": "Attica GHSL #417", "lat": 38.0,
"lon": 23.8, "value": 1000000.0, "eai": 4200.0,
"dataset_name": "Attica GHSL"}],
"total": 216000}
- The response is a page slice; the object read is not.
eai_expis a single member of one.npzobject, so there is no partial fetch — the array is downloaded whole, aligned once, ranked, and only the requested window is serialised.totalis the whole array's length. That asymmetry is whylimitis capped rather than unbounded: a 216 000-row book must not be able to answer in one response. - Alignment, never indexing. Rows come from
core.eai_alignment.load_aligned_exposure_assets, which re-runs the worker's compute-order join (EXPOSURE_COMPUTE_ORDER_BY). Array position is not a centroidarray_index(#370); indexing by one hands assets each other's damage the moment centroid assignments are sparse or duplicated. nameis the shared display name —properties.name, else"<dataset name> #<row index>"— resolved throughdb/repositories/exposure_naming.py, the same helperGET /v1/exposure-datasets/{id}/rowsandGET /v1/portfolios/{id}/memberscall, and numbered before any selection filter so the label does not move between surfaces. A row belonging to no dataset keeps its ownexposures.name.dataset_nameis the exposure's own dataset.- Error mapping, matching the rest of the
/v1/resultsfamily: unknown or cross-org result → 404E_NOT_FOUND; a result whoseeai_exp_pathis NULL → 404E_NOT_FOUNDwhose message names the missingeai_exparray (distinguishable from an absent result); a drifted exposure set → 409E_RESULT_STALE, the same refusal the GeoJSON and report routes give.
GET /v1/portfolios/{id}/results¶
VIEWER+, org-scoped. ?limit= 1–100 (default 20). {items, total}, newest
first, total unpaged:
{"items": [{"result_id": "…", "created_at": "2026-05-01T09:00:00+00:00",
"hazard_dataset_name": "Attica flood", "impact_function_id": "…",
"ead": 4200.5, "aai": 4100.25, "scenario": "ssp5-8.5",
"horizon_year": 2050}],
"total": 43}
impact_resultshas noportfolio_idcolumn, deliberately: a portfolio-addressed result is shape-identical to a dataset-addressed one (#1008). Membership is the joinimpact_results.job_id → jobs.idfiltered onjobs.params->>'portfolio_id'. That resolution has one definition —db/repositories/result_repository.portfolio_results_query, withJOB_PORTFOLIO_IDexported alongside it for callers that need the portfolio as a value (grouping results to find each book's latest run).- No N+1. Three statements whatever the page holds: the portfolio existence check, the page, and the unpaged count. The hazard dataset's name rides on the page query's outer join. Enforced by a statement-counting test, not by review.
hazard_dataset_nameisnullwhen the dataset the run scored has since been deleted — the result stays readable.scenariois the run's ownparams.scenario_labelwhen it carried one, falling back to the hazard dataset'sscenario— the same precedence the compute path uses for its answer-plane run surface key.horizon_yearcomes fromparams.horizon_yearonly (a dataset has no horizon of its own) and reads asnullrather than erroring if that JSONB value is malformed. Both arenullwhen neither side declares one.- An existing portfolio with no runs is
{"items": [], "total": 0}, not a - Only an unknown or cross-org portfolio is 404
E_NOT_FOUND.
GET /v1/portfolios/summary¶
VIEWER+, org-scoped. ?limit= 1–50 (default 5), ?sort= exposed_value
(default) or ead. One call for Home's Portfolios card (Phase 14 D5):
{"items": [{"portfolio_id": "…", "name": "Attica book",
"exposed_value": 12500000.0, "value_unit": "USD",
"ead": 42000.5, "aai": 41000.25, "member_count": 318,
"latest_result_id": "…",
"latest_result_at": "2026-05-01T11:00:00+00:00"}]}
- Everything is stored. The totals come from the attachment-plane rollup
and the latest run from
impact_results. Nothing is recomputed and no compute job is dispatched — the card costs a read. eadandaaiare one hazard's figures. They equal the list row'stotal_eadandtotal_aaiunder the PF-8 rule (issue #1571), the hazard of the portfolio's newest contributing result, never a sum across hazards.- Route ordering is load-bearing.
/summaryputs a literal in the same segment/{portfolio_id}puts auuid.UUIDparameter, so it must be declared above it inapi/v1/portfolios.py. Below it, FastAPI tries to parse"summary"as a UUID and answers 422, not 404. A test asserts a bareGET /v1/portfolios/summaryis 200 precisely to catch a re-ordering. - Sort keys.
sortnames the response field it ranks on, always descending, with name then id as tiebreaks so the ranking is stable across requests. An unknown value — including the internal metric nametotal_ead— is refused by parameter validation with a 422; no error-registryCodeis minted for it. - The ranking spans the whole organization, then is cut to
limit. It is not a page, which is why the envelope is{items}with nototal. value_unitis the members' shared unit, ornullwhen the book mixes units.exposed_valueis still the sum; it simply carries no single label to qualify it with. It readsexposures.value_unit— the same columnGET /v1/portfolios/{id}/membersreturns per member — so the two can never disagree about a book's currency. A book with no members isnulltoo.exposed_value,eadandaaiare all weighted by the attachment's weight, exactly as/aggregateweights: a half-weighted slice contributes half its value and half its loss.- A book with no members and no runs is a row of zeros with
latest_result_id: null— never omitted, never an error. A list of the org's books that drops the empty ones is not a list of the org's books. latest_result_id/latest_result_atresolve throughjobs.params->>'portfolio_id'via the same single definitionGET /v1/portfolios/{id}/resultsuses (portfolio_results_query), so they name the row that read puts first. A book whose only run failed hasnullfor both — a failed job writes noimpact_resultsrow.ead/aaiandlatest_result_idresolve on different planes, and can disagree.ead/aaiare dataset-plane:list_metrics' lateral keys the latest result onexposure_dataset_idalone — "the newest loss anyone computed over the datasets this book holds" — which is exactly what/aggregateandGET /v1/portfoliosreport, and those three agreeing is a requirement of the dataset-plane membership work (#1169).latest_result_idis portfolio-plane: which run this book launched. So a run addressed at book B but computed over a dataset book A also holds gives A a loss with no run to deep-link to, and B a run with no loss. Each number is correct for the question it answers; a UI that renders them as one story should know they are two. Pinned bytest_another_books_run_is_not_this_books_run.- No N+1. Three statements whatever the number of books: the org's
portfolio rows, one aggregate over all of them
(
PortfolioRepository.list_metrics, extended rather than duplicated), and one windowed read of every book's newest result (ResultRepository.latest_per_portfolio). Enforced by a statement-counting test that measures at 1 book and at 5 and requires the counts to be equal. GET /v1/portfoliosis unchanged — same{data, total}envelope, same three metric keys per row.list_metricsgrew wider for the summary and the list route projects back down to what it has always returned.
Async/Sync Boundary (ADR 021)¶
Celery tasks are synchronous. DB operations use async (asyncpg). Solution: wrap async code in asyncio.run() inside the task, create engine + session inside the coroutine, never at module level. Dispose engine in finally block.
Bulk Spatial Inserts (ADR 022)¶
ORM add_all() with geometry triggers per-row INSERT. Solution: raw SQL with unnest() + ST_SetSRID(ST_MakePoint()). Result: 630K centroids in ~1 second.
Deduplication¶
- Hazard:
get_by_name_and_type(name, haz_type)check before ingestion. Returns existing dataset if found. - Exposure:
exists_by_name(name)check before CSV parsing. Returns[]if exists (hard skip to preserve centroid assignments).
CSV Upload MIME Handling¶
Windows browsers send .csv as application/vnd.ms-excel or application/octet-stream. Expanded allowed types + filename extension fallback:
7. Docker Compose¶
services:
postgres: postgis/postgis:16-3.4-alpine # port 5432
redis: redis:7-alpine # port 6379
minio: docker.io/pgsty/minio (pinned tag + digest, see docker-compose.yml, ADR-082) # ports 9000/9001
api: ./docker/core.Dockerfile # port 8000, hot reload
worker: ./docker/worker.Dockerfile # Celery, `compute` queue
live-worker: ./docker/worker.Dockerfile # Celery, `live` queue only (Phase 15)
beat: ./docker/worker.Dockerfile # Celery beat, scheduler only
The UI is a standalone deployable in the climate-lama-ui repo and is not part of this compose file.
Key env vars: DATABASE_URL (asyncpg), DATABASE_SYNC_URL (psycopg2), REDIS_URL, CELERY_BROKER_URL, MINIO_ENDPOINT.
Worker queue routing:
task_routes = {
"climate_lama.worker.tasks.ingest_hazard": {"queue": "compute"},
"climate_lama.worker.tasks.assign_centroids": {"queue": "compute"},
"climate_lama.worker.tasks.compute_impact": {"queue": "compute"},
# ── live lane (Phase 15) ──
"climate_lama.worker.live_feeds.tick_live_feeds": {"queue": "live"},
"climate_lama.worker.live_feeds.poll_live_feed": {"queue": "live"},
}
The live queue and live-worker (Phase 15)¶
A feed poll must answer in minutes and must not queue behind a multi-hour
ingest, so the live lane gets its own queue and its own container:
live-worker runs celery … worker --queues=live --concurrency=1 and consumes
nothing else. worker keeps --queues=compute on prod; impact runs
(compute_impact) have their own impact queue and prod impact-worker at
--concurrency=3 (issue #1569), while the image default and the dev worker
consume compute,impact. Routing is by task name, and
task_default_queue is deliberately not compute: a task that is neither
routed above nor given an explicit apply_async(queue=…) lands on a queue no
container consumes, so a missing route fails loudly rather than silently
stealing capacity.
The lane is scheduled by beat, whose live-feed-tick entry fires
tick_live_feeds:
"live-feed-tick": {
"task": "climate_lama.worker.live_feeds.tick_live_feeds",
"schedule": crontab(minute=f"*/{max(1, settings.live_feed_tick_seconds // 60)}"),
"options": {"expires": settings.live_feed_tick_seconds},
},
Both containers are load-bearing and neither substitutes for the other: beat
publishes the tick and live-worker runs it, so a deployment missing either
one polls nothing. Both read LIVE_MODE_ENABLED; unset, the tick is a logged
no-op. The tick selects the feeds whose own cadence_seconds is due —
live_feed_tick_seconds (default 300) is the polling floor, so a feed
configured faster than the deployment's tick polls at the tick's rate, not its
own (#1444 tracks
advertising the effective interval). live-worker mounts /data read-only
(the vendored and staged replay series) and shares the /uploads volume with
worker, because a poll stages its payload there and the ingest chord — which
runs on compute — reads it back off the same volume.
The auto-compute hook. When a poll releases an observation, the lane
dispatches the ingest with origin=feed, and on an auto_compute watch the
committed snapshot dispatches one compute_event_impact job onto compute
with origin=feed, notify=False and metered=False (see
§16 Usage Metering). A person's own push
(POST /v1/live-events/{id}/observations) keeps origin=user and is both
notified and metered.
8. Code Conventions¶
- Formatter/Linter: Ruff (replaces Black, isort, flake8)
- Type hints: Required on all public functions
- Docstrings: Google style on public API
- Line length: 100 characters
- Naming: snake_case files/functions, PascalCase classes, SCREAMING_SNAKE_CASE constants, kebab-case URLs
- Error handling: Custom exceptions in
core/exceptions.py, API layer catches + transforms, never expose internals - Logging: Python logging (structlog planned for Phase 2)
Git conventions¶
Branch: <type>/<issue-number>-<short-description> (e.g., feat/42-exposure-upload)
Commit: type(scope): description (imperative, no period, max 72 chars)
Types: feat, fix, chore, docs, refactor, test, ci
Scopes: api, worker, core, db, storage, ingest, catalog, ci, docs
Final commit on branch: Resolves #N or Refs #N in footer.
9. Security Defaults¶
- Auth (MVP): API key via
Authorization: Bearer <key>, hashed in DB - Input validation: All via Pydantic. File uploads: MIME + extension check, size limits. Spatial: coordinate bounds.
- Secrets: Environment variables only,
.env.exampleas template, Docker secrets for production - Data access: Repository pattern, no raw SQL in API/core, parameterized queries (SQLAlchemy)
Live Event Mode (issue #1018, Phase 13 W0; endpoints #1026)¶
Settings.live_mode_enabled (env LIVE_MODE_ENABLED, default false) is the
default-off master switch for the live-event-replay feature line — the same
pattern as score_bands_enabled and geocoder_enabled.
GET /v1/info advertises it (issue #1280): the response data carries exactly
version, environment, git_sha, live_feed_tick_seconds and features,
where
live_feed_tick_seconds (issue #1444) is Settings.live_feed_tick_seconds, the
interval the live-feed-tick beat runs at. It is published so a client can
state an achievable cadence before a feed exists: no feed is polled more
often than the beat runs, so a UI that lets a reader ask for 2 minutes where the
deployment ticks every 5 can floor its input here instead of promising a cadence
the scheduler cannot keep. It is a number, so it is a top-level sibling of
features and never a key inside it — and it is published unconditionally, even
where live mode is off, so the key set stays exact. Both the endpoint and
core.live_feeds.effective_cadence_seconds read it through one function,
core.live_feeds.feed_tick_seconds, so the advertised grain and the per-feed
effective cadence cannot disagree.
features is a feature-advertisement object, not a configuration dump. Its
values are booleans and nothing else — never a URL, host, provider name or key —
because /v1/info is unauthenticated and everything in it is a standing
disclosure. A flag appears here only so a client can decide whether to mount a
surface: the UI mounts its Live navigation entry on features.live_mode and
omits it entirely when the flag is false, rather than rendering a section that
answers empty. tests/test_api/test_info.py pins the exact key set of both
objects in one place each (INFO_KEYS, FEATURE_KEYS), and asserts each flag
for both values — a handler that hardcoded either literal would pass a
single-value test.
features.geocoding (issue #1336) is the second flag, and it is the reason the
handler reads every flag through the core reader that owns it rather than a
bare settings.<flag>: its value is
geocoder_enabled AND geocoder_advertised (core/geocoding.py::geocoding_advertised),
never the master switch alone. docker-compose.prod.yml forces
GEOCODER_ENABLED: "true" on a host that runs no Nominatim so the 503 keeps its
diagnostic reason="connection", so advertising on that switch would promise
address search a pin-drop-only deployment cannot serve (#1322, reaffirming
534). geocoder_advertised is the separate, default-off statement that a¶
geocoder is genuinely reachable. The key has no reader yet — gating the UI's
/lookup address box on it is climate-lama-ui#424, still open — so it is a
standing contract offered to that client, not behaviour that exists today. See
docs/DEPLOYMENT.md, "The two geocoder flags".
features.assistant (issue #1493, ADR-086) is the third flag and follows the
same rule: its value is assistant_enabled AND anthropic_api_key
(core/assistant/service.py::assistant_enabled), never ASSISTANT_ENABLED
alone, because a flag without a key can only answer 503 E_ASSISTANT_UNAVAILABLE
(reason="no_key"). The UI hides its assistant panel while it is false (D15
offline acceptance). See "Assistant settings" below.
The /v1/live-events endpoints (issue #1026) are the first readers of it, and
they observe the phase doc's dark landing contract:
| Surface | Live mode off | Live mode on |
|---|---|---|
GET /v1/live-events |
200 {"data": []} — never 404/501 |
The org's events |
GET /v1/live-events/{id}, /snapshots, /snapshots/{snapshot_id}/portfolios, /results, /results/{job_id}, /results/{job_id}/delta |
404 E_LIVE_EVENT_NOT_FOUND |
The event / its replay / the portfolios in a snapshot's footprint / results / a delta between two of them |
POST /v1/live-events, PATCH /v1/live-events/{id}, /{id}/close, /{id}/compute, /{id}/observations |
422 E_LIVE_MODE_DISABLED |
Writes through |
Every /{id}/feeds… route, reads included (issue #1295), and GET /v1/live-events/replay-series (issue #1807) |
422 E_LIVE_MODE_DISABLED |
The watch's schedule / the series a replay feed can name |
The feeds sub-resource is the one read that is refused rather than darkened.
The empty-page contract exists so a UI can mount its live section without a
feature flag; a feed listing is only ever reached from that section, so it has
nothing left to protect, and an empty list there would state something false —
that a watch has no schedule — where the refusal states what is actually the
case. That gate is asked in the router rather than in core/live_feeds.py,
because the same service functions are called by the worker tick, whose own
flag check sits at its task entry point.
The listing stays a 200 because the UI mounts its live-event section on a
non-empty list and has no feature flag of its own — a 404 or 501 would be an
error state to render rather than a section to omit. The single-event reads
follow the listing rather than diverging from it: with the flag off, "absent",
"another tenant's" and "live mode is off" are one indistinguishable answer, so
a client can never read an event the listing has just denied. Writes cannot
degrade that quietly — returning success for a POST that wrote nothing would
be a lie to the caller — so they are refused outright.
One deliberate exception: POST /v1/hazards/ingest still resolves a supplied
live_event_id against the caller's org and answers the generic E_NOT_FOUND
(issue #1020's contract, pinned by its own test), and does not consult the
flag — an ingest naming an event the caller genuinely owns is a legitimate
write whether or not the replay UI is switched on.
Roles follow the hazards ladder: VIEWER reads, ANALYST and above create, patch
and close (Role has no separate EDITOR tier).
The watch: selector, AOI, summary (issue #1287)¶
The selector is exclusive-or, not at-most-one. POST /v1/live-events
requires exactly one of portfolio_id / exposure_dataset_id; both, or
neither, is E_LIVE_WATCH_SELECTOR_CONFLICT (422). This departs from the
compute lanes' at-most-one convention (issue #1008) on purpose: a compute
selector scopes one run, while a watch's selector decides the ground its AOI is
derived from and what every auto_compute job it enqueues will run against.
A watch with no selector could never compute anything; a watch with two would
have to pick one silently. The rule lives in core/live_events.py, not in the
Pydantic body — a schema validator would surface as FastAPI's generic
body-validation 422, which carries no Code a client can branch on.
A watch keeps its selector (EV-12, issue #1609). Both selector foreign
keys, live_events.portfolio_id and live_events.exposure_dataset_id, are
ON DELETE RESTRICT (migration 0106; they were SET NULL). A watch left open
with no selector has lost its scope — a Compute press would score the whole
organization and the automatic path refuses it outright (issue #1610, "Compute
entry" below) — so the row it names cannot be deleted from under it. DELETE /v1/portfolios/{id} answers
the rule before the database does (core/portfolio_service.py::delete_portfolio):
an open watch on the portfolio is a 409 with the literal code CONFLICT,
details.live_events: [{id, name}] and a message naming the watch, and the
portfolio stays. A portfolio only closed watches name is still deletable:
no route deletes a watch and a closed watch never computes again, so the route
sets their portfolio_id to NULL in the same transaction and then deletes. No
route deletes an exposure dataset, so for exposure_dataset_id the foreign key
itself is the refusal, for an open or a closed watch alike.
event_impact_results.portfolio_id keeps ON DELETE SET NULL: a result's
history is not part of this rule.
The AOI is creation-time state. Supply aoi (a GeoJSON Polygon,
EPSG:4326) and it is stored verbatim. Omit it with a portfolio_id and it is
derived once, from the book's member extent buffered by
live_aoi_buffer_km (default 10 km), and then never recomputed — so a book
that grows next week does not silently widen a watch an operator already
reviewed (design record D7). A book that resolves to no members is a
E_VALIDATION 422, never an empty polygon: an empty AOI is a watch that
matches no observation ever, which fails silently instead of loudly. Omit it
with an exposure_dataset_id and the watch stores none — a catalog
dataset's extent is not the same statement as "the ground this watch is
about". A book whose members span more than 180° of longitude is refused
the same way: ST_Extent has no date-line wrap, so "straddles the
antimeridian" and "spans most of the globe" arrive as the identical bounding
box, and guessing wrong would hand a watch covering 0.2° of ground an AOI that
matches every observation on Earth — supply an explicit aoi for those.
The buffer is applied in degrees (buffer_km / 111.32 for latitude, that
scaled by cos(lat) for longitude, at the widest latitude the box reaches —
never at a capped latitude, which would under-buffer a polar book rather than
over-buffer it) and a box that would wrap is emitted as the whole longitude
range. ST_Buffer on a geography was rejected because it projects through a
UTM zone picked from the centroid and degrades on a book spanning several.
The served aoi falls back; the stored one does not (EV-4, issue #1613).
Every response that carries a watch serves aoi through a read-time chain,
decided by core.live_events.served_aoi, so the Events map has an area to
frame for every watch that has one anywhere:
- the stored
live_events.aoi, verbatim; - else the envelope of the union of the watch's snapshots'
hazard_datasets.footprint; - else the extent of the watch's selector — its exposure dataset, or its
portfolio's attachments, read through the rule behind a portfolio's own
bbox(selection_extent_sql), unbuffered; - else
null(no stored area, no snapshot footprint, an empty selector).
A fallback is the envelope itself as a GeoJSON Polygon, so a selector
holding one asset reads a zero-area polygon at that asset. Nothing in the
chain is stored and D7 holds for the stored column: a dataset-bound watch
still stores no aoi, PATCH {"aoi": null} still clears it, and the next
read serves the fallback. Feed polling reads the stored aoi only
(aoi_bounds, and the tick's own SQL envelope): a fallback area never bounds
a provider query, and a watch whose stored aoi is null still pauses its
feed for want of one. The repository resolves both envelopes in the summary's
one statement, for a watch with no stored area only, and measures the selector
only when that watch has no snapshot footprint either.
Migration 0108 stores a one-time backfill on closed watches only, which
never poll again: a closed row with a null aoi takes the envelope of its
snapshots' footprints, else the envelope of its selector's
exposure_datasets.bbox, when that envelope has an area. Open rows keep
null, and the upgrade logs how many watches are left without a stored aoi.
kind is always written explicitly. The API default is near_real_time;
live_events.kind's DB default is replay, and that is migration 0092's
backfill of the pre-Phase-15 rows, not a value any writer inherits.
created_by_user_id is the authenticated caller, taken from the request —
a body that asserts one is a 422 (extra="forbid").
The reads carry a summary, in one statement. GET /v1/live-events and
GET /v1/live-events/{id} return the watch's columns plus snapshot_count
(COUNT(hazard_datasets WHERE live_event_id), dated or not),
latest_valid_at, latest_event_loss (the headline, below) and feed
({id, kind, enabled, cadence_seconds,
effective_cadence_seconds, last_polled_at, last_error, last_poll_summary,
paused_reason} or
null — render the cadence and any "next poll" instant from
effective_cadence_seconds, issue #1444, and what a watch with no observation
is waiting for from last_poll_summary, described under the feed resource
below, issue #1614). All four are resolved
as correlated subqueries against the same scan — the feed through a
LEFT JOIN LATERAL … LIMIT 1, because uq_live_feeds_event_kind permits one
feed per provider and a plain join would fan one watch into several rows and
break the page's limit. A five-watch page therefore costs the same one
statement a one-watch page costs, which
tests/test_api/test_live_events.py::test_the_list_costs_the_same_number_of_statements_at_any_page_size
asserts by counting them. POST, PATCH and /close answer in the same
shape, so a client never has to follow a write with a read.
PATCH /v1/live-events/{id} (ANALYST+) takes any of name,
auto_compute, aoi, kind, deductible, cover and nothing else. The selector, haz_type,
source, started_at, status and ended_at are all refused with a 422:
the first three would re-attribute the watch's own history rather than
reconfigure it, and closing is /close's one-way job. Only the keys the
request actually set are applied, so {"aoi": null} clears the area and
{"cover": null} drops the cover, while an empty body changes nothing; name,
auto_compute and kind back NOT NULL columns, so an explicit null there is
refused rather than silently dropped. A closed
watch is E_LIVE_EVENT_CLOSED (422) — it is a record of what was watched, not
live configuration.
The curve gate (EV-13, issue #1615). A watch may switch auto_compute on
only when its hazard has a default curve for its selector's exposure type: an
is_default row of impact_functions for (haz_type, exposure_type), the
pair the automatic lane resolves on (core/live_feeds.py::auto_compute_plan,
through the same select_default_impact_function_id ladder)
and the one GET /v1/impact-functions?haz_type=&exposure_type= lists. Without
it, POST /v1/live-events with auto_compute: true and a PATCH that turns
the flag on answer 422 E_VALIDATION with K19's text and nothing is
written (core/live_events.py::_require_auto_compute_curve):
| Selector | Exposure type the gate reads | details of the refusal |
|---|---|---|
exposure_dataset_id |
the dataset's exposure_type |
reason: "no vulnerability curve for exposure type buildings", exposure_type: "buildings" |
portfolio_id, members of one type |
that type | the same, naming it |
portfolio_id, members of several types |
none: one curve answers one run, so the watch is refused whichever curves exist | reason: "no vulnerability curve for exposure type buildings, population", exposure_type: "buildings, population" (the sorted types, joined) |
| a portfolio with no catalogued member | buildings, the automatic lane's fallback |
as the first row |
So a heatwave watch on a buildings dataset is refused with auto_compute:
true and created as before with auto_compute: false (a hazard with no curve
is not refused on its own, drought included). The gate is asked once, at the
switch: a PATCH on a watch that is already auto-computing is not re-asked,
so a watch created before the gate keeps its flag and can still be renamed,
and a portfolio that later gains a second exposure type is caught where it
always was, by the automatic lane's log line and no job. The gate reads the
curve only; a mixed or non-money value_unit is still the automatic lane's
refusal.
Compute entry (issue #1027)¶
POST /v1/live-events/{id}/compute is the deterministic-event lane's entry
point, dispatching worker.tasks.compute_event_impact (issue #1025) exactly
like POST /v1/compute/impact dispatches compute_impact. It shares the
platform-wide exposure selector (exposure_dataset_id | portfolio_id,
issue #1008) and impact-function auto-resolution, but three things are
event-lane-specific:
- The selector is the watch's own (EV-12, issue #1610). A body naming
neither
exposure_dataset_idnorportfolio_idcomputes against the watch's selector, so the dispatched job scores that dataset or portfolio and not every exposure the organization holds. A body naming the watch's own selector is accepted. A body naming a different portfolio or dataset isE_LIVE_WATCH_SELECTOR_CONFLICT(422,details: {field, requested, watch_selector}), never a silent replacement: the result is filed under this watch, and a client that asked for another portfolio must hear that it did not get it. The rule iscore/live_events.py::resolve_event_compute_selector. A watch with no selector (a row older than issue #1287) takes the body's selector as given, since a person pressing Compute sees the scope they get. The automatic path is stricter:core/live_feeds.py::auto_compute_planreturns no plan for an auto-computing watch with neither selector (or with both) and logs why, because nobody asked for that run. hazard_dataset_idis optional: omitted, it resolves to the event's newest snapshot; supplied, it must be one of this event's own snapshots (a hazard dataset attached to a different live event is refused withE_VALIDATION, 422) — otherwise the produced result would be filed under the snapshot's own event, not the URL's.- References are validated by
core.compute_service.validate_event_impact_refs(issue #1023), the event lane's sibling tovalidate_impact_refs— it accepts aSCENARIO_YEARdataset (the shape every live-event snapshot is) and refuses aRETURN_PERIODone (E_EVENT_LANE_PROBABILISTIC_DATASET, 422), the opposite of the annual lane's axis rule.
GET /v1/live-events/{id}/results/{job_id} mirrors GET /v1/results/{id}
rather than extending it — that route emits ead/aai unconditionally, and
this payload must never carry either (phase guardrail 6, since an event loss
is one observed footprint, not an annual expectation). assets is the
per-point loss vector (EventImpactResult.loss_exp_path) joined back to the
named exposure rows, sorted by event_loss descending and paginated.
Per-asset intensity (issue #1612). compute_event_impact stores a second
vector, intensity_exp, beside loss_exp in the same archive
(results/{job_id}/loss_exp.npz) and in the same order: the footprint's
intensity at each asset's assigned centroid, in the snapshot's
intensity_unit. When a snapshot holds more than one event row, it is the
highest of them at that centroid, the reading exposed_count uses. Each
assets[] row serves it as intensity. It is null, never 0, for a result
scored before the worker stored the vector and for an asset the engine dropped
(no assigned centroid, stored as NaN); 0 is a reading, a cell the footprint
did not touch.
Per-admin-unit sums (issue #1612). admin_units breaks event_loss down
by GADM unit:
"admin_units": [
{"admin_boundary_id": "…", "name": "Evia", "admin_level": 2,
"event_loss": 1840000.0, "asset_count": 212}
]
- One row per unit, at each
result_cell_admin_levelslevel ([0, 1, 2]), that holds at least one asset, ordered by level, then byevent_lossdescending. A level with no loaded boundaries yields no rows. - The sum covers every asset of the result, whatever
limit/offsetthe request carries. An asset is counted in the unit its own coordinates fall in (ResultCellRepository.sum_points_by_admin_unit, the annual lane's sum), so an asset outside every loaded boundary counts in none and a level's rows can add up to less thanevent_loss. - The rows have no unit of their own; they read in the result's
unit. - The worker sums once, when it scores the result, and stores one JSON array at
results/{job_id}/admin_units.json(storage_keys.result_event_admin_units_key). The route only reads that object and never sums. The write is best-effort: a sum that fails is logged and leaves the result without an object. nullmeans the result has no stored sums, because it was scored before they existed or the write failed; it is recomputed by scoring the snapshot again.[]means the result was summed and no asset stood in a loaded unit.
Ranked-asset naming. An assets[].name is a display label, not the
stored exposures.name. A gridded exposure ingest writes the dataset's own
name onto every point row it emits — exposures.name is NOT NULL and a
LitPop/GHSL/Eurostat cell has no name of its own — so a book of such rows
would otherwise repeat one string on every line. core/display_names.py
resolves the label for both surfaces that rank those rows:
| Row | name |
dataset_name |
|---|---|---|
| Carries a name of its own | that name, verbatim — never rewritten | the dataset's full name, or null |
| Unnamed or dataset-named, and both a dataset and a point are known | composed "<dataset short name> · <lat>, <lon>", coordinates at 4 dp (e.g. LitPop · 37.7534, 26.9769) |
the dataset's full name |
| Anything else — no resolvable dataset, or ungeocoded | the stored name if it has one, else null |
the dataset's full name, or null |
Composing needs both halves: without a dataset there is no product to
attribute the row to, and a bare coordinate pair is not a name — it would only
restate the lat/lon the payload already carries. That last row is the
branch an asset-plane rollup member always takes.
The short name is the dataset name up to its first separator — one of --,
—, –, |, : — minus any parenthetical qualifiers, capped at 48
characters. So "LitPop -- Greece default asset-value exposure (150 arcsec,
2018)" reduces to LitPop. It is deliberately not unique: two datasets
sharing a product prefix reduce to the same short name, and the label's
distinguishing half is the coordinate pair. dataset_name always carries the
full name, so a client can show on hover the provenance the label
shortened, and tell two same-prefix datasets apart.
The portfolio rollup's worst_members[] carries the same two keys, resolved by
the same helper, so one row reads identically in both payloads. Both stay
null for an asset-plane member on purpose: the only human label an
assets row has is address, personal data under ADR-041 that the rollup path
never loads. The composed label discloses nothing new — both payloads already
carry each row's lat/lon. Both keys are additive, so
ROLLUP_SCHEMA_VERSION does not move.
GET /v1/live-events/{id}/results lists the event's completed jobs
(event_impact_results only ever holds a row for a job that finished); poll
GET /v1/jobs/{job_id} for a pending/running/failed submission, matching
every other compute entry point's convention. Every row carries cumulative,
read from the snapshot it scored (false when that snapshot is gone): a
wildfire poll scores an overpass and the event to date at one instant, and
the flag tells the two rows apart.
Insurance terms and the insured event loss (issue #1611)¶
A watch may carry the uniform deductible and cover the annual run
accepts (POST /v1/compute/impact): live_events.deductible and
live_events.cover, NUMERIC NULL (migration 0107), set on POST and
revised or cleared on PATCH. deductible is at least 0 and cover is above
0; anything else is a 422. Both are per exposure point and in the unit of the
exposure's value.
Every compute on the watch reads them when it is dispatched, the Compute
route and the feed's automatic compute through one function
(core/live_events.py::watch_terms_params), so the two lanes insure a watch
alike and neither takes terms from a request body. compute_event_impact
hands them to the engine adapter, whose per-point insurance transform runs
beside the ground-up calculation: one event of weight 1, no engine change.
The result stores insured_event_loss (event_impact_results, nullable)
beside event_loss, which stays ground-up. It is at most event_loss, in the
same unit, and null when the run had no terms. Nothing annual is derived
from either figure (ADR-078 item 1).
An edit through PATCH applies to computes dispatched after it. A stored
result keeps the terms it ran with as provenance.terms
({deductible, cover}, a term the watch lacked is null), and the block is
omitted for a run without terms.
The watch summary serves latest_insured_event_loss: the
insured_event_loss of the same result latest_event_loss reads (the
cumulative one for a wildfire watch), in latest_event_loss_unit. It is
null when that result ran without terms and when the watch has no result.
It is one more column of the row the summary statement already picks, so it
costs no statement and no read per watch.
Adaptation measures on an event (issue #1643, AD-9)¶
POST /v1/live-events/{id}/compute takes measure_ids and
adapted_from_result_id, the job id of the unadapted event result of
the same watch. The worker applies the measures to the snapshot's footprint
and curve before the impact, through the impact lane's loader and transform
(issue #1594), each masked by its applies_to (issue #1625), records both
ids in provenance and stores an ordinary event result. The refusals are a
422 with details.reason: an unknown measure, a measure of another hazard
than the snapshot's (AD-5), and a protection standard ("a protection
standard needs return periods; an event footprint has none"). A source that
is not a result of the watch is a 404. A source that is itself adapted, or
that scored another snapshot or exposure selector than the run, is a 422, so
the avoided loss is the measures' effect on one footprint and nothing else.
The result read serves avoided_event_loss: the source's event_loss
minus the adapted result's, in unit, computed on read and stored nowhere.
It is null for a result without a source, and when the source is gone or
reads in another unit. An event has one footprint and no annual frequency,
so no BCR, NPV or payback is derived from it.
An adapted result is never the watch's headline: latest_event_loss,
latest_insured_event_loss and latest_event_loss_unit read only results
whose provenance carries no measure_ids. The results listing and the
delta still take an adapted result by its job id.
The watch headline, its unit and the severity level (issue #1608)¶
Three members of the watch summary (GET /v1/live-events and
GET /v1/live-events/{id}) are read off the watch's results and snapshots:
latest_event_lossis theevent_lossof the newest result on acumulativesnapshot (the event to date, issue #1607), and of the newest result byvalid_atwhen the watch holds none — a ShakeMap watch, a pre-17b watch. An overpass's loss falls when the next pass sees less of the fire; the cumulative one cannot, so the headline never falls between polls. The field keeps its name and type; there is nolatest_cumulative_*twin.latest_event_loss_unitis the unit that loss reads in: the onevalue_unitshared by the rows the headline result scored (one unit or none, K27). A run addressed at an exposure dataset scored that dataset's rows, so itsvalue_unitanswers. A run addressed at a portfolio (or at no selector) scored the rows holding a centroid assignment for its snapshot — the members inside the snapshot's extent, not every member: a portfolio of EUR members inside the footprint and USD members outside it readsEUR, because only euros were summed. It isnullwhen the scored rows mix units (the loss itself is not re-defined), when the unit is blank, and when the watch has no result. A portfolio result whose snapshot has no assignment left to read falls back to the unit all current members share (the member-unit rule of the portfolio rollup, issue #1579). TheunitofGET /v1/live-events/{id}/results/{job_id}is the same reading (LiveEventRepository.result_units), so the watch and the result it headlines never name different units. A page holding a headline result costs one more statement, for the page, never one per watch.latest_severity_levelis the level the feed published for the event of the watch's newest dated snapshot: the valueprovenance.severity_levelsholds for that snapshot'sevent_key(the USGS PAGER level of a ShakeMap watch's newest earthquake, issue #1605). It isnullwhen the feed publishes none (FIRMS, a replay), when none is recorded, and when the watch has no snapshot.
Exposed count / value — how much stood in the footprint (issue #1293)¶
The result payload carries exposed_count and exposed_value beside
event_loss, because a decision-maker reads "N assets, V exposed" before any
loss figure. The rule, in one sentence:
An exposure point is exposed when the peak intensity of its assigned hazard cell, across the snapshot's event rows, is finite and strictly above the peril's first damaging intensity — the intensity above which the job's own impact function stops returning zero.
Three consequences, all deliberate:
- Per peril, because it is read off the curve the job ran with. The
threshold is not a platform constant. Resolved against the seeded catalogue
it reads: tropical cyclone 25.7 m/s (exactly the
v_threshits own seeder docstring names), European windstorm 20.02 m/s, JRC river/coastal flood 0 m, wildfire 295–300 K depending on resolution, ESRM20 earthquake 0–0.154 g depending on building class. The same grid therefore yields different counts under different curves, and that is the number being right, not a rounding difference. - The threshold is the last knot at which the curve still does nothing, not
the first knot at which it does something, because the engine interpolates
linearly between knots: a curve with
y = [0, 0, 1]atx = [0, 20, 20.02]already damages at 20.01. The impact factor ismdd(x) · paa(x), so the threshold is the larger of the two curves' own thresholds. The comparison is strict (>). - It is a hazard fact scoped by the curve, not a loss fact. An asset the
curve damages by 0.01 % is exposed. The pair answers "how much was in the
way", which is why it sits next to the loss rather than being derived from
it, and why an all-zero footprint reads
0/0.0rather than inheriting the loss's own zero.
NULL and 0 are different answers. Migration 0092 added both columns
nullable and did not backfill: every result computed before Phase 15 reads
null — "this lane never measured it" — while 0 means "measured, and nothing
was in the footprint". The rule lives in core/exposed_headline.py and is
applied by worker/tasks.py::compute_event_impact.
exposed_count never appears on an annual payload (GET /v1/results/{id},
which also serves portfolio-addressed annual results). An annual expectation
integrates a return-period distribution rather than scoring one footprint, so
there is no single event for an asset to have stood in. The annual
exposed_value (#1595, see "Exposed value and band shares on a result") is a
different quantity: the summed value the engine computed over, after the clip
and growth, null on a result that stamped none — never this footprint value.
(The exposed_value on GET /v1/portfolios/summary is a different, older
quantity — the book's insured value — and is unrelated.)
The delta — what changed since the previous observation (issue #1293)¶
GET /v1/live-events/{id}/results/{job_id}/delta?against=<job_id> compares two
results of the same watch, asset by asset. Every figure is oriented
addressed − against, so a positive number means the situation got worse
between them; the two are not required to be in time order, so a client
comparing the newest snapshot back to a baseline reads the same shape as one
walking a replay forwards.
{
"job_id": "…", "against_job_id": "…",
"valid_at": "2026-09-05T18:00:00Z", "against_valid_at": "2026-09-05T12:00:00Z",
"event_loss_delta": 41230.5,
"exposed_count_delta": 12, "exposed_value_delta": 8400000.0,
"assets": [
{"exposure_id": "…", "name": "LitPop · 38.0000, 23.8000",
"lat": 38.0, "lon": 23.8,
"loss_before": 1200.0, "loss_after": 9800.0, "delta": 8600.0}
],
"total_assets": 431
}
assetsis ranked by|delta|descending — the biggest movers first, improvements included — and paged withlimit/offset(default 100, cap 500).total_assetsis the count before paging.- The two sides are joined by
exposure_id, never by array position. Each side's stored vector is ordered by its own snapshot's centroid assignment (EXPOSURE_COMPUTE_ORDER_BY), and the two snapshots need not agree, so a positional zip would hand rows each other's losses — wrong without looking wrong, the same defectE_RESULT_STALEexists to prevent. exposed_count_delta/exposed_value_deltaarenullwhen either side never measured the headline, since reading thatNULLas zero would publish "nothing changed" about a number nobody took.
Refusals, none of them a 500:
| Condition | Answer |
|---|---|
| The watch is absent, another org's, or live mode is off | 404 E_LIVE_EVENT_NOT_FOUND |
Either job has no result on this watch — an against from another watch included, which the watch scoping hides rather than confirming |
404 E_LIVE_EVENT_NOT_FOUND |
Either result stored no per-asset loss vector (loss_exp_path is NULL, which the column permits by contract) |
409 E_RESULT_VECTOR_ABSENT |
| Either result's exposure set has drifted since it was computed, or the two cover different exposures | 409 E_RESULT_STALE |
E_RESULT_STALE answered 400 before this issue: it had no
main._CODE_TO_HTTP_STATUS entry, so every raiser had to build the 409 itself.
It now has one, alongside E_RESULT_VECTOR_ABSENT.
The SDK exposes the route as LiveEventsAPI.delta(live_event_id, job_id, *,
against, limit=…, offset=…) on both the sync and asyncio clients.
The three feed kinds (Phase 15 W1)¶
Every adapter subclasses ingest/feeds/base.py::LiveFeed, declares a
kind: ClassVar[str], and is validated at import time by
ingest/feeds/registry.py — a registered kind with no params model in
schemas/live_feed.py::PARAMS_BY_KIND fails the import rather than the first
poll. A poll returns Observations; the lane ingests each one through the
real ingest pipeline, so a feed snapshot is an ordinary hazard_datasets
row with live_feed_id set.
kind |
Adapter | Provider and window | Cadence floor |
|---|---|---|---|
replay |
ingest/feeds/replay.py |
a staged directory of archived rasters plus manifest.json, released one observation per poll in manifest order |
inherited 60 s |
firms_nrt |
ingest/feeds/firms_nrt.py |
NASA FIRMS near-real-time active fire, one poll = one window over the watch AOI | 300 s — from the documented 5,000-per-10-minutes key limit |
usgs_shakemap |
ingest/feeds/usgs_shakemap.py |
USGS FDSN event service, one poll = one bbox × magnitude × updatedafter window; the grid comes from product/shakemap/{eventid}/{source}/{updateTime}/download/grid.xml |
inherited 60 s — USGS publishes no rate limit, so none is invented |
The replay series layout. A replay is real data on its original clock,
not a fixture (ADR-077, phase guardrail 17): ReplayFeedParams.series_dir
names a directory holding manifest.json and the rasters it lists.
{
"series": "evia_2021_daily",
"haz_type": "wildfire",
"intensity_unit": "K",
"source": "NASA FIRMS",
"license": "…", "citation": "…", "source_url": "…",
"observations": [
{"file": "2021-08-03_bt.tif", "valid_at": "2021-08-03T11:12:00Z", "label": "…"}
]
}
valid_atis the provider's own observation instant, carried onto the snapshot verbatim — years old if that is the truth. The tick'snowis used for logging and nothing else, so every freshness surface reports a real, and visibly old, time.source,license,citationandsource_urlare required and are written to the snapshot unchanged. A series that cannot state its licence is not released.sourceis bounded at 255 characters — the width ofhazard_datasets.source— and refused at the manifest boundary, because a wider value would otherwise pass validation, commit the cursor, and then kill the insert minutes later (issue #1390);license/citation/source_urlareTextsince migration 0093 (#1388) and are deliberately unbounded.filemust be a bare filename insideseries_dir, andseries_diritself is validated lexically against the configured worker-volume roots —settings.dataset_source_download_dirandsettings.data_mount_root(/databy default) — so a feed row cannot turn the adapter into an arbitrary local-file reader. The check is path-only and runs even in a process with no volume mounted; existence is the adapter's problem at poll time.- The cursor is the index of the next unreleased entry, so a feed resumes
where it stopped and a poll past the end returns
[]and logslive_feed.replay.series_exhausted. An entry whose raster holds no usable cell is skipped, not released — the cursor advances either way, so a run of empty entries can never wedge the feed. - The adapter copies the raster into the poll's staging dir, which the
ingest deletes; the series itself is never mutated (
/datais mounted read-only onlive-workerandworker).
Staging a series on a deployment is a documented operator step — see DEPLOYMENT.md § "Staging a replay series on prod".
The feeds sub-resource and poll-now (issue #1295)¶
A watch says what is being observed; a feed is what fetches it — which
provider, how often, with what parameters, and where the last poll got to. The
five routes in the table above are that resource; the tick that consumes it is
worker/live_feeds.py, and both reach the database through the one service
module, core/live_feeds.py (phase guardrail 12), so the schedule the API shows
can never drift from the schedule the worker keeps.
{
"id": "…", "live_event_id": "…",
"kind": "firms_nrt", "params": {"confidence_min": "high"},
"cadence_seconds": 300, "effective_cadence_seconds": 300,
"enabled": true, "paused_reason": null,
"cursor": {}, "last_polled_at": null, "last_error": null,
"last_poll_summary": null,
"snapshot_count": 0, "max_snapshots": 500,
"created_at": "…", "updated_at": "…"
}
last_poll_summaryis what the last completed poll looked at and found (issue #1614, Phase 17b EV-1), so a watch that has seen nothing can say what it looked for, since when, and what the last poll found. A nullable JSONB columnlive_feeds.last_poll_summary(migration 0109), written byworker/live_feeds.py::_commit_pollin the same transaction as the cursor, andnulluntil the feed's first completed poll. Six keys, counts and the criterion only — no per-detection and no per-event detail:
| Key | Value |
|---|---|
polled_at |
When the poll ran, ISO 8601 UTC |
observations |
Observations the poll dispatched as snapshots. One the auto-computability gate dropped is not counted |
events_in_window |
Events the provider listed in the poll's window under the criterion: the usgs_shakemap discovery count. null for firms_nrt (it lists detections, not events) and replay |
criterion |
The feed parameter that decided what the window may hold, under its own name: {"min_magnitude": 4.5} for usgs_shakemap, {"confidence_min": "nominal"} for firms_nrt, {} for replay |
window_start |
Start of the poll's discovery window, ISO 8601 UTC: the updated_after a usgs_shakemap poll sent, the poll time minus DAY_RANGE days for firms_nrt, null for replay |
refused |
The auto-computability gate's refusal text when the poll dropped an observation, else null |
An adapter reports the middle three through one attribute,
LiveFeed.poll_window (ingest/feeds/base.py::PollWindow, default empty),
which it assigns inside poll(); poll() keeps its two return values. A
failed poll writes no summary: it stamps last_polled_at and last_error
and leaves the stored summary as it was, so the row keeps what the last
completed poll found beside why the newest one did not complete. Poll now
(POST …/feeds/{feed_id}/poll) keeps its 202 answer, and its outcome is read
back from this field.
-
effective_cadence_secondsis the longest the feed can wait to be polled (issue #1444), derived read-only in the read path and stored nowhere: the first beat at or aftercadence_secondsfalls due, taken over every beat a poll could have landed on. The tick selection asksnow - last_polled_at >= cadence_secondsand is only ever asked at a beat, so a request falling due between two beats waits for the later one. Wherelive_feed_tick_secondsdivides an hour — the 300 s default and 11 other values — this is exactlyceil(cadence_seconds / live_feed_tick_seconds) * live_feed_tick_seconds. Where it does not — the other 48 valuesSettingsaccepts — that formula is wrong, because the beat iscrontab(minute="*/N")and a cron minute step restarts every hour: a 420 s tick fires at :00 :07 … :56 and then :00, so a feed polled at :56 asking for 420 s falls due at :03, when no beat fires, and is polled at :07 — 660 s, whichceilwould report as 420. The derivation walks the real grid instead, andtests/test_core/test_live_feeds.pyasserts that grid equals the beat entry's owncrontab.minuteset for all sixty legal ticks. A sub-tick cadence is accepted and reported, never refused: the row keeps what its caller asked for, and production holds areplayfeed at 120 s against the 300 s tick — refusing would make aPATCHon that row unsavable, turning a reporting bug into a 422 wall. The per-kind floor is a separate and still-enforced rule; it refuses a hot loop, this one only describes the beat.GET /v1/infoadvertiseslive_feed_tick_secondsso a client can state the same number before the feed exists, and the watch's embeddedfeedsummary carrieseffective_cadence_secondstoo, because that is the shape a feed status strip renders a cadence and a "next poll" from. -
Creating a feed sets the watch's
kind—replayfor areplayfeed,near_real_timefor every other provider. The watch's clock is the feed's clock, and attaching the schedule is the moment that becomes known. Written after the insert succeeds, so a refused create leaves the watch untouched; a laterPATCH /v1/live-events/{id}may override it. paramsis validated against a per-kind model (schemas/live_feed.py'sPARAMS_BY_KIND) that forbids unknown keys, which is what refuses ahost/url/base_urlat the boundary rather than in the adapter (phase guardrail 22). It is stored as the caller wrote it, not as the model expanded it, so a default read from settings stays dynamic. APATCHreplaces the blob wholesale — a merge would let a removed key survive.snapshot_countis derived, never stored:count(*) FROM hazard_datasets WHERE live_feed_id = …, the same countmax_snapshotsis compared against, so the number a reader watches climb is the number that will pause the feed. It counts rows, not distinct instants — a tick that re-reads a window whose dispatch failed part-way commits the same observation again, and both rows count. That is deliberate (issue #1384): the cap is a storage guard, and a re-read row occupies storage like any other.enabled=trueis the un-pause gesture and clearspaused_reasonwhatever wrote it: a snapshot cap reached, an adapter's missing credential, or akindwhose adapter is not registered on this deployment. Each is a condition an operator can resolve, and before this resource existed there was no route that could clear any of them. The storedkindis therefore not re-validated for anenabled-only patch — it is re-validated only whenparamsorcadence_secondsis supplied, which are the values that need the kind's model to check them. Switching a feed off by hand leavespaused_reasonNULL, which is how "a person switched it off" stays distinguishable from "the system paused it".- A snapshot-cap pause needs both fields in one body:
{"enabled": true, "max_snapshots": N}withNabovesnapshot_count. The cap is re-checked before every poll, so{"enabled": true}alone succeeds and the feed pauses again on the next tick with the same reason. Re-enabling never raises the cap by itself (issue #1384).
POST …/feeds/{feed_id}/poll answers 202 {"feed_id", "queued": true} and
dispatches poll_live_feed on the worker's live queue — the queue the beat's
tick runs on, under the same per-feed lock, so a manual poll landing mid-tick is
a no-op instead of a duplicate snapshot. There is no job row to poll: the answer
is the watch's snapshots, read from GET /v1/live-events/{id}/snapshots.
Refusals across the five routes:
| Condition | Answer |
|---|---|
| Live mode is off | 422 E_LIVE_MODE_DISABLED |
| The watch is absent or another org's | 404 |
| The watch is closed (create and poll-now) | 422 E_LIVE_EVENT_CLOSED — poll-now asks the same "enabled and watch OPEN" the tick selects on, watch first (issue #1384); the queued task re-asks both when it starts |
No adapter is registered for the kind |
422 E_LIVE_FEED_KIND_UNKNOWN, naming the kinds this deployment can poll |
params fails the kind's model, cadence_seconds is under the kind's floor (the stricter of the adapter's own and the column's 60 s), or max_snapshots < 1 |
422 E_VALIDATION |
The watch already has a feed of this kind (uq_live_feeds_event_kind) |
409 CONFLICT, naming the feed in the way — a state conflict, not a bad body, and deliberately not a new registry code |
| The feed is absent, another org's, or not on this watch | 404 E_LIVE_FEED_NOT_FOUND |
| Poll-now on a disabled feed | 409 E_LIVE_FEED_PAUSED, carrying paused_reason |
More polls than live_feed_poll_rate_limit (default 10/minute) |
429 with Retry-After |
The SDK exposes all five as LiveEventsAPI.create_feed / list_feeds /
patch_feed / delete_feed / poll_feed on both the sync and asyncio clients
(climate-lama-client 0.4.0).
Pushed observations (issue #1294, Phase 15 W1.5)¶
POST /v1/live-events/{id}/observations (ANALYST+) is the push seam: a
client — a drone pipeline, a partner's detection product, a person — posts one
observed GeoTIFF and the instant it was observed at, and it becomes a snapshot
of the watch exactly as a feed's would. It is a route of its own rather than a
field on POST /v1/hazards/upload, whose single-event declaration is HDF5-only.
POST /v1/live-events/{id}/observations multipart/form-data
file GeoTIFF (.tif / .tiff) required
valid_at 2021-08-06T00:12:00Z required — timezone-aware
intensity_unit K required
label VIIRS_SNPP_SP 2021-08-06 optional
source partner product name optional
→ 202 {"data": {"job_id": "…", "ingest_job_id": "…"}}
- The instant is the caller's.
valid_atis required and never defaulted to the server's clock (phase guardrail 17), and a value with no UTC offset is refused rather than assumed to be UTC. Both required fields are parsed by the route's core gate, not the framework, so a missing one answers with theE_VALIDATIONenvelope like every other refusal of the declaration. - The unit must be auto-computable for the watch's peril: the route asks
core/live_feeds.py::live_snapshot_refusal— the same rule the feed tick applies (ADR-081) — so a unit no seeded curve consumes (wildfire's ownFWI, say) is refused rather than committed as a snapshot that would score to an impact of nothing. - GeoTIFF only. The suffix is a pre-filter and the file's first bytes must
be a TIFF signature, both judged before anything is staged — so a CLIMADA
store renamed
.tifcosts neither a quota read nor a disk write. - It is the upload route's staging. The body streams to
hazard_upload_dirunderhazard_upload_max_file_bytesand the pusher'shazard_upload_quota_bytes(both413, refused mid-body), and the bytes staged are charged to the pusher as any upload's are (owner_user_id+file_size_bytes). Feed-driven work is unmetered (ADR-081 decision 3); a push is a person's request, so it is not. - It is the mode-3 ingest. The job is
data_ingestwithorigin = user, carryingevent: {label, valid_at},live_event_id,intensity_unitandsource_format: "geotiff";params["file_path"]is what the api-side reclaim sweep (core/hazard_upload_reclaim.py) finds the staged leaf by. The committed snapshot carrieslive_event_idandlive_feed_id: null— which is howGET …/snapshotstells a pushed observation from a polled one — and on anauto_computewatch the ingest's commit hook (issue #1292) scores it like any other snapshot.
Refusals, all before a byte is staged except the 413s (which refuse
mid-body and unlink the partial file):
| Condition | Answer |
|---|---|
| Caller below ANALYST | 403 |
| Live mode is off | 422 E_LIVE_MODE_DISABLED |
| The watch is absent or another org's | 404 E_LIVE_EVENT_NOT_FOUND |
| The watch is not open | 422 E_LIVE_EVENT_CLOSED |
Not named .tif/.tiff or no TIFF signature; valid_at missing, unparseable or with no offset; intensity_unit missing or one no seeded curve for the peril consumes; source over 255 characters |
422 E_VALIDATION, details.field + details.reason |
| Over the per-file cap / the pusher's quota | 413 FILE_TOO_LARGE / 413 QUOTA_EXCEEDED |
The SDK method is LiveEventsAPI.upload_observation(live_event_id, path, *,
valid_at, intensity_unit, label=None, source=None) on both clients.
Settings.download_platform_allowlist (the admin download service's SSRF
allowlist, ADR-036) gained two live event-source hosts: earthquake.usgs.gov
(USGS earthquake feed) and firms.modaps.eosdis.nasa.gov (NASA FIRMS
active-fire detections).
Live-feed settings (issue #1280, Phase 15 W0.2)¶
The knobs the feed tick, the feed adapters and the poll-now limiter read. They
land ahead of their readers — on the commit that introduced them only
live_mode_enabled is read (by /v1/info.features) — so each consuming issue
adds a reader rather than a settings migration. Every app service in both
compose files already carries env_file:, so a key here reaches a deployment
through .env with no compose edit.
Setting (config.py) |
Env var | Default | What it governs |
|---|---|---|---|
live_mode_enabled |
LIVE_MODE_ENABLED |
false |
Master switch; surfaced as features.live_mode |
live_feed_tick_seconds |
LIVE_FEED_TICK_SECONDS |
300 |
Cadence of the live-feed-tick beat entry. A feed's own cadence_seconds decides whether it is due; this is the polling floor. Advertised on GET /v1/info and echoed per feed as effective_cadence_seconds (issue #1444), so a caller asking for a sub-tick cadence is told the interval it actually gets instead of the one it asked for. Rejected at construction below 60 (MIN_LIVE_FEED_TICK_SECONDS, issue #1367): Celery's crontab has a one-minute floor and the beat entry's expires is this same number, so a sub-minute value is published already-expired and the feed silently never ticks. Also rejected unless a multiple of 60 and at most 3600 (MAX_LIVE_FEED_TICK_SECONDS, issue #1384): the schedule is crontab(minute="*/<v // 60>"), so 90 would tick every 60 s against a 90 s expiry, and a minute step cannot span an hour — 7200 would tick hourly |
live_feed_max_snapshots |
LIVE_FEED_MAX_SNAPSHOTS |
500 |
Default ceiling on snapshots one feed produces before it pauses; live_feeds.max_snapshots overrides per row |
live_aoi_buffer_km |
LIVE_AOI_BUFFER_KM |
10.0 |
Buffer grown around a bound book's extent when a watch's AOI is derived. Stored on the row at creation, never recomputed — changing it affects later watches only |
live_feed_http_timeout_seconds |
LIVE_FEED_HTTP_TIMEOUT_SECONDS |
30.0 |
Per-request timeout for an adapter's outbound HTTP. Separate from dataset_source_http_timeout_seconds: a near-real-time poll gives up and retries on the next tick where a bulk download may not |
live_feed_poll_rate_limit |
LIVE_FEED_POLL_RATE_LIMIT |
"10/minute" |
Limit on POST /v1/live-events/{id}/feeds/{feed_id}/poll, which causes real third-party egress |
firms_map_key |
FIRMS_MAP_KEY |
"" |
Secret. NASA FIRMS MAP_KEY. Empty is a supported state: the FIRMS adapter pauses its feed with an operator-facing reason instead of polling unauthenticated |
usgs_min_magnitude |
USGS_MIN_MAGNITUDE |
4.0 |
Default magnitude floor for the ShakeMap feed's event query; UsgsShakeMapFeedParams.min_magnitude defaults from it, and a feed row may raise it |
data_mount_root |
DATA_MOUNT_ROOT |
"/data" |
The reference-data mount both compose files bind the host's ./data to. Together with dataset_source_download_dir it is one of the two roots a replay feed's series_dir may live under |
All but data_mount_root are declared in .env.example and .env.example.prod
below each file's tail marker. data_mount_root follows hazard_upload_dir and
dataset_source_download_dir — container paths pinned by compose, settings
rather than literals so a deployment free to mount elsewhere is not silently
refused, but not spelled out in the templates.
firms_map_key is the only secret in the group and is declared
Field(default="", repr=False, exclude=True), so it cannot leak through the two
accidents that actually happen: a repr(settings) in a log line or traceback
frame, and a settings.model_dump() into a diagnostics payload.
tests/test_core/test_live_feed_settings.py pins both, with a control asserting
that an ordinary field carrying the same sentinel is found. The same env var
also feeds the operator-side pack scripts/packs/firms_bt_event.py, which reads
os.environ directly — one key, one name, two readers.
Assistant settings (issue #1493, ADR-086)¶
Read by core/assistant/ and the provider adapters (#1494, #1644). The first eight are declared
in .env.example and .env.example.prod below each file's tail marker (ASSISTANT_PROVIDER and the four
tuning keys commented out at their defaults) and reach the api container through
env_file: with no compose edit.
Setting (config.py) |
Env var | Default | What it governs |
|---|---|---|---|
assistant_enabled |
ASSISTANT_ENABLED |
false |
Master switch. With the active provider's key it decides features.assistant and whether POST /v1/assistant/messages answers at all (503 E_ASSISTANT_UNAVAILABLE, reason="disabled" while off) |
anthropic_api_key |
ANTHROPIC_API_KEY |
"" |
Secret (repr=False, exclude=True, on the roster in tests/test_core/test_live_feed_settings.py). Empty with the flag on and provider anthropic answers reason="no_key" |
assistant_provider |
ASSISTANT_PROVIDER |
"anthropic" |
anthropic or openai (issue #1644): which adapter build_adapter() loads and which key no_key reads, one per deployment with no fallback between them. GET /v1/assistant/options reports it as provider. No OpenAI model id ships as a default: with openai, set ASSISTANT_MODEL and ASSISTANT_MODEL_CHOICES to OpenAI ids (e.g. gpt-5.5); an empty or claude-* model starts the app with one WARNING and answers reason="no_provider_adapter", and an org override naming a claude-* id falls back to the default with a warning log. Commented out in both .env templates |
openai_api_key |
OPENAI_API_KEY |
"" |
Secret, same treatment as anthropic_api_key; read only when the provider is openai, where empty answers reason="no_key". The OpenAI adapter (core/assistant/openai_adapter.py) makes one Chat Completions call with max_retries=0, sending assistant_effort as reasoning_effort unless the model is a non-reasoning family (gpt-3.5*, gpt-4*, chatgpt-*, gpt-audio*, *-chat-latest) |
assistant_model |
ASSISTANT_MODEL |
"claude-sonnet-5-5" |
Model id the adapter calls (Sonnet 5.5 since Phase 17b AS-1, issue #1567). A missing or retired model answers reason="provider_error" |
assistant_effort |
ASSISTANT_EFFORT |
"medium" |
low, medium or high, sent as output_config.effort |
assistant_model_choices |
ASSISTANT_MODEL_CHOICES |
claude-sonnet-5-5,claude-opus-5-5 |
Comma-separated allowlist an org admin picks the organization's override from (issue #1602), listed in this order by GET /v1/assistant/options. It governs overrides only: an ASSISTANT_MODEL outside it stays the default and logs one startup WARNING, and a stored override a later allowlist drops falls back to the default with a warning log. Commented out in both .env templates |
assistant_max_tokens |
ASSISTANT_MAX_TOKENS |
8192 |
Per-call output cap, thinking included |
assistant_timeout_seconds |
ASSISTANT_TIMEOUT_SECONDS |
60.0 |
Total wall-clock deadline for one request, every model call and query round included (asyncio.timeout; the SDK client makes no retries); past it the endpoint answers reason="timeout". Refused at startup at 90 or more, so it stays below the production proxy's 90 s read timeout on /v1/assistant/ and the client gets the 503 envelope, never a 504 page |
assistant_rate_limit |
ASSISTANT_RATE_LIMIT |
"10/minute" |
Request budget per organization (issue #1496), a limits rate string checked before any model call; past it the endpoint answers 429 E_RATE_LIMITED and other organizations are unaffected. An unparsable value is refused at startup. Not in the .env templates; set it in the environment to change it |
Assistant audit and request budget (issue #1496)¶
Each assistant request may make up to five paid model calls
(MAX_QUERY_ROUNDS + 1), so the budget counts requests, not model calls. Each
organization has its own budget, ASSISTANT_RATE_LIMIT (default 10/minute), keyed on the caller's
org_id rather than the client IP the SlowAPI limits (login, geocode, live-feed
poll) use (core/assistant/rate_limit.py, a limits moving window). It is checked
after the flag and key checks and before any adapter is built, so a refused
request costs nothing. The window lives in per-process memory, like the SlowAPI
limiter: each API process counts on its own.
An answered request leaves best-effort rows in audit_events (Admin → Audit,
GET /v1/audit-events), each with the caller's org_id, user_id and client IP
and resource_type assistant:
| Action | Rows | Metadata |
|---|---|---|
assistant.message |
one per request | stop, withheld_figures, and the names of the proposed actions and the tools run |
assistant.action_proposed |
one per call returned in calls |
action, call_id, input |
assistant.tool_called |
one per query tool the backbone ran | tool, call_id, input (the tool's input: result_id for get_result_summary, country_iso3 for get_coverage, q for find_portfolios, country/region for find_exposure_datasets, job_id for get_job, the selector for list_hazard_options, portfolio_id/scenario/horizon_year for get_portfolio_rollup) |
Audit covers actions, not transcripts: neither the user's text nor the model's
reply is ever written, because free text would sit forever in an append-only
table. A 422, 429 or 503 request writes no row, and a failed audit write is
logged and never fails the request.
Assistant query tools and the figure guard (issues #1495, #1603, #1604, #1650)¶
D11's tool set is the posted manifest plus seven read-only query tools the
backbone runs itself, in the same request (core/assistant/tools.py). They need
the caller's org scope and DB session, which only the backbone holds, so they are
built by the route from org-bound repositories (RLS stays the backstop):
| Tool | Input | Output |
|---|---|---|
get_result_summary |
result_id |
id, engine, scenario, horizon, ead, aai, insured_loss, freq_curve ([{return_period, loss}], ascending) and value_unit: the unit GET /v1/results/{id} serves (resolve_value_unit: the result's stamped unit, its exposure dataset's for a result written before the stamp), null where the result's rows mix units, so the reply names no unit (AS-5) |
get_coverage |
country_iso3 |
that country's row of GET /v1/catalog/coverage |
find_portfolios |
q |
at most 10 [{id, name, asset_count}] of the caller's org whose name contains q (case-insensitive), newest first; asset_count is the exposure_count GET /v1/portfolios serves. An empty q is a tool error (AS-2) |
find_exposure_datasets |
country (a name or ISO3), optional region |
{clip, datasets}. datasets is [{id, name, value_unit, exposure_type, is_default}], the rows GET /v1/exposure-datasets?country=<iso3> lists (derived country, superseded excluded); the one is_default: true row is default_exposure_dataset's pick, the list route's default_dataset_id (BC-2), never re-ranked here. clip is admin:<boundary id> of the unit of that country named exactly region (case-insensitive, lowest admin_level first), the grammar set_scope takes, and null without region. A country with no layer answers datasets: []; a country no loaded boundary names, or a region naming no unit of it, is a tool error naming the value (AS-4, BC-4) |
get_job |
job_id |
{status, result_id, error} of one compute job (status pending, running, completed, failed or cancelled; error is the job's error_message) |
list_hazard_options |
the exposure selector: portfolio_id, or exposure_dataset_ids (never a country) |
one entry per HazardType member, in enum order: [{haz_type, climate_driven, state, reason, money, money_reason, cells: [{scenario_label, horizon_year, hazard_dataset_id}]}], the hazards of GET /v1/compute/hazard-options from the same core function (hazard_options_for_selector, #1661). state is covered, no_single_dataset or not_covered and reason is null for covered, otherwise the details.reason of the refused /impact request (#1590's resolve_hazard_option, called once per hazard type, never re-derived); cells are the run contract's resolution, empty unless covered (a Today cell has horizon_year: null). money: false with money_reason (/impact's vulnerability refusal, e.g. "no money loss for value unit m3", or "indicator: a score, no money") marks a run with no money loss. A set of several countries is answered, never refused as a whole (K41): one dataset per country reads no_single_dataset. The selector is refused as /impact refuses it, as a tool error carrying its details.reason (a mixed-unit set: "exposure datasets differ in value_unit: EUR, m3"); another organization's portfolio is not found (AS-2, AS-4, EX-19) |
get_portfolio_rollup |
portfolio_id, optional scenario and horizon_year (the rollup's precedence when omitted) |
{id, name, scenario, horizon_year, member_count, headline, hazards, value_weighted, ead_by_hazard}, read by load_portfolio_rollup (core/reports/gather.py), the read behind a report's portfolio-rollup section and the payload GET /v1/risk/portfolios/{id} serves, so a drafted commentary quotes the figures the section prints (PF-6). headline is {hazard_type, score, label}, the worst banded hazard, null when none banded; hazards is [{hazard_type, band, worst_band}], band the typical band (the band of the value-weighted mean intensity) and worst_band the worst member's, each {score, label} or null. value_weighted is {combined_score, band_distribution, share_high_or_severe, exposed_value, unit, weighting, reason} as the rollup serves it: combined_score is the value-weighted average band (ordinal, screening, K25), band_distribution [{band, exposed_value, share}], share_high_or_severe the share in bands 7 to 10. A portfolio whose members mix units (K27) answers unit: null, exposed_value: null, weighting: "member_count" and reason: "mixed units", with member_count served. ead_by_hazard is [{haz_type, ead, unit, result_id}], one per hazard and never summed (K26). Nothing is summed, converted or re-banded here; another organization's portfolio is not found |
- A result, country, job or portfolio outside the caller's org answers the tool
result
{"error": "not found"}withis_error: true(get_jobanswers it for a malformed id too), never data. Nothing on this path writes: compute and save stay UI actions the user sees in the trace. - At most 4 query rounds per request (
MAX_QUERY_ROUNDSincore/assistant/service.py); a model turn asking for a fifth ends the request withstop: "max_rounds"and the unrun call is not appended. - Each round's exchange comes back in
append: the assistant turn with its calls, then atool_resultsentry. A turn that mixes a query and a named action runs the query and returns only the action incalls; the UI's latertool_resultsfor that turn is merged with the query results before the model sees it. - A posted action named like a query tool (
get_result_summary,get_coverage,find_portfolios,find_exposure_datasets,get_job,list_hazard_optionsorget_portfolio_rollup) answers422 E_VALIDATION. - "Run this scenario" (run model § 10) has two call sequences, both asserted
with a scripted adapter in
tests/test_api/test_assistant_run_sequence.py. A portfolio (AS-2):find_portfolios → open_view → set_scope → list_hazard_options → set_analysis → compute_cell → get_job → get_result_summary, withset_scopeonportfolio:<id>andlist_hazard_optionsonportfolio_id. A place (AS-4, BC-4):find_exposure_datasets → open_view → set_scope → list_hazard_options → set_analysis → compute_cell → get_job → get_result_summary, withset_scopeondatasets:<id>of the layerfind_exposure_datasetsflagsis_defaultandlist_hazard_optionsonexposure_dataset_ids. The assistant runs a hazard only when itsstateiscovered; otherwise, or whenmoneyis false (a score-only exposure such asm3), it stops afterlist_hazard_optionsand quotes thereasonormoney_reason.compute_cell's result is the job's terminal state the UI waited for; the figure reaches the reply only throughget_job'sresult_idandget_result_summary.
The unsourced-figure guard (core/assistant/grounding.py). The grounding
sources are every tool output in the posted history and in this request, the
user's own messages and the posted context. A numeral in an assistant text that
matches none of them, exactly or rounded at its displayed precision (with
k/thousand, M/million, bn/billion and % scaling), is replaced by
[figure withheld], and the response field withheld_figures counts the
replacements across append. Numerals inside alphanumeric tokens (SSP5-8.5,
RP100, 100km) and the number of a Markdown ordered-list item are not figures.
10. MVP Scope¶
| Dimension | Choice | Rationale |
|---|---|---|
| Hazard | River Flood (JRC European Flood data) | Freely available, EU-relevant, smaller datasets |
| Region | Greece | Local knowledge, EU market alignment |
| Exposure | Buildings (CSV upload or LitPop) | Simplest to ingest and validate |
| Calculation | Expected Annual Damage (EAD) | Fundamental metric |
| Output | GeoJSON FeatureCollection | Web mapping compatible |
11. Architecture Decision Records¶
docs/DECISIONS.md holds 79 ADRs; this is a partial index (Phase 0-2, plus
ADR-042 since the CLIMADA relationship note below depends on it) and is not
kept in lockstep with every phase — treat docs/DECISIONS.md itself as the
source of truth for the full set. Phase column maps each listed ADR to the
phase in which it was authored — see docs/plan/ for phase scope.
| # | Decision | Status | Phase |
|---|---|---|---|
| 001 | Backend-first architecture | Accepted | 0 |
| 002 | Two-service architecture (API + Worker) | Accepted | 0 |
| 003 | Model interface abstraction (pluggable engines) | Accepted | 0 |
| 004 | PostgreSQL + PostGIS for primary storage | Accepted | 0 |
| 005 | MinIO for object storage | Accepted | 0 |
| 006 | Celery + Redis for job queue | Accepted | 0 |
| 007 | FastAPI over Flask | Accepted | 0 |
| 008 | Vertical slice MVP scope | Accepted | 0 |
| 009 | Open Core business model | Proposed | 0 |
| 010 | Python 3.12 pin | Accepted | 0 |
| 011 | Repository structure | Accepted | 0 |
| 012 | Git conventions | Accepted | 0 |
| 013 | River Flood as MVP hazard | Accepted | 0 |
| 014 | Greece as MVP region | Accepted | 0 |
| 015 | JRC European Flood data | Accepted | 0 |
| 016 | Clean-room engine reimplementation (Apache 2.0) | Accepted | 0 |
| 017 | Engine as independent package (climate-lama-engine) |
Accepted | 0 |
| 018 | Marginal frequencies for EAD computation | Accepted | 0 |
| 019 | Adaptation measures as stored DB entities | Accepted | 0 |
| 020 | Dual engine strategy (climate-lama-engine + CLIMADA fallback) | Deprecated — this design (in-process CLIMADA queue, explicit engine param) was retired in Phase 0 (#19); engine plurality later returned in a different form as the out-of-process CLIMADA sidecar (ADR-042) |
0 |
| 021 | Async/sync boundary in Celery workers | Accepted | 0 |
| 022 | SQL unnest for bulk spatial inserts | Accepted | 0 |
| 023 | Impact function sources for TC, WF, WS (Emanuel 2011 + Eberenz 2021 / Lüthi 2021 / Klawa–Ulbrich 2003) | Accepted | 1 |
| 024 | Worker runtime has no CLIMADA dependency | Accepted | 2 |
| 042 | CLIMADA as an out-of-process sidecar engine | Accepted | 8 |
| 084 | Oasis LMF as a second out-of-process sidecar engine | Accepted | 17 |
12. Business Model and Strategy¶
Three-project structure¶
| Project | License | Status |
|---|---|---|
climate-lama-engine |
Apache 2.0 | Separate repo, fully implemented, publish to PyPI |
climate-lama (backbone) |
Apache 2.0 | This repo. Open-source at Phase 3. |
climate-lama-ui |
TBD | Own repo (split from backbone in Phase 2). Standalone deployable, consumes backbone API over HTTP. Private until license decided. |
Open Core model (ADR 009)¶
- Engine + backbone: open source (Apache 2.0)
- UI: license TBD (BSL, proprietary, or open -- needs legal review)
- Revenue: managed hosting (SaaS/PAYG), support contracts, premium features (SSO, audit, reports), data packages, training
Pricing tiers¶
| Tier | Price | Includes |
|---|---|---|
| Community | Free | Self-hosted, no support |
| Professional | $500-2,000/mo | Managed hosting, SSO, 10K calcs/mo, support |
| Enterprise | $2,000-10,000/mo | Dedicated instance, SLA, custom integration |
| Academic | Free managed | Verified .edu/.ac, limited compute |
CLIMADA relationship¶
- CLIMADA is GPL-3.0, maintained by ETH Zurich
- The backbone never imports CLIMADA in-process: no module in
src/climate_lama/(core service or worker) imports fromclimada.*, and no deployment artefact for the core service or worker installs it (ADR-024).climate-lama-engine(Apache 2.0, clean-room reimplementation) is the default, in-process engine. - CLIMADA is available as an opt-in, out-of-process sidecar engine
(
docker/climada-sidecar/, ADR-042): a separate container that onlyworker/models/climada_adapter.pytalks to over HTTP, selectable per request via theX-Engineheader. It runs only whereCLIMADA_SIDECAR_URLis configured; the backbone itself never has CLIMADA loaded into its own process. - Oasis LMF follows the same pattern (ADR-084):
docker/oasis-sidecar/is the only code allowed to importoasislmf,ods_toolsoroasis_data_manager. The backbone reaches it throughworker/models/oasis_adapter.pywhereverOASIS_SIDECAR_URLis configured. - Position: "CLIMADA as a service" -- same domain, better accessibility (API-first, self-hostable, no Python required)
- Maintain goodwill: cite CLIMADA, contribute upstream, do not position as fork
13. Phased Roadmap¶
Phase 0 -- Stabilize PoC (Weeks 1-4)¶
Engine: publish 0.1.0 to PyPI, CLEAN_ROOM.md, CHANGELOG, CI/CD.
Backbone: climada_adapter.py and [full-worker] removed, pre-commit hooks, GitHub Actions (lint/test/build), integration tests, health check, demo script.
UI: confirm control panel works end-to-end.
Phase 1 -- Multi-Hazard + Scenario + Cost-Benefit (Months 2-4)¶
Closed 2026-04-17 (~3 months ahead of the 2026-07-15 target). Engine 0.2.0 live on PyPI. Backbone and UI shipped all 22 scoped deliverables:
- Ingest:
BaseGeoTIFFIngestor(#42), TC/WF/WS ingestors (#47, #43, #48), scenario plumbing (#44) - Impact functions: sources ADR-023 (#49), TC/WF/WS seeds (#50–#52), per-hazard auto-resolution (#53)
- Measures + cost-benefit: table (#54), CRUD API (#55), Celery task (#56), endpoints (#57)
- Frequency curve: persist (#58), endpoint (#59)
- Insurance:
deductible/cover+insured_loss(#60) - UI: scenario selector (#62), TC/WF/WS activation (#63), freq-curve panel (#64), CB panel (#65)
- Polish (merged same phase): ExposureStep prop cleanup (#83), hazard-grid skeleton (#85),
is_defaultin impf response (#86), centralised exception handlers (#87)
Loose ends (tracked in docs/plan/phase-1-multi-hazard.md): #16 (bbox-scoped centroid assignment — defer to Phase 3), #37 (session-scoped event loop — informational).
Phase 2 -- Production Backend + UI Maturity (Months 4-7)¶
Auth (JWT + API key), RBAC, multi-tenancy (row-level security), structured logging, report generation (PDF/Excel), portfolio aggregation, UI split to own repo, production Docker Compose.
Phase 3 -- Managed Hosting + First Revenue (Months 7-10)¶
Open-source backbone, managed hosting infra, billing, landing page, academic tier, pilot engagements (GIZ, World Bank, Greek agency), data packages (JRC flood for all EU).
Phase 4+ -- Expansion (10-18+ months)¶
Uncertainty quantification in engine 0.3.0, custom impact function builder, data marketplace, SDKs, desktop app, CMIP6 adjustments, Kubernetes scaling.
14. Open Points¶
| Topic | When |
|---|---|
| UI license (BSL vs proprietary vs open) | Before Phase 3 |
| Apache 2.0 legal review for backbone | Before Phase 3 |
| Pluggable engine concept (user-selectable engines) | Phase 4 |
| Desktop app (Electron/Tauri + embedded engine) | Phase 4 |
| Uncertainty quantification in engine | Engine 0.3.0 |
| Pilot customer timing | Phase 2 entry |
15. Domain Concepts¶
- Hazard: Probabilistic event set (intensity at centroids + frequency). Types: river_flood, tropical_cyclone, wildfire, storm_europe.
- Exposure: Assets at risk (buildings, population). Points with value and location.
- Vulnerability (Impact Function): Intensity -> damage fraction. MDD x PAA = MDR.
- Impact: Hazard x Exposure x Vulnerability. Key metrics: EAD, AAI, per-event, per-exposure.
- Job: Async computation request. Submitted via API, executed by worker, results in DB.
- Centroid assignment: Links exposures to nearest hazard centroid. Required before impact calculation.
- Marginal frequency: ETL-computed frequency bands for return period maps (not exceedance frequency).
16. Usage Metering¶
Per-tenant usage is tracked so admins can see consumption today and so the future Stripe metered-billing wiring can read from the same schema without a migration. The schema decisions below are deliberately load-bearing — any change after launch means a data migration, so they are stable from this section forward.
What we record¶
- Billable actions in v1:
impactcompute andcost_benefitcompute. Storage and ingest are explicitly out of scope; they will get their own event types when needed. - Per-event row, not per-period roll-up at write time. Every billable
action writes one row to
usage_events; period totals are produced at read time from a materialized view. This keeps the write path cheap and the audit trail full. - A feed auto-compute is not a billable action (Phase 15).
compute_impacttakes ameteredparam, defaultTrue; the live lane dispatches its per-snapshot compute withorigin=feed,metered=Falseandnotify=False, so it writes nousage_eventsrow and raises nojob_completenotification. Nobody asked for it: a near-real-time watch enqueues one compute per released observation, indefinitely and without a person in the loop, so billing it would charge a tenant for the passage of time and notifying it would bury their own runs in the bell feed. What it does write, whatevernotifysays, is onewatch_updateitem to the watch's creator per scored frame snapshot (ADR-037, issue #1617). The flag covers both terminal states — an unmetered run books nothing on success or failure. A pushed observation (POST /v1/live-events/{id}/observations) is somebody's request, so it staysorigin=user, metered and notified, and its staged bytes are charged to the pusher's upload quota. The same split is whatGET /v1/jobs?include_automated=pages on (defaultfalse).
Schema (usage_events)¶
| Column | Type | Notes |
|---|---|---|
id |
UUID PK | gen_random_uuid() |
org_id |
UUID FK orgs | RLS-isolated via app.current_org_id |
user_id |
UUID FK users null | Forensic only; not used in aggregation |
job_type |
TEXT | 'impact' | 'cost_benefit' (CHECK constraint) |
job_id |
UUID | Links back to jobs.id (or impact_results.id on cache hit) |
started_at |
TIMESTAMPTZ | Task entry time, not enqueue time — queue wait is not billable |
completed_at |
TIMESTAMPTZ | Terminal state time |
compute_seconds |
DOUBLE PRECISION | Wall-clock; completed_at - started_at |
cpu_seconds |
DOUBLE PRECISION | os.times() user+system delta; nullable (subprocess engines may not report) |
result_bytes |
BIGINT null | Total artifact size (eai_exp.npz); null when not applicable |
cache_hit |
BOOL default false | Cache-hit events have compute_seconds≈0 and cpu_seconds=NULL |
status |
TEXT | 'succeeded' | 'failed' (CHECK constraint) |
recorded_at |
TIMESTAMPTZ default now() | Insert timestamp |
Indexes: (org_id, completed_at) for period queries, (job_id) for join-back.
RLS is enabled on the table with the same app.current_org_id GUC pattern
used by other domain tables.
Period semantics¶
Months, anchored on first-of-month UTC. The materialized view
usage_by_period aggregates (org_id, period) over status='succeeded'
rows only and is refreshed nightly at 02:00 UTC by Celery beat
(refresh_usage_view) using REFRESH MATERIALIZED VIEW CONCURRENTLY.
Consumers should treat the view as up-to-24h stale; the staleness window
is acceptable for billing-style aggregates.
Stripe-readiness contract¶
These fields are stable from this revision forward; future Stripe metered- billing integration will read them as-is:
compute_seconds,result_bytes,cache_hit,status,period- The CHECK constraint values (
'impact','cost_benefit','succeeded','failed') - The first-of-month UTC period anchor
New event types (storage, ingest) and additional metrics may be added in later revisions; existing column meanings will not be reinterpreted.
Admin endpoints¶
GET /v1/admin/usage?org_id=...&from=YYYY-MM&to=YYYY-MM— JSON envelope of{period, compute_seconds, run_count, result_bytes, cache_hit_count, tile_requests, api_calls}. A month appears when it has usage events or request counters.GET /v1/admin/usage.csv?...— same data, streamed as CSV (an unknown request count is an empty cell).GET /v1/admin/usage/daily?org_id=...&days=30(days1–90, issue #1653, registryadmin-12) —{days, summary}: one row per UTC day, oldest first and ending today,{date, runs, runs_by_engine, compute_hours, tile_requests, api_calls}, andsummary={runs, compute_hours, tile_requests, api_calls, active_people, members}. Runs (a cache hit is not a run) and compute hours come straight fromusage_events;runs_by_enginekeys each run by its job'sparams.engine_name(the default engine when absent).active_peoplecounts active members whoseusers.last_login_atfalls in the window;memberscounts active members.
RBAC: Role.ORG_ADMIN is required. The default org_id is the caller's own
org; explicit cross-org org_id returns 403 unless the caller's
users.is_platform_admin flag is true.
Map-tile and API-call counters (issue #1653)¶
core/request_usage.py is the one adapter over Redis for them. Every request
get_auth_context authenticates increments one field of two Redis hashes,
usage:{org_id}:{YYYY-MM} (UTC month, kept about three years) and
usage:{org_id}:{YYYY-MM-DD} (UTC day, kept 100 days), in one pipeline:
tile_requests for a /v1/tiles/... path, api_calls for any other /v1
path. An unauthenticated request counts nothing. The increment waits at most
50 ms; a Redis error or timeout is logged at WARNING at most once a minute
and never fails the request. A period with no counter reads both counts as
null, never 0. The counts are informational (D20: no billing, tier or
price) and live only in Redis — they are not durable usage events.
Admin audit log, user lens and admin level (issue #1652)¶
GET /v1/audit-events?source=people|assistant|api|system— the audit log filtered by who acted.sourceis derived from stored fields, never stored, in this order:assistantfor an action prefixedassistant.,apifor a row whosemetadata.viaisapi_key,systemfor a row with nouser_id,peoplefor the rest. Every row carries its derivedsource; an unknownsourceis422 E_VALIDATION.GET /v1/audit-events.csv— the same filters (action,target_type,target_id,user_id,source,from,to,all_orgs) and the same RBAC as the JSON route, streamed off a server-side cursor astext/csvwith the headercreated_at,user_email,source,action,target_type,target_id. Read-only.- Every audit row written for an authenticated request records how the caller
signed in:
metadata.via(jwt,api_keyorcookie) and, for a key,metadata.api_key_id. Rows written before this change carry no method and keep the source they derive to. GET /v1/admin/usersrows carrypersona, the user's own role lens (nullwhen none).POST /v1/admin/userstakes an optionalpersonathat becomes the invitee's starting lens. A lens never gates anything (D2).GET /v1/auth/mecarriesis_platform_admin, the level the UI shows its Organisation and Platform groups of Admin tabs by (K45). The flag hides tabs; each route keeps its own check —require_role(Role.ORG_ADMIN)for users, audit, usage and downloads, the platform-admin checks for orgs, pending ingests andall_orgs=true(tests/test_api/test_admin_gates.pypins them).
Prometheus¶
climate_lama_compute_seconds_total{org_id, job_type} is incremented in
the same code path that writes a successful usage_events row, so the
Grafana fleet view can render the same data live without hitting the DB.
The high-cardinality org_id label is acceptable while the platform
serves a small-multitenancy tenant base; once the tenant count exceeds
~1000 orgs the label should be bucketed (e.g. by plan tier).
17. What NOT To Do (Lessons from RISK WISE)¶
- No monolithic data loading -- use streaming, chunking, DB queries
- No synchronous compute -- all calculations via job queue
- No tight engine coupling -- always through adapter interface
- No Windows-only assumptions -- everything in Docker/Linux
- No business logic in API endpoints -- extract to
core/ - No bare
Exceptioncatches -- be specific - No print statements -- use logging
- No hardcoded config -- use settings
- No skipped migrations -- every schema change needs one