Skip to content

Exploration 04 — Answer-layer architecture RFC

Status: session 4, drafted 2026-07-26; ratified 2026-07-26 with amendments (see Decisions section — amendment A1 on surface economics, A2 reversing the ORSA-parallel sequencing). Inputs: 00-frame.md (diagnosis, granularity ladder), 02-personas.md (P1 product shapes), epics #366 / #367. Unblocks: #367 (portfolio workbench), Phase 7 re-cut, the phase definition.

Problem

The platform answers only scenario-shaped questions ("what happens if…?") through a batch path: configure → enqueue job → poll → read result. P1–P3 personas ask location-shaped questions ("what is the risk here?" — asset, portfolio, municipality) and expect a sub-second answer. Today every answer costs a compute job; an address has no identity; an admin unit is a rendering concern, not a queryable risk object.

What already exists (the design builds on, not beside)

Existing piece Role in the answer layer
ImpactResult.eai_exp_path — per-location expected-annual-impact vector (NPZ in MinIO) from every impact run The raw material of per-location risk already computed; today write-only for reports
result_cache — content-addressed cache keyed by deterministic SHA-256 The serving precedent: precomputed answers looked up, engine untouched
admin_boundaries — GADM 0/1/2 polygons, org-less reference data The rollup substrate for admin-unit risk objects
Hazard COGs + /v1/tiles/cog|mvt proxies (titiler/martin) Millisecond windowed reads of exact intensity at a point; visualization serving
Provenance stamp (engine/backbone versions, dataset SHA-256s, curve version) Explainability drawer content, already produced per run
Celery worker + scenario model The factory plane — unchanged

Conclusion: the answer layer is mostly persistence + indexing + banding of things already computed, not new science.

Design: two planes

FACTORY PLANE (exists)                       ANSWER PLANE (new)
scenario → job → worker → engine             lookup(lat/lon | address | admin unit | asset)
        │                                            │
        ├─ ImpactResult (as today)                   ├─ risk_cells (indexed, banded)   ← DB
        └─ NEW: surface writer ──────────────►       ├─ hazard COG windowed read       ← exact value
           (RiskSurface + RiskCell rows,             └─ provenance refs → explainability
            provenance-stamped)

Factory runs are ordinary scenario runs (same worker, same engine adapters, same provenance) whose results are additionally written to the surface store. The interactive path never computes; it reads.

A. RiskSurface — the factory-run registry

One row per hazard × region × scenario × horizon × engine × version: status, provenance stamp, source dataset refs, produced_at, cell resolution set. Surfaces built from public datasets are org-less reference data (like admin_boundaries); surfaces from private client data are org-scoped and RLS-covered.

B. RiskCell — the precomputed grid

H3 cells (recommended over native hazard grids or admin polygons as the index): resolution ladder maps to zoom and to rollups; deck.gl/tile-friendly; uniform across hazards with different native grids. Columns: h3_index, resolution, surface_id, metrics JSONB (intensity per RP, EAI density, banded score), geometry generated for tiling.

Storage discipline (the main cost trade-off): cells live in Postgres only at coarse/medium resolutions (r5–r8; Greece at r8 ≈ ~180k cells per surface — fine even across dozens of surfaces). Finer-than-r8 truth is not duplicated into the DB: a point lookup combines (1) the r7/r8 cell for banded score + context, and (2) a windowed COG read for the exact intensity/depth at the coordinate. This keeps the DB bounded and the answer exact.

C. Asset — the identity entity (new, distinct from Exposure)

Exposure stays what it is: a dataset-shaped bulk row (point + value + JSONB, dataset SHA-256) that engines consume. Asset is identity-shaped: one real-world thing with a lifecycle — address, geocoded point, optional building-footprint ref, attributes (construction, floors, use), org-scoped, portfolio membership, score history. Relationship: an Asset generates exposure rows for factory runs (one direction, never the reverse), and resolves against risk_cells + COGs for instant scoring. Evolving Exposure into Asset was considered and rejected: it would overload a bulk data row with identity semantics and break the dataset-hash contract.

D. Score layer — versioned banding + rollups

  • score_schemes (versioned, seeded like impact functions): per hazard, a banding function (e.g. EAI/value ratio or intensity thresholds → 1–10 and RAG), with citation. Scores are recomputed cheaply from stored metrics when a scheme changes — bands are presentation over metrics, never a replacement for them.
  • Rollups: asset (per-hazard scores + combined), portfolio (value-weighted + worst-N accumulation view), admin unit (aggregate of cells within the GADM polygon — admin-unit risk profiles become queryable objects).
  • Every score carries surface_id → provenance + curve + attribution = the explainability drawer, assembled from data we already stamp.

E. Query path (new read-only API surface)

  • GET /v1/risk/lookup?lat&lon[&scenario&horizon] → score card across hazards (cells + COG reads; no job, no engine).
  • GET /v1/risk/admin-units/{id} → unit profile (rollup + top drivers).
  • GET /v1/portfolios/{id}/risk → book rollup + accumulation (reads asset scores).
  • Geocoding behind an adapter (ADR-026 license check; self-hosted Nominatim is the default candidate for GR/EU) — core/geocoding.py + provider adapter, no SDK coupling.

What wave 1 (ORSA module) needs from this

The ORSA module is scenario-shaped and does not require the surface, Asset, or H3. It needs: (1) a scenario-matrix runner (two EIOPA scenarios × three horizons × portfolio = batched jobs over existing compute), and (2) the report composer (shared primitive; today reports.py is CSV + placeholder PDF).

Amendment A2 (ratification, 2026-07-26) — sequencing reversed. The draft proposed running ORSA as a fast-tracked stream in parallel with the foundations. Rejected by the owner: there is no delivery pressure on this repo (the corporate-track demo runs on the RAT codebase, not here), so build the foundations properly first and build ORSA on top of them — as an integrated report product from day one (the future workbench report tab), not a standalone tool retrofitted later. Phase-definition consequence: one sequenced track — shared primitives + answer-layer foundations (composer, beat, surface store, Asset, score schemes) → then the ORSA module riding on them.

Decisions to ratify (→ ADR candidates)

Numbering note: phase-7-extensibility.md still reserves ADR-033/034, which were meanwhile assigned to UI Layout / Secrets. Next free number is ADR-039; fix the phase-7 doc during its re-cut.

  1. ADR-039 candidate — Two-plane answer layer: RiskSurface + RiskCell on H3; DB cells to r8, exact values via COG windowed reads; factory = existing pipeline with a surface writer. RATIFIED with amendment A1:
  2. The factory plane keeps primacy. The fresh-run path (new exposures × hazards × engines on demand — the researcher's and analyst's workflow) is first-class forever; the answer plane accelerates repeat/location questions, it never replaces or degrades ad-hoc compute. (User note: precompute was already his proposal in the EIOPA/GIZ era — the caveat is about who it doesn't serve.)
  3. Surface saves are cache-semantic and resource-accounted. Runs write their spatial results into the surface store so surfaces mature organically over time (each run saves; saved surfaces resurface answers — both directions). Every save is size-tracked (per-surface row count, bytes in DB and MinIO), surfaced in observability, and subject to a retention/eviction policy like result_cache. Curated reference surfaces (public datasets, org-less) are built deliberately; org-scoped run saves accumulate within accounted bounds. The r8 DB cap stands.
  4. ADR-040 candidate — Asset as a first-class entity distinct from Exposure; one-directional Asset→Exposure generation. RATIFIED.
  5. ADR-041 candidate — Versioned score schemes + rollup semantics (asset / portfolio / admin unit); scores as presentation over stored metrics. RATIFIED.
  6. ADR-042 candidate — Geocoding adapter + provider choice (self-hosted Nominatim default). RATIFIED.

Open questions — resolved at ratification (2026-07-26)

  • Score-scheme calibration — resolved: v1 bands are self-set from published science / common-sense thresholds (with citations), shipped, and polished iteratively. External actuarial/scientific review is planned polish, not a blocking gate. (Advisory note kept on record: revisit before the first paid insurer engagement.)
  • Precompute scope — resolved: no big up-front precompute campaign. Greece-first reference surfaces where deliberately needed; otherwise surfaces mature organically through the cache-semantic run-saves of amendment A1, hardened by tests and demos over time.
  • H3 resolution set per hazard — resolved: set empirically during the first factory runs using the decision rule (coarsest resolution at which banding stops changing between adjacent cells); flood/wildfire expected ~r8, windstorm/heat ~r6–r7.
  • Seismic surfaces (#366 gate): OpenQuake outputs → same RiskSurface/RiskCell contract — validates that the surface schema is engine-agnostic from day one. (Still open; lands with the OpenQuake adapter scoping.)

Risks

  • Storage blowup if cells go too fine → mitigated by the r8 cap + COG reads; revisit only with evidence.
  • Score credibility — banded scores without sign-off invite the "unrealistic outputs" objection already raised in stakeholder discussions; schemes ship with citations and a validation note, and stay behind a feature flag until reviewed.
  • Asset/Exposure double bookkeeping — mitigated by the one-directional generation rule; any sync logic lives in one service.
  • Freshness — surfaces are versioned and rebuilt by scheduled factory runs; the celery-beat deployment gap (#291 + missing beat container) becomes load-bearing and must be fixed before the answer plane ships.

Amendment A3 — reassessment corrections (2026-07-26, code-grounded refuter pass)

Six corrections from adversarial verification against the actual codebase; each is binding on the ADRs when authored:

  1. eai_exp is per-EXPOSURE-POINT, not per-centroid (contract pinned in tests/test_worker/test_engine_adapter.py:121), ordered by assigned centroid array_index. The surface writer must geolocate values by re-joining exposure rows with the same ORDER BY — never by array position. Two existing serving paths already misassume position→centroid for sparse portfolios (api/v1/results.py geojson, api/v1/reports.py top-exposures): filed as #370. Also: eai_exp is not write-only — geojson/report paths serve it today.
  2. National reference surfaces need a synthetic national exposure dataset — eai_exp exists only where an org's exposures exist; EAI-density cells for a public/citizen surface require a LitPop-style national proxy exposure (source, ingest, document). Explicit work item in the phase definition; only intensity-per-RP metrics come "free" from hazard COGs.
  3. COG point reads: the path does not exist yet. Only the tile proxy exists; a /cog/point-style lookup is new code (titiler now deployed in prod via #369). Banding precedence corrected per hazard: for flood/wildfire the address-level band derives from the COG intensity at the coordinate (r8 cell = context/rollup only — a 0.74 km² hex spans floodplain and hillside); cell-derived bands are authoritative only for area rollups and smooth hazards (windstorm/heat).
  4. RLS pattern decision: the existing exact-match RLS policy cannot serve one table mixing org-less reference rows and org-scoped rows. Separate tables: reference surfaces/cells as org-less reference data (like admin_boundaries, no RLS), org-scoped surfaces/cells under the standard FORCE-RLS pattern. Isolation invariant now explicit: cells derived from any org-scoped exposure dataset are never readable cross-org; promotion to reference status is a deliberate, manual act, never automatic.
  5. ADR-040 completions: Portfolio must be re-pointed at Asset (new portfolio_assets junction — portfolio_exposures FKs exposures.id today) with a migration story for existing exposure-based portfolios; and dataset_sha256 needs a defined recipe for Asset-generated exposure batches (hash over canonicalized asset-derived rows) or the result cache silently never hits for workbench runs.
  6. Eviction rules corrected (the result_cache analogy fails — the answer plane has no compute fallback on miss): evict org-scoped run-saves only (owner can re-run to restore); reference surfaces are never evicted, only replaced atomically by versioned rebuilds; lookups define explicit miss semantics (partial answer + "surface not built here" provenance), never a silent fallback to a coarser resolution.

Decided / Parked / Killed (session 4, ratified)

Decided - All four ADR candidates ratified (039 with amendment A1: factory-plane primacy + cache-semantic, resource-accounted surface saves). - Sequencing (A2): foundations first, ORSA on top — no fast-tracked parallel ORSA stream; the phase definition is one sequenced track. - Score bands v1 self-set with citations, ship-and-polish; external review advisory, non-blocking (revisit before first paid engagement). - Surfaces mature organically via run-saves; Greece-first for deliberate reference builds; per-hazard H3 resolutions set empirically in the first factory runs.

Parked - Building-footprint data source choice (OSM vs ELSTAT) — needed for Asset at wave 2/3, not for the schema. - Live/forecast surfaces (EFAS-driven alerting) — same RiskSurface contract with a time dimension; design later, don't preclude now. - Seismic surface contract validation — with OpenQuake adapter scoping (#366 gate).

Killed - Storing full-resolution risk rasters in Postgres (COGs + windowed reads win). - Evolving Exposure into the Asset entity. - The "ORSA in parallel to demo in weeks" sequencing (superseded by A2).