Architecture Decisions Log¶

This document captures key architectural decisions made during the design of Climate-Lama. Each decision includes context, the choice made, alternatives considered, and rationale.

Deferred work that arises from these decisions is tracked as GitHub Issues. Where a decision produces a follow-up task, the relevant ADR entry includes a ### Follow-up note with the issue number.

Template¶

## ADR-NNN: [Title]

**Date**: YYYY-MM-DD
**Status**: Proposed | Accepted | Deprecated | Superseded by ADR-NNN
**Phase**: 0 | 1 | 2 | 3 | ...   ← phase in which the ADR was authored; see docs/plan/

### Context
### Decision
### Alternatives Considered
### Rationale
### Follow-up  ← only when there is a linked GitHub Issue or successor ADR

ADR-001: Backend-First Architecture¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Previous projects (RISK WISE, CLIMADA-App) were desktop-first applications with embedded backends. This made them difficult to extend, integrate with other systems, or deploy as services.

Decision¶

Build Climate-Lama as a backend-first platform where the API is the primary product. Desktop and web applications are consumers of the API, not the core.

Alternatives Considered¶

Desktop-first (like RISK WISE): Electron + Flask, portable bundle
Web-first: Traditional web app with backend
Backend-first: API as product, multiple consumers

Rationale¶

Enables multiple consumers (web, desktop, CLI, third-party integrations)
API can be deployed independently of any UI
Better separation of concerns
Easier to test and maintain
Supports both cloud and self-hosted deployments

ADR-002: Two-Service Architecture for MVP¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

The original design proposed four services (Ingest, ETL, Compute, API). This is appropriate for a mature product but adds overhead for an MVP.

Decision¶

Collapse to two deployable units for MVP: 1. Core Service: FastAPI monolith with internal modules (Ingest, Data, Catalog, API) 2. Compute Worker: Isolated Celery worker running climate-lama-engine via EngineAdapter

Alternatives Considered¶

Full microservices (4 services): Maximum flexibility, maximum overhead
Complete monolith (1 service): Simplest deployment, but CLIMADA isolation problem
Two services (chosen): Balance of simplicity and necessary isolation

Rationale¶

CLIMADA has heavy dependencies that should be isolated
No need for 4 inter-service contracts before we have users
Can split further when scaling demands it
Reduces deployment complexity for MVP

ADR-003: Model Interface Abstraction¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

CLIMADA is the default engine, but users have expressed interest in: - Using other models - Bringing their own models - Custom impact functions

Decision¶

Define an abstract ModelInterface that all computation engines must implement. climate-lama-engine is the current implementation; the original ClimadaAdapter stub was removed in Phase 0 (#19).

Alternatives Considered¶

Direct CLIMADA integration: Simpler but locks us in
Model interface (chosen): Extra abstraction layer but enables extensibility
Plugin system: More complex, overkill for MVP

Rationale¶

Enables future model support without core changes
Standardizes result format regardless of engine
Keeps CLIMADA-specific code in one place
Users who want custom models have a clear integration path

ADR-004: PostgreSQL + PostGIS for Primary Storage¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Climate data is inherently spatial. Need efficient storage and querying of: - Point data (hazard centroids, asset locations) - Polygons (regions, building footprints) - Bounding box queries - Spatial joins

Decision¶

Use PostgreSQL with PostGIS extension as the primary database.

Alternatives Considered¶

PostgreSQL + PostGIS (chosen): Mature, powerful spatial support
MongoDB with GeoJSON: Flexible schema, but weaker spatial query performance
Specialized GIS database (e.g., CockroachDB Geo): Less mature ecosystem
File-based (GeoPackage): Not suitable for concurrent access

Rationale¶

PostGIS is the industry standard for spatial data
Strong ecosystem (GeoAlchemy2, Shapely, etc.)
Supports complex spatial queries efficiently
Can be self-hosted easily
Rich indexing options (GiST, BRIN)

ADR-005: MinIO for Object Storage¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to store: - Raw ingested files (HDF5, NetCDF, GeoTIFF) - Large computation results - User uploads

Files can be large (GBs) and shouldn't bloat the database.

Decision¶

Use MinIO for object storage with S3-compatible API.

Alternatives Considered¶

MinIO (chosen): S3-compatible, self-hostable, easy to swap for real S3
Local filesystem: Simple but doesn't scale, no API
PostgreSQL large objects: Bloats database, backup complexity
Direct S3: Vendor lock-in, cost for self-hosted scenarios

Rationale¶

S3 API compatibility means easy migration to cloud S3
Self-hostable for offline/air-gapped deployments
Handles large files efficiently
Separate concern from relational data

ADR-006: Celery + Redis for Job Queue¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Climate risk calculations can take seconds to minutes depending on dataset size. Cannot block HTTP handlers.

Decision¶

Use Celery with Redis as message broker for async job processing.

Alternatives Considered¶

Celery + Redis (chosen): Battle-tested, Python-native
Celery + RabbitMQ: More features, but Redis suffices and is simpler
Dramatiq: Simpler API, but smaller ecosystem
ARQ: Async-native, but less mature
Background threads: No persistence, no distribution

Rationale¶

Celery is proven at scale
Redis serves dual purpose (queue + cache)
Easy to add workers horizontally
Job persistence across restarts
Good monitoring tools (Flower)

ADR-007: FastAPI Over Flask¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

RISK WISE used Flask. Need to choose framework for new implementation.

Decision¶

Use FastAPI for the API layer.

Alternatives Considered¶

FastAPI (chosen): Modern, async-native, automatic OpenAPI
Flask: Familiar from RISK WISE, but sync-only without extensions
Django REST Framework: Full-featured but heavyweight for API-only
Starlette: Too low-level, FastAPI builds on it anyway

Rationale¶

Async support out of the box
Automatic OpenAPI/Swagger documentation
Pydantic integration for validation
Better type hint support
Modern Python patterns

ADR-008: Vertical Slice MVP Scope¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to prove the architecture works end-to-end without building everything.

Decision¶

MVP scope: - ONE hazard type (River Flood, likely) - ONE region (TBD) - ONE exposure type (Buildings) - ONE calculation (Expected Annual Damage) - ONE output format (GeoJSON)

Alternatives Considered¶

Full feature set: Everything from original spec — too much for MVP
Data layer only: No compute, doesn't prove end-to-end
Vertical slice (chosen): Full pipeline for narrow scope

Rationale¶

Proves entire architecture works
Delivers tangible output quickly
Identifies integration issues early
Can demo to stakeholders
Foundation for horizontal expansion

ADR-009: Open Core Business Model¶

Date: 2026-04-03
Status: Proposed
Phase: 0

Context¶

Need to balance: - Open source values (transparency, community, adoption) - Sustainability (revenue for ongoing development) - Target market (dev/research/regulatory, not enterprise SaaS)

Decision¶

Open Core model: - Core backend (all 4 modules) — open source (Apache 2.0 or MIT) - Reference web app — open source - Monetization via: managed hosting, support contracts, custom integrations, training

Desktop app licensing TBD.

Alternatives Considered¶

Fully open source, donations only: Unsustainable
Proprietary: Doesn't fit target market
Open Core (chosen): Balance of openness and sustainability
SaaS-only: Conflicts with self-hosted requirement

Rationale¶

Target users (GIZ, UNU, EIOPA) value transparency
Open source enables adoption and contribution
Services revenue is sustainable model
Self-hosted deployments are core requirement

ADR-010: Python 3.12 Pin¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to choose Python version. CLIMADA and some dependencies have version constraints.

Decision¶

Pin to Python 3.12 for Docker and development.

Alternatives Considered¶

Python 3.10: Too conservative, missing useful features
Python 3.11: Viable but 3.12 is current stable
Python 3.12 (chosen): Current stable, good ecosystem support
Python 3.13+: Too new, dependency compatibility issues

Rationale¶

climate-lama-engine and all runtime dependencies support 3.12
Modern features (better type hints, performance)
Good library compatibility
Consistent with lama-chat project conventions

ADR-011: Repository Structure¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to organize code for maintainability and clear boundaries.

Decision¶

Modular monolith structure:

src/climate_lama/
├── api/       # HTTP layer
├── core/      # Business logic
├── models/    # Domain models
├── db/        # Database access
├── storage/   # Object storage
└── worker/    # Celery tasks

Alternatives Considered¶

Flat structure: All modules at top level — messy at scale
By feature: Group by domain feature — harder to enforce layer boundaries
By layer (chosen): Clear dependency direction, easy to enforce rules

Rationale¶

Clear separation of concerns
Dependency direction is obvious (api → core → db)
Easy to understand for new contributors
Can be split into separate packages later if needed

ADR-012: Git Conventions¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need consistent git workflow across the project.

Decision¶

Adopt conventions from lama-chat: - Branch naming: feat/42-short-description - Commit format: type(scope): description - Resolves #N only on final commit of branch

Alternatives Considered¶

GitHub Flow (branch + PR only): No commit format standard — harder to trace changes to decisions
Conventional commits (chosen): Machine-readable type prefix, issue traceability, clean history
Trunk-based development: Too risky without a comprehensive test suite

Rationale¶

Consistency with existing projects
Clear traceability to issues
Clean git history
Standard conventional commits format

ADR-013: River Flood as MVP Hazard¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to choose one hazard type for the vertical slice MVP.

Decision¶

Use River Flood as the MVP hazard type.

Alternatives Considered¶

Tropical Cyclone: Best CLIMADA support, dramatic visuals
River Flood (chosen): EU-relevant, JRC data available, smaller datasets
Wildfire: Growing relevance, but less mature in CLIMADA

Rationale¶

JRC European Flood data is freely available from European Commission
More relevant to Greece/EU market than tropical cyclones
Flood datasets tend to be smaller than TC, easier for development
Strong regulatory push (EU Floods Directive) creates demand

ADR-014: Greece as MVP Region¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need to choose one region for the vertical slice MVP.

Decision¶

Use Greece as the MVP target region.

Alternatives Considered¶

Caribbean: Strong TC test data, insurance market relevance
Switzerland: CLIMADA's home base, lots of examples
Greece (chosen): Home turf, accessible test data, EU market

Rationale¶

Local knowledge reduces unknowns
Test data accessibility (can validate against known conditions)
EU market alignment (CSRD, EU Taxonomy)
Potential for early adopter connections

ADR-015: JRC European Flood Data as Primary Source¶

Date: 2026-04-03
Status: Accepted
Phase: 0

Context¶

Need a reliable flood hazard data source for MVP.

Decision¶

Use JRC European Flood data from the European Commission's Joint Research Centre.

Alternatives Considered¶

CLIMADA API flood data: Convenient but may have coverage gaps
JRC European Flood (chosen): Authoritative, EU-wide, open access
Copernicus Emergency Management: More recent events, but different format

Rationale¶

European Commission is authoritative source for EU
Data is open access
Well-documented format
Covers entire EU including Greece
Return period maps available (10, 20, 50, 100, 200, 500 year)

ADR-016: GPL-3.0 Clean Room Reimplementation¶

Date: 2026-04-04 Status: Accepted
Phase: 0

Context¶

CLIMADA is licensed under GPL-3.0 (copyleft). Climate-Lama aims to use Apache 2.0 or MIT for its open core components (ADR-009). A clean-room reimplementation of CLIMADA's calculation engine is planned as the open-source foundation of the platform.

Decision¶

The lean engine (climate-lama-engine) is a clean-room reimplementation. No CLIMADA source code is copied or derived. The mathematical formulas (linear interpolation, matrix multiplication, trapezoidal integration) are mathematics and not subject to copyright. The implementation is written from scratch using the mathematical specification in CLIMADA_SEED.md.

Alternatives Considered¶

Fork CLIMADA under GPL-3.0: Simpler starting point but locks all downstream code to GPL-3.0 copyleft
Clean-room reimplementation (chosen): Free to license as Apache 2.0 or MIT; enables open core model
Wrapper only (no reimplementation): Does not solve the heavy dependency problem

Rationale¶

Target users (GIZ, UNU, EIOPA-type) and the open core model (ADR-009) require a commercially permissive license
The math is ~490 lines of numpy — clean-room is entirely feasible
Direct CLIMADA imports were constrained to ClimadaAdapter; that adapter was removed in Phase 0 (#19) once climate-lama-engine was validated

ADR-024 records the operational consequence: the worker runtime no longer imports CLIMADA at all, which is what makes the permissive-license story watertight.

ADR-017: Engine as Independent Python Package¶

Date: 2026-04-04 Status: Accepted
Phase: 0

Context¶

climate-lama-engine could be implemented as a module inside src/climate_lama/worker/models/ or as an independent, separately versioned Python package.

Decision¶

Implement as an independent pip-installable package (climate-lama-engine). Climate-Lama's worker adds it as a dependency via pip. The package has no knowledge of FastAPI, SQLAlchemy, Celery, PostGIS, or MinIO.

Alternatives Considered¶

Module inside Climate-Lama worker: Simpler initially, but couples versioning and prevents reuse
Independent package (chosen): Independent versioning, desktop-embeddable, testable in isolation, open-source foundation of the open core model

Rationale¶

Independent versioning: engine improvements do not require a Climate-Lama release
Embeddable in desktop applications without pulling in the full backend stack
Testable in isolation without Docker Compose
Becomes the "open core" referenced in ADR-009
Interface contract: arrays in, arrays out — no framework dependencies

Related¶

ADR-024 captures the runtime-side follow-through: because the engine is an independent package, the worker can depend on it exclusively and drop CLIMADA from the deployed image.

ADR-018: EAD Computation Using Marginal Frequencies for Return Period Map Inputs¶

Date: 2026-04-04 Status: Accepted
Phase: 0

Context¶

JRC flood data provides deterministic intensity maps at 6 return periods (10, 20, 50, 100, 200, 500 year). These are exceedance intensity levels, not independent probabilistic events. Applying CLIMADA's formula aai_agg = sum(damage_i × 1/RP_i) directly overcounts EAD because a 1-in-100-year event also contains 1-in-50-year level damage — these are not mutually exclusive occurrences.

Decision¶

The ETL converts return period maps to marginal event frequencies before writing to the database. For sorted return periods RP_1 < RP_2 < ... < RP_n with exceedance frequencies f_i = 1/RP_i:

marginal_frequency[i] = f_i − f_{i+1}   for i = 1 … n−1
marginal_frequency[n] = f_n              (tail probability)

With marginal frequencies stored in hazard_events.frequency, the engine formula aai_agg = at_event · frequency is numerically equivalent to trapezoidal integration under the loss-exceedance-probability curve — the correct actuarial EAD.

For JRC return periods [10, 20, 50, 100, 200, 500]:

RP (yr)	Exceedance freq	Marginal freq (stored)
10	0.1000	0.0500
20	0.0500	0.0300
50	0.0200	0.0100
100	0.0100	0.0050
200	0.0050	0.0030
500	0.0020	0.0020

For true probabilistic event sets (e.g., TC synthetic tracks): use occurrence frequency directly. The hazard_events table includes a frequency_type column: 'marginal' | 'occurrence' | 'exceedance'.

Alternatives Considered¶

Direct exceedance frequencies: Simple but produces incorrect (overcounted) EAD for RP map inputs
Marginal frequencies at ETL time (chosen): Correct EAD; engine formula stays simple
Exceedance curve integration in engine: Also correct, but requires more complex engine logic; better to solve at ETL boundary

Rationale¶

Correctness is non-negotiable for regulatory and research users
Solving at the ETL boundary keeps the engine formula simple and universally correct
frequency_type tag makes the conversion auditable and reproducible

ADR-019: Adaptation Measures as Stored Database Entities¶

Date: 2026-04-04 Status: Accepted
Phase: 0

Context¶

CLIMADA's Measure class applies parametric modifications to hazard intensity, MDD, and PAA at compute time. The current POST /v1/compute/impact endpoint has no mechanism for measures. The cost-benefit analysis job type also requires them.

Decision¶

Measures are stored entities in the measures database table, referenced by UUID in job submissions. A new POST /v1/compute/cost-benefit endpoint is added alongside the enhanced POST /v1/compute/impact.

Key measure parameters stored per row: - haz_inten_a, haz_inten_b: linear transform of hazard intensity (new_I = a × I + b) - mdd_a, mdd_b: linear transform of MDD - paa_a, paa_b: linear transform of PAA - freq_cutoff: remove events above this annual frequency (low-severity filter) - cost, cost_unit: for cost-benefit ratio calculation

Job submission body gains "measure_ids": ["uuid1", "uuid2"] and "engine" fields.

Full schema and API specification in CLIMATE_LAMA_ENGINE_SPEC.md Section 5.1 and Section 6.

Alternatives Considered¶

Inline parameters in job body: Flexible but non-reusable and harder to audit
Stored entities (chosen): Named, reusable, auditable; consistent with how impact functions are stored

Rationale¶

Measures are reusable across multiple jobs and scenarios
Named measures (e.g., "Flood Wall 0.5m") are meaningful to end users in the UI
Consistent with the existing pattern for impact functions
Supports cost-benefit analysis across a set of pre-defined measure options

ADR-020: Dual Engine Strategy with Explicit Selection Parameter¶

Date: 2026-04-04 Status: Deprecated
Phase: 0

Context¶

The full CLIMADA adapter and climate-lama-engine would coexist during a transition period. Results need to be validated before climate-lama-engine becomes the default.

Decision¶

Add engine: "climada" | "climate-lama-engine" parameter to all compute job submissions
Default: "climada" during transition; "climate-lama-engine" after validation passes
Celery routing keys direct jobs to separate worker pools:
climada queue → full CLIMADA worker (existing container, ~2GB)
compute queue → climate-lama-engine worker (new container, ~200MB)
This enables A/B validation: same job submitted to both engines, aai_agg compared

Alternatives Considered¶

Hard cutover: Simpler but risky — no fallback if engine has edge case bugs
Dual engine with explicit selection (chosen): Safe migration path; A/B validation built in
Feature flag only: Less explicit, harder to control per-job

Rationale¶

Target users (researchers, regulators) require validated, trustworthy results
A/B comparison is the proof that validates the clean-room reimplementation
Engine container reduces worker memory footprint; more workers can run on the same hardware
Fallback to full CLIMADA remains available indefinitely for edge cases

Follow-up¶

Deprecated in Phase 0: the full CLIMADA adapter and its queue were removed (#19) after climate-lama-engine was validated as the sole compute engine. The compute queue and worker remain; the climada queue and ClimadaAdapter no longer exist.

ADR-021: Async/Sync Boundary in Celery Workers¶

Date: 2026-04-06 Status: Accepted
Phase: 0

Context¶

Celery tasks are synchronous (run in thread pool, no native event loop). Ingest and centroid assignment require async DB operations (asyncpg for async PostgreSQL, PostGIS spatial queries).

How to safely mix async code in sync Celery context?

Decision¶

Wrap async functions inside fresh coroutines. Execute with asyncio.run() inside the sync task body.

Critical rule: Create engine and session inside the coroutine, never at module import time.

# ❌ WRONG — session binds to import-time loop
engine = create_async_engine(url)  # at module level
AsyncSessionLocal = async_sessionmaker(engine)

@celery_app.task
def my_task():
    asyncio.run(_work())  # creates NEW loop, but session still bound to old loop
    # → RuntimeError: Future attached to different loop

# ✓ RIGHT — fresh engine per asyncio.run()
@celery_app.task
def my_task():
    async def _work():
        engine = create_async_engine(url)  # fresh engine inside coroutine
        session_factory = async_sessionmaker(engine)
        try:
            async with session_factory() as session:
                # ... do async work
        finally:
            await engine.dispose()

    result = asyncio.run(_work())

Alternatives Considered¶

Module-level session factories: Simpler but causes event loop binding errors
Fresh engine per task (chosen): Extra setup cost is negligible, avoids loop conflicts
Thread-based sync wrapper: Works but adds overhead; async is cleaner

Rationale¶

AsyncSessionLocal created at import time binds to the first event loop that uses it
Each asyncio.run() creates a brand-new event loop
Re-using a session across loop boundaries → "Future attached to a different loop" error
Disposing engine after use closes connection pool gracefully
Cost: ~10ms per task startup (negligible vs ~1-60s of actual work)

ADR-022: SQL Unnest for Bulk Spatial Inserts¶

Date: 2026-04-06 Status: Accepted
Phase: 0

Context¶

Ingesting large hazard datasets (630K centroids from JRC European Flood grid) requires fast bulk inserts.

SQLAlchemy ORM: session.add_all(centroid_records); session.flush() with GeoAlchemy2 geometry objects + RETURNING clause falls back to "batch not supported" — per-row INSERT statements.

630K rows → 630K round-trips → 30+ minutes.

Decision¶

Use raw SQL unnest() for batch array inserts. HazardRepository.create_centroids_bulk() sends 10K rows per batch in 63 SQL statements (vs 630K).

Pattern: Use CAST(:param AS type[]) syntax, not :: — asyncpg parser confuses :: after :name as a second parameter.

INSERT INTO hazard_centroids (id, dataset_id, array_index, geometry, lat, lon, created_at, updated_at)
SELECT
    unnest(CAST(:ids AS uuid[])),
    CAST(:dataset_id AS uuid),
    unnest(CAST(:indices AS integer[])),
    ST_SetSRID(ST_MakePoint(
        unnest(CAST(:lons AS double precision[])),
        unnest(CAST(:lats AS double precision[]))
    ), 4326),
    unnest(CAST(:lats AS double precision[])),
    unnest(CAST(:lons AS double precision[])),
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP

Result: 630K rows inserted in ~1 second (vs 30+ minutes).

Alternatives Considered¶

ORM add_all(): Semantic but hits per-row fallback for spatial data
SQL COPY: Fastest but requires file I/O; less flexible for computed values
SQL unnest with CAST (chosen): Balance of speed and maintainability; no file I/O

Rationale¶

PostgreSQL unnest() + SELECT is idiomatic for bulk inserts
GeoAlchemy2 geometry constructor in ORM triggers RETURNING optimization blocker
Raw SQL avoids the ORM → per-row fallback penalty
Array assembly in Python is trivial; SQL execution dominates runtime

ADR-023: Impact Function Sources for TC, WF, and WS¶

Date: 2026-04-16 Status: Accepted
Phase: 1

Context¶

Phase 1 requires seeding default impact functions for three hazard types: tropical cyclone (TC), wildfire (WF), and storm-europe (WS). Each seed migration (#50, #51, #52) needs a concrete published damage curve with known provenance and a license compatible with the project's Apache 2.0 target (ADR-009, ADR-016).

The primary risk is CLIMADA's GPL-3.0 codebase: CLIMADA ships reference implementations of all three curve families. If Climate-Lama copies CLIMADA code, it inherits GPL-3.0. The question is whether the underlying curve parameters can be independently implemented from their original publications without GPL obligation.

Legal basis: mathematical formulas, equations, and numerical calibration constants are not copyrightable under 17 U.S.C. § 102(b) (idea-expression dichotomy; see also NEC Corp. v. Intel Corp., 1990). A GPL-licensed program that implements a formula from a peer-reviewed paper does not make that formula GPL — only the specific source code expression is covered. Therefore, re-implementing a curve from its original peer-reviewed publication (not from CLIMADA source) carries no GPL obligation.

Decision¶

Tropical Cyclone — Emanuel (2011) sigmoid with Eberenz et al. (2021) regional calibration¶

Functional form (Emanuel, Wea. Climate Soc. 3(4), 2011; DOI 10.1175/WCAS-D-11-00007.1):

v_norm = max(v - v_thresh, 0) / (v_half - v_thresh)
MDD(v) = v_norm³ / (1 + v_norm³)
PAA(v) = 1  (full exposure above threshold)

Parameters: - v_thresh = 25.7 m/s — onset wind speed below which no damage occurs - v_half — wind speed at which MDD = 0.5; region-specific values from Eberenz et al. (2021)

Regional v_half values (Eberenz et al., NHESS 21:393–415, 2021; CC BY 4.0; DOI 10.5194/nhess-21-393-2021), TDR1.0 calibration:

Region	Code	v_half (m/s)
Caribbean / Mexico	NA1	66.3
USA / Canada	NA2	89.2
North Indian Ocean	NI	70.8
SE Asia	WP1	66.4
Philippines	WP2	188.4
China coast	WP3	112.8
NW Pacific	WP4	190.5
Oceania	OC	64.1
South Indian Ocean	SI	52.4
Global default	GLB	110.1

The global default (v_half = 110.1 m/s) is used when no regional match applies.

License status: Emanuel (2011) is published in a paywalled AMS journal; a freely-circulating author-archived copy exists (MIT DSpace: https://dspace.mit.edu/handle/1721.1/75143). The Eberenz et al. (2021) paper and its calibration data are CC BY 4.0 (Copernicus/NHESS). The formula and numerical constants are mathematical results — not copyrightable. Implementation must be written from scratch from the publications, not transcribed from CLIMADA source.

Wildfire — Lüthi et al. (2021) FIRMS brightness temperature sigmoid¶

Functional form (Lüthi et al., GMD 14:7175–7196, 2021; CC BY 4.0; DOI 10.5194/gmd-14-7175-2021):

i_norm = max(I - I_thresh, 0) / (I_half - I_thresh)
MDD(I) = i_norm³ / (1 + i_norm³)
PAA(I) = 1  (full exposure above threshold)

Parameters (intensity = FIRMS fire radiative brightness temperature, K): - I_thresh = 295 K — FIRMS detection threshold - I_half — resolution-dependent; calibrated globally against 84 EM-DAT damage records:

FIRMS resolution	I_half (K)
1 km	295.01
4 km	409.4
10 km	484.4

License status: Paper is CC BY 4.0 (Copernicus/GMD). Formula and calibration constants are freely usable. No proprietary data origin. This is the only globally calibrated, open-access parametric wildfire economic damage function in the literature; Hazus-MH has no wildfire module.

Storm Europe (Winter Storm) — Klawa & Ulbrich (2003) cubic excess-over-threshold¶

Functional form (Klawa & Ulbrich, NHESS 3:725–732, 2003; open access; DOI 10.5194/nhess-3-725-2003):

MDD(v) = max(v / v98 - 1, 0)³
PAA(v) = 1  (full exposure above local threshold)

where v is the maximum 3-second wind gust (m/s) and v98 is the location-specific climatological 98th percentile of daily maximum wind gust, derived from ERA5 or equivalent reanalysis. There are no hardcoded numerical coefficients — the threshold is site-specific.

License status: Paper is open access (Copernicus/EGU 2003 archive). The cubic functional form is a parametric mathematical relationship with no proprietary-data origin. v98 is computed from publicly available ERA5 reanalysis (Copernicus CDS, free for research and commercial use).

Why not Schwierz (2010)? The CLIMADA default (ImpfStormEurope.from_schwierz()) uses a 12-point tabulated PAA/MDD lookup table derived from proprietary reinsurance loss records and published in a Springer subscription journal. Tabulated values originating from proprietary industry data carry higher copyright risk than parametric formulas, and their provenance is opaque. The Welker (2021) rescaling (× 1.332518) does not resolve the underlying Schwierz provenance. Klawa–Ulbrich is the safer choice.

Alternatives Considered¶

CLIMADA default curves copied directly: Simplest path; GPL-3.0 contamination, rejected.
Schwierz (2010) for WS: CLIMADA default; tabulated values from proprietary reinsurance data published in a paywalled journal — higher copyright risk than a parametric formula. Rejected in favour of Klawa–Ulbrich.
Welker (2021) rescaling for WS: CC BY 4.0 scaling factor, but depends on the problematic Schwierz table underneath. Rejected.
Hazus-MH for WF: No wildfire module exists in Hazus as of version 7.0. Rejected (does not exist).
Synthetic placeholder curves for all three: Zero legal risk but scientifically indefensible for regulatory/research users. Rejected.
Eberenz (2021) RMSF calibration for TC: Also CC BY 4.0; slightly lower v_half values. Viable alternative. TDR1.0 chosen as the primary because it is cited more widely as the default in CLIMADA documentation and has better cross-regional validation in the paper.

Rationale¶

All three chosen sources have open-access publications (CC BY 4.0 or Copernicus open archive).
The underlying formulas are parametric mathematical results, not copyrightable under 17 U.S.C. § 102(b) and international equivalents (EU Database Directive Article 3 — only original selection/arrangement is protected, not facts or mathematical results).
Each implementation must be written from scratch using the paper as specification, never copying CLIMADA source expressions. This is consistent with ADR-016 (clean-room approach).
The Klawa–Ulbrich formula requires a v98 climatology raster; this is computed from ERA5 at seed time and stored alongside the impact function record, not hardcoded.

Follow-up¶

Refs #50 — seed TC impact function(s) via Alembic data migration
Refs #51 — seed WF impact function(s) via Alembic data migration
Refs #52 — seed WS impact function(s) via Alembic data migration
Resolves #105 — pre-ingest unit-compatibility invariant. Each BaseGeoTIFFIngestor subclass declares supported_units (defaulting to frozenset({intensity_unit})); the /v1/hazards/ingest endpoint cross-checks it against the units carried by the seeded curves in _BUILTIN_FUNCTIONS for that haz_type and refuses the request with IngestUnitMismatchError (HTTP 422 via the centralized handler in main.py) when they do not intersect. This converts the silent-zero failure modes documented in #96 — FWI rasters fed to the Kelvin-calibrated Lüthi curve, raw m/s gusts fed to a v/v98-normalised WS curve — into hard errors before any Celery job is queued. Adding a new hazard ingestor without a matching seeded curve (or vice versa) is now caught at the API boundary.

ADR-024: Worker Runtime Has No CLIMADA Dependency¶

Date: 2026-04-19 Status: Accepted
Phase: 2

Context¶

ADR-016 established that climate-lama-engine is a clean-room reimplementation; ADR-017 established that it ships as an independent pip package. Under the dual-engine migration plan (ADR-020, now deprecated), the compute worker still carried CLIMADA as a fallback runtime and jobs were routed to one of two queues based on an engine parameter.

Phase 0 closed out that migration (#19): the ClimadaAdapter stub was deleted, the climada Celery queue and compute_impact task were removed, and the [full-worker] optional dependency block was dropped from pyproject.toml. The deployed worker image now installs only climate-lama-engine>=0.1.0 plus the thin numpy/scipy stack it pulls in.

The licensing (ADR-016) and packaging (ADR-017) decisions are recorded, but the operational decision — that the worker runtime never imports CLIMADA and that there is no fallback path — has no explicit ADR. Without it, a future contributor reading the code in isolation might reasonably re-introduce a CLIMADA dependency to unblock a missing feature, unaware that the absence is load-bearing.

Decision¶

The Climate-Lama worker runtime depends only on climate-lama-engine and its lean numerical stack (numpy, scipy). No module in src/climate_lama/ imports from climada.*, and no deployment artefact installs CLIMADA.

src/climate_lama/worker/models/engine_adapter.py is the sole adapter boundary. Its module docstring already asserts: "This is the ONLY file allowed to import from climate_lama_engine.*" — all other modules must remain engine-free.
There is no fallback path to CLIMADA. Jobs that cannot be served by climate-lama-engine fail fast with a clear error; they are not silently routed to a heavier runtime.
Features not yet ported into climate-lama-engine are unavailable in Climate-Lama until the port lands upstream. Adding them by reaching back into CLIMADA is explicitly out of scope.
pyproject.toml's worker extra is the authoritative dependency list for the compute image. Re-adding a full-worker extra or an engine: "climada" code path requires a superseding ADR.

Alternatives Considered¶

Keep a CLIMADA fallback worker (ADR-020 status quo): Preserves feature parity during transitions, but drags a ~2 GB dependency graph into production, keeps a GPL-3.0 runtime on the critical path, and doubles CI cost. Rejected once the lean engine was validated.
Allow opportunistic import climada in individual modules for unported features: Fastest path to shipping a missing calculation, but quietly re-couples the platform to GPL-3.0 and breaks the single-adapter boundary. Rejected as it would erode ADR-016 in practice.
Wrapper around CLIMADA with no clean-room engine: Addressed and rejected under ADR-016; listed here for completeness.

Rationale¶

Footprint. The worker image shrinks from ~2 GB (CLIMADA + HDF5 + cartopy + CLIMADA's petl/numba stack) to ~200 MB (engine + numpy/scipy). More workers fit on the same hardware; cold-start and image-pull costs drop proportionally.
Licensing clarity. ADR-016 allows Apache 2.0 / MIT licensing of Climate-Lama only if no GPL-3.0 code is loaded into the running process. Removing CLIMADA from the worker makes that story hold in production, not just in theory.
Install reliability. CLIMADA's transitive dependencies (geopandas, cartopy, GDAL) are a frequent source of broken installs across platforms. A worker that depends only on numpy/scipy is reproducibly installable in Docker, Linux CI, and local dev without system-level GIS libraries.
CI cost. Worker test runs no longer provision the full CLIMADA environment. Job definitions that used to take several minutes to stand up a full-worker container run in seconds.
Single boundary, enforceable. With one adapter file, the rule "no CLIMADA imports outside the engine" collapses to "no CLIMADA imports anywhere" — cheaper to enforce by grep and by review.

Trade-offs¶

Any CLIMADA feature not yet reimplemented in climate-lama-engine is unavailable. The remediation path is to port it into the engine (guided by CLIMATE_LAMA_ENGINE_SPEC.md and the clean-room approach in ADR-016), not to shim it into the worker.
There is no runtime A/B comparison against CLIMADA. Validation against CLIMADA reference outputs, when needed, happens offline in the engine repo's test suite, not in the deployed worker.

Implications¶

The worker/ package must stay free of climada imports. Reviewers and CI-time grep treat any reintroduction as a regression.
worker/models/engine_adapter.py is the only place where engine objects are constructed or consumed; new compute tasks route through it rather than importing climate_lama_engine directly.
Documentation that still describes Climate-Lama as "built on CLIMADA" is historical framing; the runtime reality is captured here.

Related¶

Supersedes the transitional runtime shape defined by ADR-020 (deprecated).
Reinforces ADR-016 (clean-room, permissive license) by making the no-CLIMADA property a runtime invariant, not just a source-level one.
Depends on ADR-017 (engine as independent package): the worker can only drop CLIMADA because climate-lama-engine is separately installable.

ADR-025: MapLibre GL Commitment and Library Whitelist¶

Date: 2026-04-22 Status: Accepted Phase: 3

Context¶

Round 3 planning committed to MapLibre GL as the map rendering library from the start rather than deferring a Leaflet migration. The guardrail is that only well-maintained, actively adopted libraries enter the dependency tree. This decision freezes the stack so Phase 4 can execute without re-litigating the choice.

MapLibre GL JS is a community-maintained fork of Mapbox GL JS v1, released under the BSD-3-Clause license. It has active governance, frequent releases, and a large ecosystem. It supports native vector tile (MVT) rendering — a prerequisite for the zoom-aware, data-dense visualisations planned in Phase 4.

Decision¶

Adopt MapLibre GL JS as the sole map rendering library in climate-lama-ui. Leaflet is not used at all; no migration path from Leaflet is required. The permitted dependency set is:

maplibre-gl (BSD-3-Clause) — core renderer; native MVT + WebGL raster support
@maplibre/maplibre-gl-inspect (BSD-3-Clause) — dev-mode layer inspector
@mapbox/mapbox-gl-draw (ISC) — polygon/point draw tools; tested against MapLibre and used without a Mapbox token
Base-tile providers (evaluated at integration time): MapTiler Cloud, Stadia Maps, self-hosted OpenMapTiles. Provider is configured via env var; no provider is hardcoded.

No other map-adjacent libraries (overlapping renderers, tile loaders, or projection utilities) are added without an explicit ADR update or a follow-up design decision.

Alternatives Considered¶

Stay on Leaflet: Mature, but raster-only tile model cannot efficiently render large MVT datasets; no WebGL support.
Leaflet + VectorGrid plugin: Adds MVT support, but VectorGrid is not well-maintained and the hybrid is harder to style and extend than a native WebGL renderer.
Mapbox GL JS v2: Functionally equivalent, but the Mapbox GL JS v2 license requires a Mapbox token and prohibits forking — incompatible with the open-source, self-hostable goal.
deck.gl: Powerful for data visualization layers, but not a complete map library; would require a co-renderer for base tiles and adds significant bundle weight for the feature set needed in Phase 4.

Rationale¶

Native MVT rendering is a hard requirement for Phase 4 zoom-aware aggregation and impact-layer overlays (ADRs 4.2–4.5 in the Phase 4 plan).
BSD-3-Clause and ISC licenses are compatible with the platform's open-source stance.
MapLibre's active governance and large fork community reduce abandonment risk relative to older Mapbox forks.
Pinning the whitelist now prevents ad-hoc additions during Phase 4 implementation, which is the pattern most likely to introduce license or maintenance risk.

Follow-up¶

Phase 4 map-stack work (§4.1 MapLibre migration of existing components, §4.2 layer descriptor schema, §4.3 MVT tile renderer, §4.4 zoom aggregation, §4.9 map layer toggle) — see docs/plan/phase-4-map-stack.md.
Base-tile provider selection (MapTiler vs Stadia vs self-host) to be finalised in Phase 4 §4.1 once the rendering pipeline is in place.

ADR-026: Dependency Policy¶

Date: 2026-04-22 Status: Accepted Phase: 3

Context¶

The engine deliberately stays lean (~20 KB installed footprint vs CLIMADA's ~500 MB). The backbone and UI had no equivalent written policy, leaving every new dependency to be re-argued at the PR level and creating inconsistent gatekeeping. Round 3 gap-13 flagged this; action item A7 resolved "yes, write the ADR."

Decision¶

All runtime dependencies added to the backbone, engine, or UI must satisfy the following criteria before merging:

Rationale: The PR body must include a one-paragraph rationale covering why the dependency is needed, why an existing dep or the standard library cannot serve the same purpose, and any known risks (breaking changes, abandonment, license shifts).
Health criteria (all three must pass):
Activity: last commit or release within 6 months of the PR date
Test coverage: the library ships its own tests (CI badge or visible test suite)
Adopters: evidence of production use (weekly downloads, corporate adopters, or GitHub stars ≥ 500)
Stdlib / already-present preference: If a task can be accomplished with the Python / Node standard library, or with a transitive dep already in the lock file, the new dep is not added.
License allowlist: Only the following SPDX identifiers are permitted without an additional approval step:
Apache-2.0
MIT
BSD-2-Clause
BSD-3-Clause

ISC, MPL-2.0, and similar permissive licenses require a one-line note in the PR body but are otherwise allowed. Copyleft licenses (GPL, AGPL, LGPL) require explicit approval from a maintainer before merging.

Dev-only dependencies (ruff, mypy, pytest-*, coverage, etc.) bypass the full rationale and health-criteria requirements but must still satisfy the license allowlist. A short note ("dev dep, MIT") in the PR is sufficient.

Alternatives Considered¶

No policy — case-by-case review: Status quo. Inconsistent; depends on reviewer knowledge; creates ADR debt.
SBOM-based automated gate: Fully automated dependency scanning. Valuable long-term, but overkill for the current team size and release cadence.
Single shared allowlist file per repo: A deps-allowlist.txt checked in each repo. Adds friction without adding signal; the PR rationale paragraph captures the same intent more readably.

Rationale¶

Written policy removes per-PR re-litigation; reviewers can point to this ADR instead of arguing from first principles.
Health criteria are lightweight enough that any widely-used library clears them automatically; they catch abandonware and hobby projects before they land.
License allowlist keeps the platform open-source-safe without requiring legal review on every PR.
Distinguishing runtime from dev deps reflects real risk: dev deps don't ship to end users or create transitive license obligations in deployed artifacts.

Follow-up¶

Add PR-template checkbox "New runtime dep? Include one-paragraph rationale (ADR-026)" in the backbone, engine, and UI repos. Backbone template updated in this PR; engine and UI templates are tracked as a follow-up cross-repo task (see issue #196).

ADR-027: Adapter-by-Design Architectural Commitment¶

Date: 2026-04-22 Status: Accepted Phase: 3

Context¶

Vision issue #32 and the Round 4 architectural review both identified that Climate-Lama's extensibility story was implicit rather than explicit: the compute engine had a defined interface (ModelInterface), but ingest formats, report renderers, dataset sources, and auth providers had no equivalent contract. Without a written policy, extension points accumulate ad-hoc, adapters land in the wrong repos, and security boundaries drift.

This ADR names the five extension points, specifies their current implementation status, establishes the packaging rule (separate climate-lama-adapter-* repos, optional extras on the backbone), and defines the in-process vs. remote trust boundary.

Decision¶

Extension points¶

Extension point	Current status
Compute engine	`ModelInterface` ABC exists in `worker/models/`; `climate-lama-engine` is the sole implementation (ADR-024).
Ingest formats	No ABC yet; concrete ingestors share a common pattern but lack a formal interface. Needs ABC.
Report template engines	No abstraction; report generation is not yet implemented. Needs abstraction before the first implementation lands.
Dataset sources	No ABC; dataset connectors (JRC, future ERA5, OpenStreetMap, etc.) are ad-hoc. Needs ABC.
Auth providers	Already abstracted via FastAPI dependency injection; swapping providers requires no core changes.

Each extension point will receive its own sub-ADR when its ABC or interface is implemented. The sub-ADR records the interface contract, the versioning policy, and any known constraints on implementors.

Packaging rule¶

First-party adapter implementations live in their own repositories:

climate-lama-adapter-climada        # reference CLIMADA adapter (offline validation use)
climate-lama-adapter-gem            # Global Earthquake Model adapter
climate-lama-adapter-...            # future third-party engines or ingestors

The backbone declares optional extras so that adapters are opt-in at install time:

[project.optional-dependencies]
climada = ["climate-lama-adapter-climada>=1.0"]
gem    = ["climate-lama-adapter-gem>=1.0"]

Adapters are never listed as core dependencies. The default install (pip install climate-lama) does not pull any adapter.

Security boundary¶

Adapters fall into two trust tiers:

In-process Python adapters (pip-installed, loaded via import): trusted. They run inside the worker process, have access to the same DB credentials and MinIO keys, and must satisfy the dependency policy (ADR-026) and the clean-room / license rules (ADR-016) before they can be added to the optional extras list.
Remote webhook adapters (HTTP callback, #24 Level 3b): Tier 4 — untrusted by default. Results returned over HTTP must pass schema validation and numeric range checks before being written to the database. The worker never forwards raw webhook payloads to downstream consumers.

Rejected pattern: dynamic Python package reference at runtime (#24 Level 3a — resolving an adapter by a user-supplied package name and installing it on demand) is explicitly out of scope. It introduces arbitrary code execution risk and bypasses the license and health checks required by ADR-026.

Alternatives Considered¶

Implicit extensibility (status quo): No contracts, adapters land wherever convenient. Results in divergent patterns, no security guarantee, and re-litigation of packaging at every PR. Rejected.
Monorepo with all adapters included: Simpler discovery, but couples adapter release cycles to the backbone and bloats the default install. Rejected — the whole point of ADR-024 was to keep the default image lean.
Plugin registry (dynamic discovery via entry-points): Powerful but adds a runtime dependency-resolution layer and makes the install surface unpredictable. Deferred — entry-point discovery can be added in Phase 4 once multiple adapters exist and the interface contracts are stable.

Rationale¶

Named extension points prevent future contributors from introducing tight coupling where an interface is expected, without realising the contract exists.
Separate adapter repos with optional extras enforce the lean-by-default principle and keep adapter-specific dependencies out of the core image.
Explicit trust tiers close the gap between "extensible" and "secure": in-process adapters are audited at merge time; remote adapters are never trusted at runtime.
Writing this ADR now (Phase 3) — before most extension points have implementations — ensures the contracts are designed, not reverse-engineered.

Follow-up¶

Each extension point gets its own sub-ADR when its ABC is implemented:
Ingest format ABC (linked to Phase 3 ingest work)
Report template engine abstraction (Phase 3 / Phase 4 reporting)
Dataset source ABC (Phase 3 catalog work)
Compute engine sub-ADR is already covered by ADR-024; no new sub-ADR needed unless ModelInterface itself changes.
Add climate-lama-adapter-* repo scaffolding when the first non-engine adapter is needed.
PR-template checkbox for adapter repos: "Does this adapter satisfy ADR-026 (deps) and ADR-016 (clean-room / license)?"

ADR-028: Error Taxonomy¶

Date: 2026-04-22 Status: Accepted Phase: 3

Context¶

Climate-Lama's backbone raised internal exceptions that were converted to HTTP responses using ad-hoc string codes ("NOT_FOUND", "VALIDATION_ERROR", "INGEST_UNIT_MISMATCH"). These codes were not catalogued, had no stability contract, and lacked a severity dimension. As a result:

The UI surfaced generic error messages with no path to i18n (Gap-7).
Future SDK clients would have no stable error contract to code against (issue #I).
Operational alerts could not filter by severity (error vs. warning vs. info).

A machine-readable, catalogued error-code system unblocks i18n of errors (#15), SDK quality, and consistent UI presentation.

Decision¶

Error Envelope¶

Every error response from the backbone carries:

{
  "error": {
    "code":       "E_HAZARD_INTENSITY_UNIT_MISMATCH",
    "severity":   "error",
    "message_en": "Hazard intensity unit is incompatible with the available impact functions.",
    "details":    { "haz_type": "TC", "ingestor_units": ["m/s"], "seeded_units": ["km/h"] }
  },
  "meta": {
    "request_id": "...",
    "timestamp":  "..."
  }
}

Fields:

Field	Type	Description
`code`	string	Machine-readable identifier. Never changes once published.
`severity`	`"error"` \| `"warning"` \| `"info"`	Signals client-side handling urgency.
`message_en`	string	Default English text. UI clients translate by `code`.
`details`	object (optional)	Structured context specific to the error type.

Code naming convention¶

Errors — E_ prefix, SCREAMING_SNAKE_CASE. Signal a request that cannot be fulfilled.
Warnings — W_ prefix. Signal a degraded-but-successful outcome clients should surface.
Info — I_ prefix. Signal non-error state transitions (e.g. job accepted).

Code stability contract¶

Codes are immutable identifiers. Once published in a tagged release:

A code is never renamed or removed.
Deprecation is done by alias: the old code value remains valid and maps to the new one in the registry.
New codes may be added at any time; clients must handle unknown codes gracefully.

Registry location¶

src/climate_lama/core/errors.py — single source of truth containing:

Severity enum (error / warning / info).
Code enum — all published codes with E_ / W_ / I_ prefixes.
CATALOGUE — maps each Code to (Severity, default_message_en).
severity(code) / message_en(code) — lookup helpers used by main.py handlers.

HTTP exception handlers in main.py import from this registry so that message strings and severity are never duplicated.

Initial catalogue (Phase 3)¶

Code	Severity	Trigger
`E_NOT_FOUND`	error	Requested resource does not exist (`NotFoundError`).
`E_VALIDATION`	error	Input data failed semantic validation (`ValidationError`).
`E_INTERNAL`	error	Unhandled exception caught by the catch-all handler.
`E_RATE_LIMITED`	error	Client exceeded the request rate limit.
`E_EXPOSURE_INVALID_CSV`	error	Exposure CSV could not be parsed.
`E_EXPOSURE_EMPTY`	error	Exposure CSV contains no data rows.
`E_EXPOSURE_OUT_OF_BBOX`	error	Coordinate outside valid geographic bounds.
`E_EXPOSURE_ENCODING`	error	Exposure file is not UTF-8 encoded.
`E_HAZARD_INTENSITY_UNIT_MISMATCH`	error	Hazard unit incompatible with impact functions (`IngestUnitMismatchError`).
`E_HAZARD_NO_INGESTOR`	error	No ingestor registered for the given hazard type.
`E_HAZARD_DATASET_NO_EVENTS`	error	Hazard dataset contains no events.
`E_HAZARD_DATASET_NO_EVENT_FREQUENCIES`	error	Hazard dataset's events carry no annual frequency, so it has no EAI path (ADR-069).
`E_IMPACT_FUNCTION_NOT_FOUND`	error	Impact function not found.
`E_MEASURE_NOT_FOUND`	error	One or more requested measures not found.
`E_JOB_FAILED`	error	Async compute job failed during execution.
`E_ENGINE_UNAVAILABLE`	error	Compute engine is unavailable or returned an unexpected error.
`E_PORTFOLIO_EMPTY`	error	Portfolio has no exposures.
`E_STORAGE`	error	Object storage (MinIO) operation failed.
`E_DATABASE`	error	Unexpected database error.
`E_AUTH_INVALID_TOKEN`	error	Authentication token is invalid or expired.
`E_AUTH_INSUFFICIENT_SCOPE`	error	Principal lacks required permission for the operation.
`E_AUTH_OIDC`	error	OIDC authentication flow failed.
`W_RETURN_PERIOD_EXTRAPOLATED`	warning	Return period is outside observed range; result extrapolated.
`I_JOB_QUEUED`	info	Job accepted and queued for processing.

Alternatives Considered¶

HTTP status codes only (status quo): No machine-readable error body. Simple, but prevents i18n, SDK bindings, and fine-grained error handling. Rejected.
RFC 9457 Problem Details: Standardised type URI + title + detail structure. Avoids the E_-prefix convention but ties us to a URI namespace and doesn't cleanly support severity. Deferred — the envelope can be mapped to RFC 9457 in a future ADR if interoperability becomes a requirement.
Per-resource error namespacing (e.g. EXPOSURE_INVALID_CSV without E_): Loses the severity signal that the prefix provides at a glance. Rejected.

Rationale¶

Machine-readable codes unblock i18n without requiring the backbone to ship translation strings or be locale-aware.
A single CATALOGUE prevents code–message drift: the registry is the only place where a code's default message lives.
The E_ / W_ / I_ prefix convention makes severity visible in log output and OpenAPI docs without reading the envelope body.
Freezing the shape and an initial catalogue now (Phase 3) — before the UI i18n work lands — gives the UI team a stable target.

Follow-up¶

Phase 5 — retrofit all existing raise HTTPException(...) calls in api/v1/ to use Code enum values from the registry (Gap-7 UI presentation, issue #15). Auth endpoints currently use ad-hoc strings; each will be mapped to a registered code at that time.
Add code to the OpenAPI response schema once the Phase 5 retrofit is complete.
Consider exposing a GET /v1/error-codes endpoint (machine-readable catalogue) once the full taxonomy is stable.

ADR-029: DO Spaces Bucket Layout and Dataset Versioning¶

Date: 2026-04-22 Status: Accepted Phase: 3

Context¶

Phase 3 backend work (R4 B2) settled the storage bucket layout for DigitalOcean Spaces (or any S3-compatible object store). Without a written layout, ingest pipelines, the tile-serving layer (Phase 4), and the catalog + admin UI (Phase 6) would each make independent assumptions about path structure, making cross-feature joins and reproducibility audits impossible.

The layout must satisfy three consumers simultaneously:

Ingest pipeline — writes raw source files and converted outputs to deterministic paths.
Tile server — reads processed rasters and vector tiles at zoom-dependent paths; must be able to determine regenerability without reading every file.
Catalog — joins bucket objects to database records via the dataset UUID; manifest.json is the bridge.

This ADR freezes the layout so Phase 4 and Phase 6 cannot drift independently. No object storage infrastructure is provisioned or changed by this ADR — it defines naming only.

Decision¶

Bucket layout¶

{bucket}/
├── raw/
│   └── {source}/               # e.g. jrc, era5, osm, user-upload
│       └── {data-type}/        # e.g. flood, wind, exposure
│           └── {version}/      # upstream version label, pinned literally (see invariants)
│               └── {dataset-id}/   # catalog UUID — the join key
│                   ├── manifest.json   ← MANDATORY at every leaf
│                   └── <source files>  # original download, untransformed
│
├── processed/
│   └── {org_id}/               # org-scoped; public datasets use org_id "public"
│       └── {dataset-id}/
│           ├── manifest.json   ← MANDATORY at every leaf
│           └── <derived files> # reprojected, clipped, tiled GeoTIFF, GeoPackage, etc.
│
├── tiles/
│   └── {dataset-id}/
│       └── {z}/
│           └── {x}/
│               └── {y}.pbf     # MVT tiles; regenerable from processed/ (see invariants)
│
└── reports/
    └── {org_id}/
        └── {job-id}/
            ├── manifest.json   ← MANDATORY at every leaf
            └── <output files>  # PDF, GeoJSON, CSV — job artefacts

Invariants¶

{dataset-id} is the catalog UUID join key. Every leaf directory that holds data includes the catalog datasets.id UUID as a path component. Object paths are therefore derivable from the catalog record and vice versa; no out-of-band path registry is needed.
manifest.json is mandatory at every leaf directory. A leaf is any directory that holds data files (raw source, processed output, or report artefact). A directory without a manifest.json is considered incomplete and must not be read by the ingest pipeline, tile server, or catalog indexer.
{version} pins the upstream release label literally. For JRC flood data, this is the data-release identifier (e.g. v2_2020); for ERA5 it is the product version string. The version component is set at ingest time from the source metadata and never normalised or transformed. This makes the path a stable reference that survives future ingest re-runs with newer source versions — old versions stay at their original paths.
Tiles are regenerable. The tiles/ prefix holds derived read-through cache artefacts. All tile content can be reconstructed from the corresponding processed/ files. The tile server may delete and regenerate any tiles/{dataset-id}/ subtree without data loss.
Org data is scoped under processed/{org_id}/ and reports/{org_id}/. Public reference datasets (JRC, ERA5, OSM) use the reserved org_id value "public". Org-private data (user uploads, custom exposures) use the organisation's UUID from the organisations table. Cross-org reads require an explicit permission grant and are never assumed.

`manifest.json` schema¶

Every leaf manifest.json conforms to the following JSON Schema (draft-07):

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://climate-lama.io/schemas/manifest.json",
  "title": "Climate-Lama Bucket Manifest",
  "type": "object",
  "required": [
    "source",
    "source_url",
    "fetched_at",
    "checksum_sha256",
    "license",
    "crs",
    "units",
    "dataset_id",
    "version"
  ],
  "properties": {
    "source":          { "type": "string", "description": "Short source identifier, e.g. 'jrc', 'era5', 'user-upload'." },
    "source_url":      { "type": "string", "format": "uri", "description": "Canonical URL of the upstream dataset or download endpoint." },
    "fetched_at":      { "type": "string", "format": "date-time", "description": "ISO-8601 UTC timestamp of when the data was downloaded." },
    "checksum_sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$", "description": "SHA-256 hex digest of the primary data file(s). For multi-file leaves, this is the digest of the concatenated sorted filenames + their individual digests." },
    "license":         { "type": "string", "description": "SPDX identifier or free-text license name, e.g. 'CC-BY-4.0', 'Apache-2.0'." },
    "crs":             { "type": "string", "description": "Coordinate reference system as an EPSG string or WKT, e.g. 'EPSG:4326'." },
    "units":           { "type": "string", "description": "Physical units of the primary intensity or value field, e.g. 'm', 'K', 'm/s'." },
    "dataset_id":      { "type": "string", "format": "uuid", "description": "Catalog UUID — foreign key to datasets.id in the backbone database." },
    "version":         { "type": "string", "description": "Upstream version label, pinned literally from source metadata (invariant 3)." }
  },
  "additionalProperties": false
}

The schema is the normative specification. Future fields must be added as additionalProperties with backward-compatible defaults and an ADR update or a follow-up design decision before they are made required.

One sanctioned exception exists: leaves under reports/ carry a report-shaped manifest per ADR-039 — a rendered document has no CRS, units or upstream fetch to declare. The manifest-written-last invariant is unchanged there.

Alternatives Considered¶

Flat bucket with UUID-only paths ({bucket}/{dataset-id}/file): Simple, but opaque to humans and tooling; no source, data-type, or version signal without reading the manifest. Rejected — the human-readable path hierarchy has operational value during debugging and migrations.
Separate buckets per purpose (one bucket for raw, one for processed, one for tiles): Clean isolation, but multiplies IAM policies and bucket-level configuration. More complex for self-hosted deployments where the operator may only have one bucket. Rejected in favour of prefix namespacing.
No manifest.json; all metadata in the catalog DB only: Simpler write path, but breaks the storage layer's ability to operate as an independent audit trail. If the database is lost or migrated, manifests allow reconstruction. Rejected.
{dataset-id} as directory root (path starts with UUID): Eliminates source/type/version path hierarchy. Fast for UUID lookups but requires the catalog to reconstruct context for everything else. Rejected — the hierarchy before the UUID makes paths self-describing.

Rationale¶

A fixed prefix hierarchy (raw/, processed/, tiles/, reports/) makes IAM policies and lifecycle rules (e.g. "delete tiles older than 30 days") expressible as simple bucket-prefix rules without enumerating dataset UUIDs.
The {source}/{data-type}/{version} prefix sequence matches the natural ingest workflow: a new ingest job knows its source and data type before it has a UUID; the UUID is assigned after the catalog entry is created and is appended as the final path component.
Mandatory manifest.json at every leaf closes the gap between object storage and the database: any leaf can be validated or re-indexed from storage alone, enabling disaster recovery and offline audits.
Pinning {version} literally from upstream preserves reproducibility: rerunning an ingest job for a different source release writes to a different path and does not overwrite earlier data.

Follow-up¶

Phase 4 — tile-serving (#27b): The tiles/ prefix layout defined here is the path contract the tile server reads. The tile server must treat tiles/{dataset-id}/ as a regenerable cache and must read and validate manifest.json before serving processed data.
Phase 6 — catalog + admin UI (#19): The catalog indexer must enforce the mandatory manifest.json invariant: any leaf directory without a manifest must be flagged as incomplete and excluded from catalog queries. The admin UI should surface manifest metadata (source, license, fetched_at) alongside catalog records.
Phase 6 — ingest pipeline: All ingest workers must write manifest.json as the final step of a successful ingest run (write-last = atomic commit signal). A partial write that does not reach the manifest step leaves the leaf directory in an incomplete state, which is detectable by the catalog indexer and recoverable by re-running the ingest job.

ADR-030: Observability Stack — Prometheus + Grafana + Loki¶

Date: 2026-04-23 Status: Accepted Phase: 3

Context¶

Phase 2 shipped structured logs via structlog. Phase 3 issue #203 adds the metrics and dashboard surface: a Prometheus-compatible /metrics endpoint, Celery task instrumentation, a committed Grafana dashboard, and an opt-in log-aggregation path. We need to pin the stack now so managed-hosting (Tier 4) and self-hosters do not drift onto incompatible backends.

Two secondary questions needed answers during /build:

Whether /metrics should be authenticated.
Where the OpenTelemetry tracing dependency should live (required vs. optional extra).

Decision¶

Metrics backend: Prometheus. Prometheus is the de facto standard for the scrape-based pull model we want; it is natively compatible with prometheus-fastapi-instrumentator (already a first-class dep) and with prometheus_client counters we emit from the worker.
Dashboards: Grafana, with dashboards provisioned from JSON committed at infra/monitoring/grafana/provisioning/dashboards/ — no hand-clicking in the UI to reproduce a deployment.
Logs: Loki + Promtail, running in the same compose profile. Promtail tails Docker containers that carry the logging=promtail label (applied to api and worker) and forwards to Loki; Grafana queries Loki via a provisioned datasource.
Deployment: opt-in Compose profile monitoring on the main docker-compose.yml. Activated with docker compose --profile monitoring up. Base deployments do not pay the container footprint unless they opt in.
/metrics is unauthenticated on the API pod. Self-hosters are expected to restrict access at the network layer (e.g. via a reverse proxy or firewall rule); managed hosting adds its own bearer-token-protected ingress in front of the API and does not expose the raw port. Prometheus scrape endpoints conventionally carry no application auth, and every extra dependency on the hot path is a reliability risk for the signal we most need during incidents.
OpenTelemetry is an optional extra, not a base dependency. The scaffold lives in src/climate_lama/observability.py and is a no-op unless OTEL_ENABLED=true AND the otel extra (pip install -e .[otel]) is installed. This keeps the default install small and predictable while still committing to OTLP as the tracing surface if/when operators opt in.
Exporters: Postgres connection metrics come from prometheuscommunity/postgres-exporter; Celery queue depth comes from oliver006/redis_exporter (REDIS_EXPORTER_CHECK_SINGLE_KEYS=0=compute). Both run only under the monitoring profile.

Alternatives Considered¶

Cloud-vendor APM (Datadog / New Relic / Honeycomb): Zero ops burden, best-in-class UX. Rejected — it violates the open-source, self-hostable, offline-capable positioning. The bundled stack must work for an air-gapped NGO.
Grafana Cloud as the default backend with local stack optional: Similar tradeoff to the above; also couples the default to an account signup. Rejected for the same reason.
OpenTelemetry as the sole metrics pipeline (OTel Collector → Prometheus): More "future-proof" but forces every self-hoster to run the collector just to see a Grafana dashboard. Rejected for the default path; the OTel scaffold exists so operators who already run a collector can opt in without re-plumbing.
ELK (Elasticsearch / Logstash / Kibana) for logs: Much heavier footprint than Loki and duplicates Grafana's role. Rejected — Loki's cost model (index labels, not content) suits append-heavy JSON logs from structlog, and Grafana already renders them.
Auth-gated /metrics with a shared bearer token: Adds a shared secret that must be synced to Prometheus's scrape config; no meaningful security benefit for a self-hoster who controls the network, and breaks Kubernetes service-monitor ergonomics. Rejected.

Rationale¶

The Prometheus + Grafana + Loki triad is the most widely documented self-hostable stack in 2026; sysadmins encountering this repo will already know it, shortening time-to-debug.
Keeping the monitoring services in the main docker-compose.yml (gated on a profile) avoids fragmenting the deployment surface — one docker compose invocation boots everything needed, including exporters wired to the same Docker network.
Making OTel an extra dependency rather than a base one preserves the small-container commitment (see ADR-002) for workers that do not need distributed tracing. The ~40 MB of OTel wheels only land when the operator asks for them.
Leaving /metrics unauthenticated matches the norms of the tools the rest of the ecosystem uses (kube-prometheus, Docker Swarm, Nomad) and pushes the authz concern to the one layer where it composes well with everything else: the ingress.

Follow-up¶

Phase 3 / Tier 4 managed hosting: The managed gateway must terminate the /metrics path with a bearer-token check and forward to the API pod. Tracked alongside the managed-hosting work item; not a backbone concern.
Alerting: infra/monitoring/prometheus_alerts.yml currently holds a single failure-rate rule. Expand as SLOs are formalised (ties into the incident-response plan).
Queue-depth metric parity: redis_key_size approximates Celery queue length but does not see in-flight reservations. If we ever adopt Celery's broker_heartbeat or a real broker (RabbitMQ), revisit — a native queue-length gauge would be more accurate than the Redis-list view.

ADR-031: GADM v4.1 as the Administrative-Boundary Source¶

Date: 2026-04-24 Status: Accepted Phase: 4

Context¶

Zoom-aware aggregation (issue #226) needs authoritative admin-0 / 1 / 2 polygons so damage and exposure values can be summarised at country, region, or district level. The source has to be global, consistent across countries, stable enough to cite in a report, and redistributable under terms compatible with the platform's self-hostable, open-source positioning.

Decision¶

Use GADM v4.1 as the single source of administrative boundaries. Ship a one-time loader (scripts/load_admin_boundaries.py) that populates the admin_boundaries table (migration 0031) from GADM's global levels archive. Admin-0 and admin-1 load by default; admin-2 is gated on LOAD_ADMIN_2=true because the dump is large (~300k rows). The loader upserts on (source, version, iso3, admin_level, name, parent_id) so re-running the same version is a no-op.

GADM's editorial choices on disputed territories (Crimea under Ukraine, Kashmir split along the Line of Control, Western Sahara as its own admin-0) are passed through unmodified. The UI surfaces GADM's attribution block so readers can reach the canonical source for any position they wish to challenge.

Alternatives Considered¶

Natural Earth. Simpler license (public domain) but admin-1 coverage is sparse and inconsistent between countries, and there is no first-class admin-2 layer. Rejected — the main reason we want this data is consistent sub-country rollups.
OpenStreetMap admin boundaries. Higher-resolution and community-maintained, but geometry quality varies wildly per country and extracting a coherent level-0/1/2 hierarchy requires either Overpass scripting or a derived product like OSM-Boundaries. Rejected for the v1 loader — the ingest surface is too complex for a one-time script, and the license (ODbL) has share-alike obligations we would rather not impose on every tenant.
Per-country national statistical offices. Best geometry per country but no uniform schema; stitching 200 sources is a continuous maintenance burden. Rejected.
GeoBoundaries (CGAZ). Reasonable alternative with a permissive license and explicit admin-level harmonisation. Close second. Rejected for v1 because the hierarchical parent keys are less direct than GADM's GID_N chain and because GADM's research citation is more widely recognised in the climate-risk domain where most of our target users already work.

Rationale¶

One file, one schema, one parent-key chain — the loader stays small and the aggregation surface in #226 can assume a tidy hierarchy instead of reconstructing it.
GADM's research/non-commercial clause fits the self-hosted + NGO + academic user profile we are optimising for. The managed-hosting deployment is the only path that needs a commercial agreement, and that conversation is scoped to one deployment.
The version column and the upsert key let us stage a new GADM release without deleting the old rows — tenants can pin queries to version='4.1' while a newer version is validated.

Follow-up¶

Issue #226 (zoom-aware aggregation) is the immediate consumer; it should join the results surface against admin_boundaries by ST_Intersects / ST_Within.
Managed-hosting licensing. Before Tier 4 ships commercially, obtain GADM's written permission or swap the source. Track as part of the managed-hosting work item.
Disputed-borders editorial policy. If a deployment needs a non-GADM rendering of a contested region, do it in a tenant-specific view or a UI layer — do not mutate the canonical rows.

ADR-032: martin for MVT Tile Serving¶

Date: 2026-04-24 Status: Superseded — retired 2026-08-02 (issue #562). The path never gained a consumer: LayerSpec never wired an MVT URL into a vector layer, so the martin service, the mvt_hazard_centroids function, and the /v1/tiles/mvt/... proxy route were all removed. See Tile Serving for the retirement notes. The /v1/results/{id}/geojson path (option 1 below) remains the point-data route. Phase: 4

Context¶

Phase 4 vision #5 targets 30+ FPS map interaction and <500 ms first-tile latency on datasets up to 100 M points. The existing /v1/results/{id}/geojson endpoint returns a full GeoJSON payload per request; at 10 M points this is several hundred MB per viewport pan, making smooth interaction impossible.

Two paths were evaluated:

Stream GeoJSON from the API with server-side viewport filtering (implemented in #240, useful up to ~50 k visible points).
Serve Mapbox Vector Tiles (MVT) directly from PostGIS so the browser only downloads the pixels it needs.

Path 2 is required to hit the 100 M-point target.

Decision¶

Add martin as a sidecar container alongside the backbone API. martin auto-discovers PostgreSQL functions that return bytea and accept (z integer, x integer, y integer), exposing each as a tile endpoint with no additional config.

A single SQL function mvt_hazard_centroids (migration 0033) handles all zoom levels by applying a zoom-dependent ST_SnapToGrid cell size, so low-zoom tiles show density clusters and high-zoom tiles approach raw point resolution — all in one function body without pre-aggregated materialized views.

Alternatives Considered¶

Custom FastAPI MVT endpoint. Requires writing tile encoding, caching, and CORS handling ourselves. More control, but reinvents what martin already provides. Rejected — kept as the descope fallback if martin proves flaky.
pg_tileserv. Same auto-discovery model, less actively maintained (last release 2022), lacks native query_params JSON forwarding for per-org scoping. Rejected.
Pre-aggregated materialized views (one per zoom band). Fastest possible tiles (~5 ms). Adds a refresh pipeline and schema migrations whenever a new hazard dataset is ingested. Deferred to Phase 6 if martin latency is insufficient at scale; see tile-cache issue #4.8.
Client-side clustering (Supercluster). Zero server change; works well up to ~500 k points before the initial GeoJSON download becomes the bottleneck. Already in the UI for the GeoJSON path — insufficient for 10 M+ points.

Rationale¶

martin is a single ~30 MB binary container with no runtime config file; adding it is a five-line change to docker-compose.
The auto-discovery model keeps the SQL function as the single source of truth for tile aggregation logic — no separate tile-server schema to maintain.
query_params JSON forwarding lets the UI scope tiles per-org with a standard URL query string (?org_id=<uuid>), keeping the tile endpoint stateless.
STABLE PARALLEL SAFE on the SQL function allows Postgres to parallelise the tile query when the planner chooses.
Descope path is low-risk: if martin proves unreliable, the on-the-fly FastAPI endpoint (#4.4) remains available and the SQL function itself is still useful for debugging.

Follow-up¶

Issue #230 (this issue) — backbone delivery: martin container + migration.
climate-lama-ui #4.5-ui — MapLibre source wiring and zoom-handler simplification.
Issue #4.8 (tile cache) — HTTP caching layer in front of martin; depends on this ADR landing first.
Issue #4.11 (perf tests) — automated benchmark CI for the 500 ms target; scripts/bench_tiles.py is the local runner until that lands.

ADR-033: UI Layout Commitment — Map-as-Canvas¶

Note: This ADR was originally authored as ADR-031 in commit 0b3cc5f but that number was simultaneously claimed by ADR-031 (GADM v4.1) in a later merge. It has been renumbered to ADR-033 to avoid a collision; git history references to "ADR-031 UI layout" refer to this content.

Date: 2026-04-23 Status: Accepted Phase: 4

Context¶

Phase 4 adds a MapLibre rendering surface, zoom-aware aggregation, and multi-layer hazard overlays. Before any Phase 4 UI code ships, the layout paradigm and visual direction must be pinned so every work item implements against a fixed target — not re-litigating the same design choices per component.

The two candidate paradigms were:

Map-as-canvas: The map fills the viewport; all controls, panels, and results float over it as sidebars or overlays. The map is always visible and spatially grounding. Users interact with the map first, not forms.
Document-mode: The page is a scrolling document. Maps appear as embedded figures within a longer result report. Familiar to report readers; loses geographic continuity as the user scrolls.

Three inspirations informed the direction:

Kepler.gl: Map-first, full-viewport; layer panel as a collapsible left sidebar; legend and filter controls as overlays. Proven for high-density point and polygon datasets. Good precedent for the Wizard → Result transition.
Mapbox Atlas: Sidebar + map split with a clear spatial hierarchy. The sidebar never competes with the map; it serves the map. Useful model for the ConfigPanel placement.
Probable Futures: Story-driven scrollytelling on a full-bleed map. Not the right mode for an interactive tool, but the "climate scenarios as spatial layers" mental model is the right one. The scenario selector pattern transfers directly.

Decision¶

Adopt map-as-canvas as the primary layout paradigm for Climate-Lama UI from Phase 4 onward.

Layout structure:

┌─────────────────────────────────────────────────────────┐
│  TopBar (logo, nav, auth)                               │
├────────────┬────────────────────────────────────────────┤
│            │                                            │
│  Left      │                                            │
│  Sidebar   │         Map Canvas (MapLibre GL)           │
│  (Wizard / │         fills remaining viewport           │
│  Config /  │                                            │
│  Results   │                                            │
│  Summary)  │                                            │
│            │                                            │
├────────────┴────────────────────────────────────────────┤
│  Status Bar (job progress, CRS, zoom level)             │
└─────────────────────────────────────────────────────────┘

Floating overlays on the map canvas:

Layer toggle panel (top-right) — shows/hides hazard and exposure layers
Legend (bottom-right) — auto-generated from active layer style
Asset detail tooltip / popover (on click) — replaces the current modal

Component taxonomy:

Component	Disposition	Notes
`Wizard`	Re-skin, not re-architect	Moves into the left sidebar; step progression unchanged
`ConfigPanel`	Re-skin, not re-architect	Collapsible section within the sidebar
`ResultMap`	Re-skin + MapLibre migration (4.1)	Becomes the full-viewport canvas
`AssetDetailPanel`	Re-skin, not re-architect	Renders as a floating popover over the map
`HazardUploadModal`	Keep as modal	Upload flow is task-modal; map context not needed
`TopBar`	Re-skin only	Brand, nav, auth state — no structural change
`LayerToggle`	New (4.9)	Floating map overlay; sources from `LayerSpec` (4.2)
`Legend`	New (4.9)	Floating map overlay; driven by active layer style

What does not change:

API client hooks (useScenarios, useExposures, etc.) — not a UI layout concern
Auth flow (Keycloak redirect, token refresh) — unaffected
Wizard step logic and validation — only the container changes
Vitest suite structure — components under test remain the same, wrappers change

Alternatives Considered¶

Document-mode layout: Rejected. The platform's primary value is spatial — "where is the risk?" Embedding maps as figures in a scrolling document loses the geographic grounding that makes climate risk data actionable. It also compounds poorly with multi-layer hazard overlays (Phase 4.3–4.5) where users need to compare layers by panning, not by scrolling between figures.
Split-pane (map left, document right): A reasonable middle ground, but it halves the map canvas at every viewport size. At laptop resolution (1366×768) the map becomes too narrow for continental-scale hazard rasters. Rejected in favour of the sidebar-over-canvas model which collapses gracefully to full-map on smaller screens.
Kepler.gl full clone: Kepler's exact UI was considered but rejected — its layer-list paradigm is designed for exploratory data analysis, not for a guided wizard workflow. The inspiration is the layout; the interaction model must stay wizard-first.

Rationale¶

Climate risk data is inherently spatial. The map must be the primary focus, not a secondary figure in a report. Map-as-canvas commits to this.
The left-sidebar model (Kepler.gl / Mapbox Atlas pattern) collapses to a drawer on mobile and an overlay on tablet — one responsive solution, not two separate layouts.
Keeping existing components as re-skins rather than rewrites limits Phase 4 scope and preserves the test coverage already in place. Architectural changes in the same phase as a renderer migration (4.1) would be too risky.
Pinning the paradigm now means Phases 5 and 6 (UX quality, i18n, datasets browser) have a stable layout contract to implement against. Without this ADR, every Phase 5/6 issue would reopen the layout question.

Follow-up¶

4.1 MapLibre migration implements ResultMap as the full-viewport canvas per this ADR.
4.9 Layer toggle and legend implement as floating overlays per the component taxonomy above.
Phase 5 UX quality items (loading states, empty states, error boundaries) apply within the sidebar and overlay surfaces defined here — see docs/plan/phase-5-ux-quality.md.
Wireframes committed at docs/design/wireframes/ — see wireframe-01-main-layout.md, wireframe-02-wizard-active.md, wireframe-03-result-view.md.

ADR-034: Secrets and Config Management Strategy¶

Date: 2026-04-28 Status: Accepted Phase: 6

Context¶

Phase 0 retrospective flagged that no ADR covered how secrets and configuration are managed across local development, CI, and (future) hosted deploys. Issue #102 was filed to fill that gap and pulled forward to Phase 6 §6.14 to land alongside the deployment ADR (§6.13) — both are operational decisions that self-hosters need documented before any hosted deploy, and pairing them avoids re-litigating the same ground when managed hosting unparks (phase-managed-hosting.md).

The status quo, never written down, is: .env files loaded through pydantic-settings in dev; GitHub Actions repository secrets injected as env vars in CI (.github/workflows/); no production target yet. The risk of leaving this implicit is that every new contributor or self-hoster has to reverse-engineer the convention from config.py, and that the future hosted deploy reopens the question without a documented baseline.

Decision¶

Climate-Lama uses a single config plane: environment variables, materialised differently per environment. All three layers — dev, CI, hosted — converge on the same Settings class in src/climate_lama/config.py; only the source of those env vars changes.

Per-environment sourcing¶

Environment	Source of env vars	How `Settings` reads them
Local dev	`.env` file in repo root (gitignored)	`pydantic-settings` loads `.env` automatically; `.env.example` is the canonical schema
CI	GitHub Actions repository / environment secrets	Workflow steps export `secrets.FOO` as `env:` entries; no `.env` file is written
Hosted	Per-environment secrets backend (deferred — see below)	Backend injects env vars into the container at start; `Settings` is unchanged

This means the application code never branches on environment to decide how to read a secret. The boundary between "where the secret lives" and "how the app consumes it" is the OS environment, and that contract is the same everywhere.

Hosted-path backend candidates (final pick deferred)¶

The hosted backend is not chosen in this ADR. The decision is deferred until managed hosting unparks (phase-managed-hosting.md) because the choice is coupled to the platform decision (Kubernetes provider, ArgoCD vs. Helm-only, etc.) made in §6.13. Candidates to evaluate at that time:

Candidate	Fits when	Watch-outs
Hetzner vault	Hosting lands on Hetzner; want minimum vendor surface area	Operator-run; rotation tooling is DIY
Doppler	Want a managed UI for secret editing across dev/staging/prod with one tool	Per-seat pricing; another vendor in the trust chain
1Password Connect	Team already uses 1Password; want a self-hosted sync container reading from existing vaults	1Password Business required; Connect server must be HA in prod
AWS Secrets Manager	Hosting lands on AWS; want native IAM-scoped access and built-in rotation hooks	Locks the hosted plane to AWS; cross-cloud egress costs if other components live elsewhere

All four expose secrets as env vars to the running container — the application code does not change with the choice. The decision criteria captured for the future evaluation: (1) does it match the hosting platform decided in §6.13, (2) does it support per-environment scoping (dev/staging/prod) without manual file shuffling, (3) does it have a rotation story that does not require redeploying the app to roll a single secret.

Rotation story¶

Who rotates. The owning team for each secret rotates it. APP_SECRET_KEY, API_KEY_PEPPER, database passwords, and MinIO/object-storage credentials → backbone maintainers. SMTP, OIDC, Cloudflare, and SDK-smoke credentials → whoever owns the upstream account. The CODEOWNERS file is the source of truth for "owning team."
Cadence. Long-lived shared credentials (DB password, MinIO root key, APP_SECRET_KEY, API_KEY_PEPPER) rotate at least every 90 days and immediately after any suspected exposure or contributor offboarding. Per-tenant or per-user credentials follow whatever the issuing service mandates (e.g., OIDC client secrets follow the IdP's policy).
How rollouts pick up the new value. In dev, contributors edit .env and restart the process — no further mechanism. In CI, the rotation is a single GitHub repository / environment secret update; the next workflow run picks it up. In the hosted plane, the chosen backend (above) must support either (a) restart-free reload via a sidecar that re-renders env on rotation, or (b) a controlled rolling restart triggered by the rotation event. Whichever backend is chosen, the rotation procedure must be documented as a runbook in docs/ops/ before the first hosted deploy.

Env var naming conventions¶

The conventions below codify what is already in use in config.py and .env.example, so future additions stay consistent.

Casing: UPPER_SNAKE_CASE for the env var name; the matching Settings field is lower_snake_case (pydantic-settings handles the case fold automatically because case_sensitive=False).
Subsystem prefix as scope marker: env vars are grouped by subsystem prefix — APP_*, DATABASE_*, REDIS_*, MINIO_*, CELERY_*, SMTP_*, OIDC_*, AUTH_*, HAZARD_UPLOAD_*, TITILER_*. New subsystems get their own prefix; do not reuse APP_* as a catch-all.
CL_* reserved for backbone-specific runtime hooks: operator-only feature flags or one-shot bootstrap toggles that are not a subsystem of their own use the CL_ prefix (e.g., CL_SEED_DEMO). This makes them grep-able as "things only an operator should set."
No platform prefix on shared keys: DATABASE_URL, REDIS_URL, CORS_ORIGINS keep their conventional ecosystem names so containers can be wired up by any operator who has run a Python web service before.
Secret values in placeholder form: .env.example ships placeholder defaults that match _PLACEHOLDER_SECRETS in config.py; Settings._reject_placeholder_secrets_in_prod refuses to boot in prod with these values. This contract — placeholder-rejection in prod — is part of the convention; new secrets that ship with a placeholder default must be added to _PLACEHOLDER_SECRETS.

What does not change¶

pydantic-settings stays the single config-loading mechanism. No second config layer (YAML, TOML, Consul KV) is introduced.
.env is never committed; .env.example is the only checked-in env file and serves as schema documentation.
The 12-factor convention of "config in env, code in repo" continues to apply to all three planes.

Alternatives Considered¶

Committed encrypted secrets (sops, git-crypt, blackbox). Tempting because the secret travels with the code and there is no second system to operate. Rejected because: (a) it inverts the "secrets are not source code" mental model and trains contributors to treat the repo as a secret store; (b) every rotation is a commit, so audit history publicly records which secrets exist and when they changed even when the values are encrypted; (c) key management for the encryption key just moves the same problem one layer down — someone still has to distribute and rotate the master key out of band; (d) GitHub Actions and the future hosted backend already give us secret stores with proper RBAC and audit logs, so the marginal value over the env-var path is low while the cognitive cost is high.
Per-service secrets in their own backend (e.g., DB password lives in a Postgres-aware vault). Rejected for now: the application code would have to learn each backend's SDK, breaking the "Settings is the only config plane" property. Revisit only if a specific subsystem (e.g., short-lived DB credentials via cloud IAM) makes the env-var pattern unsafe.
HashiCorp Vault as the universal backend. Powerful and well-known, but operationally heavy for the team size; introduces a hard dependency on a Vault cluster or HCP Vault subscription before the first hosted deploy. Kept on the candidate list for the future evaluation, not as the recommended path.
No ADR — keep it implicit in config.py. The status quo. Rejected for the reason this ADR exists: every new contributor and self-hoster reverse-engineers the convention, and the future hosted decision has no documented baseline to extend.

Rationale¶

One config plane (env vars) keeps the application code identical across dev, CI, and hosted, which is the property that lets the hosted backend decision be deferred without blocking work today.
Codifying the existing convention (rather than inventing a new one) means this ADR is descriptive, not prescriptive — it lowers the change-management cost to zero for current contributors while still giving self-hosters and future-us a written contract.
Capturing four hosted backend candidates with explicit fit-criteria, instead of picking one now, avoids the worst failure mode for ADRs in this area: locking in an opinion before the constraints (hosting platform, team size, rotation tooling) are knowable. The §6.13 deployment ADR will resolve the platform; this ADR resolves the contract that the platform decision plugs into.
Rejecting committed-encrypted-secrets explicitly is load-bearing: it is the most common alternative proposed in OSS projects, and a future contributor will ask "why not sops?" The answer needs to be in writing.
Naming conventions are codified so that subsystem prefixes stay grep-able and the CL_* operator-flag namespace does not get colonised by feature flags.

Follow-up¶

Hosted backend selection happens in tandem with §6.13 deployment ADR when managed hosting unparks (phase-managed-hosting.md). At that point, file a follow-up ADR (or amend this one) recording the chosen backend and the rotation runbook.
Rotation runbook: docs/ops/secret-rotation.md — authored 2026-08-06 (#640). It covers the plane that actually exists: .env.prod on the single-tenant Hetzner host, the host's ghcr-pull-hetzner GHCR PAT, the prod-host deploy keypair, and the SDK_SMOKE_* repo secrets — with per-secret blast radius, a verification command that proves the new value works, and an explicit revocation step for the old one. It does not cover a hosted secrets backend, because none is chosen; when one is (#285), the runbook's "edit .env.prod and restart" steps are superseded by that backend's rotation path and the runbook is rewritten alongside the resolving ADR.
Cross-referenced from phase-managed-hosting.md retrospective backlog table so the parked phase points at this ADR as the resolved decision for #102.
Cross-referenced from ADR-035: the release pipeline and deployment posture that the secrets plane documented here plugs into.

ADR-035: Deployment Posture and Release Pipeline¶

Date: 2026-04-28 Status: Accepted Phase: 6 Amended: 2026-08-01 — release images are signed with cosign keyless OIDC (#487); see §Image signing below.

Context¶

Phase 6 §6.13. Self-hosters need a documented deployment posture and a release pipeline so versioned GHCR image tags drop automatically on git tags. ADR-034 (secrets) established the config plane; this ADR establishes the artifact plane — how images are built, tagged, and distributed — and the upgrade path operators follow. The managed-hosting path (Kubernetes + Helm + ArgoCD) is sketched here as a decision record but deferred until that phase unparks.

Decision¶

Self-hoster path (current)¶

Self-hosters run the stack via Docker Compose, pulling versioned images from GHCR:

ghcr.io/cortomaltese3/climate-lama:<tag>   # backbone (API + worker)
ghcr.io/cortomaltese3/climate-lama-ui:<tag> # UI (published from sibling repo)

The docker-compose.yml api and worker services carry image: ghcr.io/cortomaltese3/climate-lama:${CLIMATE_LAMA_TAG:-latest} alongside their build: block so operators can pin a specific release tag while local-dev builds remain unaffected (Compose builds when build: is present and the image is not already pulled).

Universal artifact¶

Backbone image (API + worker): built from docker/core.Dockerfile / docker/worker.Dockerfile and published from this repo's release workflow.
UI image: built and published from climate-lama-ui independently. Already in use via the fullstack Compose profile.

Both images are the deployable units for all environments, whether self-hosted or (eventually) managed.

Tagging contract¶

Tags are v{major}.{minor}.{patch} semver only. No commit SHAs, no branch-name tags, in user-facing image tags.
A moving latest tag is updated on every stable release (non-rc push).
Pre-release candidates use v{major}.{minor}.{patch}-rc{N} (e.g., v0.1.0-rc1). The release workflow builds and pushes the rc tag but never updates latest.

Release pipeline¶

A GitHub Actions workflow (.github/workflows/release.yml) triggers on push of tags matching v*. Two parallel jobs build and push the backbone image family to GHCR using the workflow's GITHUB_TOKEN (packages: write). Layer caching uses type=gha via docker/build-push-action@v6. See .github/workflows/release.yml.

Publishing is additionally gated on a green CI run for the exact tagged commit — see ADR-046. Both build jobs are needs:-gated on a verify-ci job; the signing steps added below live inside those jobs, so signing never runs on a commit CI has not proven green.

Image signing (cosign keyless, added 2026-08-01)¶

Method: cosign keyless signing via GitHub Actions OIDC. sigstore/cosign-installer@v3 runs in both release jobs, each of which carries id-token: write; after the push, cosign sign --yes "${IMAGE}@${DIGEST}" signs the digest emitted by docker/build-push-action — never a tag. No key material is generated, stored, or rotated, and no new repository secret is introduced: the short-lived Sigstore certificate is minted from the job's OIDC identity and recorded in the public Rekor transparency log.

Signing by digest rather than tag is the load-bearing choice. :latest is re-pointed on every stable release, so a tag-bound signature would either go stale or, worse, appear to vouch for whatever image later claimed the tag. A digest-bound signature is valid for every tag that resolves to that digest and cannot be inherited.

Who verifies: verification is a documented manual operator step, not an automated gate — docs/deployment/upgrading.md §"Verifying image signatures" carries the exact cosign verify invocation as a pre-upgrade step. Nothing in the Compose stack verifies signatures automatically, so an operator who skips the step upgrades unverified. This is a deliberate, revisitable trade-off: an enforcing gate belongs in the deploy path, and adding one there is tracked as follow-up rather than bundled here. The verification identity is pinned to release.yml on a v* ref (--certificate-identity-regexp '^https://github\.com/CortoMaltese3/climate-lama/\.github/workflows/release\.yml@refs/tags/v') with issuer https://token.actions.githubusercontent.com, which is a strictly stronger claim than a repo-wide identity pattern.

Scope: backbone and worker images only. The UI image is built and published from climate-lama-ui and must adopt signing in its own repo. Tags v0.4.0 and earlier predate signing and are unsigned.

Upgrade path¶

Pull the new image tag: CLIMATE_LAMA_TAG=vX.Y.Z docker compose pull
Run migrations: docker compose run --rm migrations alembic upgrade head
Restart services: docker compose up -d

Downgrade: re-pin CLIMATE_LAMA_TAG to the prior tag and run alembic downgrade only if the release notes explicitly note a reversible migration.

Hosted path (sketched, deferred)¶

When managed hosting unparks (phase-managed-hosting.md), the target architecture is:

Kubernetes (provider TBD — see ADR-034 §hosted-path-backend-candidates for coupling to secrets backend).
Helm chart skeleton per service; chart sources will live under infra/helm/.
ArgoCD GitOps sync from the release branch.
cert-manager for automatic TLS certificate provisioning.
Multi-arch builds (arm64): identified as a next-easy-win but not implemented here — call it out in the release workflow when the hosted platform decision lands.

No Helm chart, K8s manifests, or ArgoCD application are shipped in this issue.

Alternatives Considered¶

Publish images on every push to main (SHA-tagged). Rejected: SHA tags are not meaningful to operators and latest on every merge would make it impossible to pin a stable version without knowing the SHA. Semver tags on explicit git tags are unambiguous.
Single combined image (API + worker in one container). Rejected: the API and worker have different scaling and restart profiles; separate images let operators scale them independently and avoid restarting the API when the worker crashes.
Docker Hub instead of GHCR. Rejected: GHCR is already used for the UI image (climate-lama-ui); consolidating on one registry reduces credential surface and the GITHUB_TOKEN flow requires no extra secret. Docker Hub would add a push-rate-limit concern for self-hosters.
Build images in CI (not just on tags). Out-of-scope for this issue; a dev or edge tag from main pushes is a reasonable follow-up once the release pipeline is proven.

Rationale¶

Publishing on git tags — not commits — keeps latest meaningful and gives self-hosters a stable pinning target without requiring them to track commit SHAs.
Keeping build: alongside image: in docker-compose.yml means local development is unaffected by the pin variable; the workflow for contributors does not change.
Sketching the hosted path in this ADR (rather than a future one) avoids re-litigating the universal artifact decision when managed hosting unparks: the Helm/ArgoCD work extends this ADR's artifact model, it does not replace it.
Cross-reference: ADR-034 resolves the config/secrets plane that the hosted path will plug into; ADR-029 covers the DO Spaces bucket layout that sits alongside image storage as an ops concern.

Follow-up¶

Helm chart skeleton, K8s manifests, ArgoCD application — tracked in phase-managed-hosting.md.
~~Image signing (cosign / Sigstore)~~ — done 2026-08-01 (#487); see §Image signing above.
Enforcing signature verification in the deploy path (rather than the documented manual pre-upgrade step) — open; the deploy workflow is the natural home for a cosign verify gate.
SBOM generation — separate compliance item.
Multi-arch builds (arm64) — separate item, low-cost once the hosted platform is chosen.

ADR-036: SSRF Policy for Admin Download Endpoints¶

Date: 2026-04-29 Status: Accepted Phase: 6

Context¶

Phase 6 §6.7 introduces an admin endpoint that takes a URL and streams the body into MinIO under raw/{source}/{dataset_id}/{filename}. Without controls, that endpoint is a generic SSRF primitive: an authenticated platform admin could probe internal infrastructure with http://localhost:8080, file:///etc/passwd, redirects to 169.254.169.254 (cloud metadata), or DNS rebinds against private IPs. A compromised or malicious admin account is a real threat for public-facing operator consoles.

This ADR records the policy that closes that surface — a layered allowlist (platform default + per-org), strict scheme + DNS rules, and defence-in-depth re-validation on every redirect hop and again at task entry.

Decision¶

The SSRFPolicy.validate(url, *, org_id) chokepoint enforces all of the following before any HTTP traffic leaves the worker process. A failure at any layer raises ClimateLamaError(code=Code.E_SSRF_REJECTED) with a structured reason (scheme, hostname_not_allowed, private_ip, dns_failure, ip_in_url, redirect_no_location, too_many_redirects).

Scheme restriction¶

https:// only. http://, file://, ftp://, gopher://, etc. all reject outright. Plaintext fetches are not permitted regardless of allowlist membership — the SSRF surface is too wide and the legitimate sources we care about all support TLS.

Allowlist match semantics¶

Two layers, unioned:

Platform default — download_platform_allowlist in config.py, overridden via the DOWNLOAD_PLATFORM_ALLOWLIST envvar (comma-separated). Adding entries here is a code change so it goes through PR review. v1 ships with: data.jrc.ec.europa.eu, cds.climate.copernicus.eu, noaa.gov, worldpop.org, data.openstreetmap.org, data.climateanalytics.org.
Per-org — rows in org_download_allowlist, scoped via RLS to the caller's org. Writes are platform-admin only and land via the 6.2 admin endpoints; the table + RLS are introduced here so the policy has somewhere to read from.

A hostname matches a suffix when it is exactly the suffix or is a subdomain of it (data.jrc.ec.europa.eu matches jrc.ec.europa.eu). The dot boundary is required so evilnoaa.gov does not match noaa.gov.

DNS resolution + private-IP rejection¶

Resolve the hostname once before opening the connection. If any returned A/AAAA record falls in any of these address spaces, reject:

RFC 1918: 10/8, 172.16/12, 192.168/16
Loopback: 127/8, ::1
Link-local: 169.254/16 (covers AWS/GCP IMDS), fe80::/10
Unique-local IPv6: fc00::/7
is_reserved, is_multicast, is_unspecified (0.0.0.0/::)

The policy returns the resolved IP set so callers can pin connections to those IPs (mitigating DNS rebinding). The pin-to-IP transport is a // TODO in v1 — the chokepoint validation + re-validation on the worker covers most of the rebind window in practice; full pinning is a follow-up tracked outside this issue.

Redirect re-validation¶

Up to MAX_REDIRECTS = 5 hops. Each redirect target is validated against allowlist + private-IP from scratch. Reject if any hop fails — a redirect from an allowlisted host to 127.0.0.1 is the canonical SSRF rebind and must not slip through.

Defence in depth¶

The policy runs twice:

At enqueue time, in the POST /v1/admin/downloads endpoint, against the org's combined allowlist. This ensures a rejected URL never produces a failed row.
At task entry, in download_to_spaces, against the same combined list. This guards the window between enqueue and pickup — a malicious admin race or a tampered row would otherwise slip through. The Celery task also re-validates on every redirect hop the worker walks.

Both validations raise the same E_SSRF_REJECTED and stamp the row with that error code so the admin UI can render typed failure states.

Error code¶

E_SSRF_REJECTED (registered in core/errors.py, regenerated into the SDK as SsrfRejectedError). HTTP status 400 — the request was syntactically valid but rejected by policy.

Alternatives Considered¶

Blocklist private IPs only, no hostname allowlist. Rejected: the private-IP check alone is bypassable via DNS rebinding and does nothing about file:// schemes or unwanted egress to legitimate but untrusted public hosts. Allowlist-first is the only safe default.
Allow http:// for the platform default list. Rejected: TLS-less fetches expose download contents to network observers and to MITM tampering; HTTPS-only matches the threat model and the public sources we care about all support it.
Single global allowlist, no per-org layer. Rejected: orgs need controlled exceptions for their own internal data services without forcing a code change on the platform. Per-org rows give that flexibility while keeping the platform list tight.
Validate once at enqueue, not at task entry. Rejected: gives an attacker a window to tamper with the row between enqueue and run. The cost of re-validation (one DNS lookup) is negligible against a real attack scenario.

Rationale¶

Allowlist + scheme + private-IP rejection is the layered defence the OWASP SSRF cheat sheet recommends; each layer alone is bypassable but the union closes the documented attack patterns.
A code-change platform list keeps the surface auditable in PR review; a per-org table keeps operator-scope changes self-serve without widening the global default.
Validating at enqueue and at task entry costs one extra DNS lookup per download and removes the TOCTOU window between the two — cheap insurance for a security-sensitive surface.
Rejecting literal IPs in URLs sidesteps a class of trivial allowlist bypasses (https://10.0.0.5/) without restricting any legitimate source.

Follow-up¶

DNS-rebinding mitigation by pinning the connection to the resolved IP via a custom httpx.AsyncHTTPTransport — separate hardening item once the base policy is in place.
Per-org allowlist write endpoints — filed under §6.2 (admin user/org endpoints).
Bandwidth throttling and per-org quotas — future ops item.

ADR-037: In-App Notification Taxonomy¶

Date: 2026-06-10 Status: Accepted Phase: 6b

Context¶

Phase 6b.10 (issue #322) adds an in-app notification feed surfaced by a bell in the global nav. The backbone persists notifications (notifications table, migration 0049) and emits them on user-facing events; the UI polls and renders them. Each notification carries a type that the UI maps to an icon, copy, and deep link. An open-ended or drifting set of types would force the UI to handle unknown values and make the feed inconsistent across the stack, so the issue locks the taxonomy to a fixed set and asks for a registry — analogous to the error-code registry in ADR-028, but for notification types rather than error codes.

Decision¶

The notification type is locked to exactly five values. This ADR is the registry; the authoritative enum is NotificationType in src/climate_lama/models/enums.py, and a DB CHECK constraint (ck_notifications_type, migration 0049) enforces the same set at the storage layer.

Type	Emitted when	Payload keys
`job_complete`	A compute job (impact or cost-benefit) finishes successfully.	`job_id`, `job_type`, `result_id` / `result_ids`
`dataset_available`	An async hazard ingest commits and the dataset is ready.	`dataset_id`
`system_announcement`	A platform-wide operator broadcast.	free-form
`admin_approval_needed`	An action is waiting on admin approval (6b.8).	free-form
`download_ready`	An admin-triggered raw download lands in object storage.	`download_id`, `target_path`

Emission goes through core/notifications.py (emit_notification for request handlers, emit_notification_sync for Celery workers), mirroring core/audit.py: both are append-only and best-effort, so a failed notification write never fails the originating operation. The payload is opaque to the backbone — only the UI interprets it.

Stability contract¶

The five values are immutable identifiers; once published they are never renamed or removed (same contract as ADR-028 error codes).
Adding a type is a deliberate change that touches three places in one PR: the NotificationType enum, the ck_notifications_type CHECK (new migration), and this table.
Clients must render an unknown type gracefully (generic fallback) rather than erroring.

Alternatives Considered¶

Free-text type column, no constraint: simplest, but invites drift and forces the UI to defensively handle arbitrary strings. Rejected — the issue explicitly asks to lock the set.
Reuse the ADR-028 error registry (Code): the issue's wording ("ADR-028 registry") hints at this, but notifications are not errors — they have no severity and a different lifecycle (read/dismiss). Folding them into the error catalogue would overload it. Rejected in favour of a parallel, purpose-built registry that follows the same immutability discipline.

Rationale¶

A fixed, catalogued set gives the UI a stable target to map icons/copy/links against, consistent with how ADR-028 stabilised error presentation.
Enforcing the set in both the enum and a DB CHECK prevents code–schema drift: an unrecognised value cannot be written even if a caller bypasses the enum.
Routing all emits through one best-effort helper guarantees a lost notification can never turn a successful job, ingest, or download into a failure.

Follow-up¶

Wire admin_approval_needed emits once the admin-approval flow (6b.8) lands; the helper already supports the type.
Wire system_announcement once an operator-broadcast surface exists.
The UI half (bell + dropdown + 30s poll) is tracked as the climate-lama-ui slice of #322.

ADR-038: Image Distribution — Private GHCR, Pinned Tags¶

Date: 2026-06-11 Status: Accepted Phase: 2 (Production)

Context¶

Container images are distributed via GitHub Container Registry (GHCR): climate-lama (api), climate-lama-worker, and climate-lama-ui. Two issues surfaced while wiring up the deploy flow:

Production built from source. docker-compose.prod.yml ran up -d --build, and rollback was git checkout <sha> + rebuild. There was no immutable, versioned artifact: a deploy's contents depended on the working tree, builds needed a toolchain on the prod host, and rollback meant a rebuild rather than re-pulling a known-good image. The api/worker images had in fact never been published — tags v0.1.0–v0.3.0 predate release.yml, so it never ran.
Registry access needed a deliberate choice. The packages are private, so anything that pulls them needs a credential. The realistic set of pullers is small: a single prod host (Hetzner) and, optionally, a developer pulling the UI image instead of building it. CI is not a puller — the smoke workflows (sdk-smoke.yml, e2e-smoke.yml) build images from source.

Decision¶

GHCR packages stay private. Even though the source is public, the built artifacts are not published openly. The only routine puller is the prod host, which authenticates to GHCR once (docker login) with a classic PAT scoped to read:packages; the credential is cached and reused for every later pull. See docs/DEPLOYMENT.md → "Registry Authentication".
Production pulls pinned, published images; it never builds. docker-compose.prod.yml references ghcr.io/cortomaltese3/climate-lama{,-worker}:${CLIMATE_LAMA_TAG} with a required variable (${CLIMATE_LAMA_TAG:?…}, no :latest fallback). Deploys and rollbacks are a tag change + pull + up -d.
Images are published on v* tags by release.yml (api/worker) and publish.yml (ui). A release tag is the only thing that mints a deployable artifact.

The UI remains a separately deployed unit (its own repo/instructions); the fullstack profile in the dev compose is for demos/smoke-tests only. Because the packages are private, running that profile from a clean clone requires a docker login first (or building the UI locally) — an accepted trade-off given the only routine consumer is the prod host.

Alternatives Considered¶

Make the packages public. Zero-auth pulls everywhere, and it would restore a frictionless "clean clone → fullstack up" demo. Rejected: the owner prefers not to publish container artifacts openly, and the only routine puller is a single host where a one-time authenticated login is cheap.
Keep prod building from source. Works, but yields non-reproducible deploys (working-tree-dependent), needs a build toolchain on the host, and makes rollback a rebuild. Rejected in favour of immutable artifacts.
Default prod to :latest. Convenient but mutable — no reproducibility, no clean rollback, cache ambiguity. Rejected; the tag is required.

Rationale¶

Pinned tags give reproducible deploys and one-step rollback (change the tag, re-pull) without a build host.
Keeping images private costs almost nothing operationally: one docker login on the single prod host, cached thereafter. CI is unaffected because it builds from source.
Publishing only on v* tags keeps a clean line between "merged to main" and "released and deployable".

Consequences / Follow-up¶

Token rotation. The prod host's read:packages PAT expires; it must be rotated (regenerate + docker login) before it lapses, or an update deploy fails with unauthorized on docker pull. Track the expiry.
The first publish of the api/worker images requires cutting v0.4.0 (the v0.1.0–v0.3.0 tags predate release.yml).
Consider adding workflow_dispatch to release.yml so an existing tag can be (re)published without a version bump.

ADR-039: Report Generator — WeasyPrint in the Worker¶

Date: 2026-07-27 Status: Accepted Phase: 8 (Foundations + Showcase)

Context¶

Reports are the shared primitive every persona consumes (phase 8.2): the insurer's ORSA pack, the decision-maker's briefing and the researcher's methods appendix are all the same pipeline with different templates. What existed was a placeholder — api/v1/reports.py drew an fpdf2 document cell-by-cell, synchronously, inside the request handler. It had no template system (layout was imperative Python, so a second report meant a second function), no charts, no provenance citation, no attribution, and no persisted output.

Two candidates were on the table for the replacement.

Decision¶

Render composed reports with WeasyPrint, running in the Celery worker.

Templates are Jinja2 HTML + one print stylesheet, under src/climate_lama/core/reports/templates/. A new report is a new HTML file plus one TEMPLATES registry entry.
The pipeline is split into four stages with a pure data seam between gathering and everything after it: gather (DB → ReportContext) → compose (→ HTML) → render (→ PDF) → store.
jinja2 is a core dependency (composing is pure Python and useful in any process); weasyprint is a worker extra (only rendering needs the native stack).
Rendering is queue work: POST /v1/results/{id}/report returns 202 with a job id; GET /v1/results/{id}/report?format=pdf serves the stored document.
Charts are emitted as hand-written inline SVG, adding no plotting library.

License check per ADR-026. WeasyPrint is BSD-3-Clause; Jinja2 is BSD-3-Clause; both are on the ADR-026 allowlist and neither is copyleft, so nothing here constrains the platform's own (still deferred) licensing. The transitive stack WeasyPrint pulls — pydyf, tinycss2, cssselect2, tinyhtml5, fonttools, Pyphen — is BSD/MIT/MPL-family, with Pillow under the permissive HPND. fpdf2 (LGPL-3.0) is removed by this change, which also retires the only weak-copyleft dependency in the core install.

Alternatives Considered¶

WeasyPrint in the worker (chosen) — HTML/CSS templates rendered by a pure-Python library over a native Pango text-shaping stack.
A Quarto container — Pandoc + a TeX or Typst engine in a sidecar image, driven by .qmd documents.
Keep fpdf2, add structure — build a layout layer over the existing imperative primitive.
Headless Chromium (Playwright) print-to-PDF — considered briefly since the templates are HTML either way.

Rationale¶

Weight. A Quarto image with a TeX distribution is 1.5–3 GB; Chromium is ~400 MB plus a process supervisor. WeasyPrint adds ~15 MB of Python plus ~40 MB of Pango/HarfBuzz .so files to a worker image that is deliberately ~200 MB (ADR-024). For one document family, the heavy options cost an order of magnitude more image for no capability we need.
One templating language, already in the stack. Quarto would introduce .qmd + Pandoc filters as a second authoring system alongside the Jinja templates the project already reads; Chromium would need HTML anyway and a browser. WeasyPrint renders the HTML directly.
Print semantics without a browser. @page, page counters, running footers and display: table-header-group are exactly the paged-media features a multi-page risk report needs, and WeasyPrint implements them natively. Chromium's print path supports a narrower slice of them.
In-process, so failures are ordinary. No sidecar to health-check, no subprocess to reap, no shared volume to hand a file across. A render error is a Python exception the task's existing failure handler already records on the job row.
fpdf2 could not get there. Its model is imperative cursor movement; the cover/metrics/chart/methodology/annex structure needs cascading layout, and building that over fpdf2 would be reimplementing a CSS engine.

The cost, stated plainly: WeasyPrint is a cffi binding, so import weasyprint raises OSError — not ImportError — when libgobject/libpango are absent. That is the normal state on Windows developer machines. Three things follow, and they are load-bearing:

core/reports/renderer.py is the only module that imports WeasyPrint, and it translates both ImportError and OSError into one typed ReportRendererUnavailableError.
Because composition is pure, everything except the final rasterisation is testable everywhere. The PDF-bytes smoke test skips when the native stack is missing, and CI installs the libraries explicitly so that skip never fires there.
The API image deliberately does not carry the native stack. It cannot render, only serve stored documents — which is the intended shape, not a limitation.

If a future template needs LaTeX-grade typesetting (numbered theorems, BibTeX), that is the trigger to revisit alternative 2 — for a specific template, not for the pipeline, since gather/compose/store stay unchanged.

Storage layout¶

Rendered documents follow ADR-029's reports/ prefix:

reports/{org_id}/{job_id}/{template}/report.pdf
reports/{org_id}/{job_id}/{template}/manifest.json

{job_id} is the compute job that produced the result (impact_results.job_id, unique per result), not the render job. That makes the path derivable from a result id alone, so the GET endpoint finds the document without being told which render job wrote it. {template} is the final path component so several templates can coexist per result, each with its own leaf.

The manifest is written last and is the leaf's completion signal, per ADR-029 — which also gives the read path its readiness test for free: no manifest means not-yet-rendered, so a half-uploaded PDF is never served as a finished report. Its contents deviate from ADR-029's dataset manifest schema (a rendered document has no CRS, units or upstream fetch); it carries template, checksum_sha256, bytes, result_id, render_job_id and rendered_at instead. The invariant ADR-029 exists to protect — written last, sole marker of a complete leaf — is preserved exactly.

API semantics¶

GET /v1/results/{id}/report?format=pdf returns 404 with E_REPORT_NOT_RENDERED when no document is stored. The alternative — 200 carrying a job-status pointer — would force every client to content-sniff a response it asked to be application/pdf, and would make "no document" a success for anything that only checks the status code. The distinct error code keeps "render it first" separable from "no such result", which shares the 404. JSON and CSV exports stay synchronous and unchanged.

Follow-up¶

The ORSA-specific template is issue #382; it extends base.html and adds a registry entry, with no change to gather/render/store.
No retention policy yet: rendered PDFs accumulate under reports/. Unlike tiles/, they are not regenerable once the underlying result is deleted, so a sweep needs a decision about whether reports outlive their result.
Report rendering is not metered into usage_events; the compute tasks are. Worth revisiting if rendering becomes a material cost.

ADR-040: Two-Plane Answer Layer — RiskSurface + RiskCell on H3¶

Date: 2026-07-27 Status: Accepted Phase: 8 (Foundations + Showcase)

Number provisional: the phase-8 plan (docs/plan/phase-8-foundations-orsa.md) reserves ADR-039 for the report-generator choice (8.2) and ADR-040 for this decision (8.3); if 8.2 merges after this one, renumber at that time so ADR numbers stay assigned in merge order.

Context¶

The backbone's only spatial-answer path today is the factory plane: a scenario run computes an ImpactResult on demand, every time. Interactive questions ("what's my flood risk at this address", "score this portfolio") would each pay a full compute — unnecessary for locations already covered by prior runs or by curated public datasets. docs/plan/exploration/04-answer-layer-rfc.md (ratified 2026-07-26, amendments A1/A3) proposes a second, read-optimized answer plane that accelerates repeat/location questions without changing how the factory plane works. This ADR covers the plane's schema only — the surface writer, read API, scoring, and eviction logic are separate, later items (8.4 onward).

Decision¶

Two planes, factory keeps primacy. The factory plane (scenario -> job -> worker -> engine, i.e. ImpactResult as it exists today) is unchanged and stays first-class forever — the answer plane never replaces or degrades ad-hoc compute, it only accelerates questions a prior run or a curated build already answered.
RiskSurface — the factory-run registry: one row per hazard x region x scenario x horizon x engine x version, carrying status, provenance, and the set of H3 resolutions actually built (cell_resolutions).
RiskCell — the precomputed grid, indexed by H3 (recommended over native hazard grids or admin polygons: uniform across hazards with different native grids, and zoom/tile-friendly by construction). Columns: h3_index, resolution, a surface_id FK, metrics (JSONB — intensity per return period, EAI density, banded score), and a geom polygon derived from h3_index for tiling.
Two table pairs, one contract. Surfaces built from public datasets are org-less reference data — reference_risk_surfaces / reference_risk_cells, precedent admin_boundaries (migration 0031): no org_id, no RLS, every tenant reads the same rows. Surfaces derived from a tenant's own private data are org-scoped — org_risk_surfaces / org_risk_cells, both carrying a NOT NULL org_id and the standard FORCE ROW LEVEL SECURITY org-isolation policy (migration 0022 et al.), applied directly on each table rather than joined through the parent surface.
Isolation invariant. An org-derived surface or cell is never readable from a different org, and promoting an org-derived surface into the reference plane is a manual, deliberate act performed by a future reference-surface build task — there is no automatic or implicit path from the org-scoped tables into the reference tables.
Storage discipline: DB cells capped at r8. Cells live in Postgres only at coarse/medium resolutions (r5-r8; e.g. Greece at r8 is roughly 180k cells per surface — inexpensive even across dozens of surfaces), enforced by a CHECK (resolution <= 8) constraint on both cell tables. Finer-than-r8 truth is not duplicated into the DB: a point lookup combines the r7/r8 cell for banded score/context with a windowed COG read for the exact intensity/depth at the coordinate, keeping the DB bounded and the answer exact.
Surface saves are cache-semantic and resource-accounted (amendment A1). Factory runs additionally write their spatial results into the surface store so surfaces mature organically over time; every save is size-tracked and subject to a retention/eviction policy, like result_cache. This ADR only lands the schema — the writer that populates it is a separate, later item.
h3 added as a runtime dependency (Apache-2.0, ADR-026 allowlist; official Uber Python binding) for the surface writer's future h3_index <-> cell-boundary conversions. geom is a plain geometry column populated in Python at write time, not a native Postgres GENERATED ALWAYS AS column — that would need the h3-pg extension's SQL-level boundary function, which the postgis/postgis image this project runs does not ship, and adding a new Postgres extension is out of scope for a schema-only change.

Alternatives Considered¶

Single table with a nullable org_id. Fewer tables, but a NULL org_id sentinel for "reference" rows weakens the RLS invariant — a forgotten IS NOT NULL filter would leak org-scoped rows into a reference-plane read path. Rejected in favor of the same explicit org-less/org-scoped table split already proven by admin_boundaries vs. every other domain table.
Native hazard grids or admin (GADM) polygons as the cell index. Simpler for a single hazard, but different hazards ship on different native grids, so there is no shared index to roll up or tile across hazards. H3's fixed hierarchical grid is uniform regardless of source grid and matches deck.gl/MVT tiling directly. Rejected the native-grid option for this reason.
Store all resolutions (down to native pixel size) in the DB. Exact everywhere, no COG fallback needed. Rejected: storage blows up fast at fine resolutions (the RFC's principal risk), for a precision the r7/r8 cell + COG combination already delivers exactly.
GENERATED ALWAYS AS geometry column via the h3-pg Postgres extension. Would keep geom always in sync with h3_index at the DB layer with no application code. Rejected for this schema-only issue: it requires swapping the project's postgis/postgis base image for one bundling h3-pg (or installing it) across every environment (dev/CI/prod), a much larger change than a spatial answer-plane schema warrants on its own; revisit if the Python-side population proves to be an operational pain point.

Rationale¶

Reusing the admin_boundaries org-less pattern and the migration-0022 RLS pattern means the two-plane split introduces zero new isolation primitives — reviewers and future readers already know both shapes.
The r8 DB cap plus COG windowed reads is the schema-level lever that keeps the answer plane's storage cost bounded regardless of how many surfaces mature over time (amendment A1's "surfaces mature organically" is only safe because of this cap).
Denormalizing org_id onto the cell tables (rather than relying on a join back through surface_id) makes the RLS policy on org_risk_cells a direct exact-match check, identical in shape to every other RLS policy in the codebase, instead of a bespoke join-based policy.

Consequences / Follow-up¶

This ADR covers schema only (issue #371). The surface writer (8.4), Asset entity (8.5, ADR-041), score schemes (8.6, ADR-043), and the read-only lookup API are separate, sequenced items — the tables created here stay empty until 8.4 lands.
The celery-beat deployment gap flagged in the RFC's Risks section must be closed before any scheduled reference-surface rebuild can run reliably; tracked against the beat container work landed in #368.
If a future access pattern needs geom to be guaranteed in sync with h3_index at the database layer (not just at write time), revisit Alternative 4 (h3-pg + a native generated column) as a deliberate infrastructure change, not a retrofit onto this migration.

ADR-041: Asset Entity Distinct from Exposure + portfolio_assets Junction¶

Date: 2026-07-27 Status: Accepted Phase: 8 (Foundations + Showcase)

Number provisional: the phase-8 plan (docs/plan/phase-8-foundations-orsa.md) reserves ADR-039 for the report-generator choice (8.2), ADR-040 for the answer-layer schema (8.3, merged), and ADR-041 for this decision (8.5); if 8.2 merges after this one, renumber at that time so ADR numbers stay assigned in merge order.

Context¶

The backbone's only "one location" concept today is Exposure — a bulk, dataset-shaped row (point + value + JSONB, dataset_sha256) that engines consume directly. docs/plan/exploration/04-answer-layer-rfc.md (ratified, section C) identifies a second, identity-shaped concept the answer layer and a future portfolio-management UI both need: one real-world insured location with an address, a lifecycle (create, update, soft- or hard-delete), and attributes that describe the building rather than the compute row (floors, construction, use). Overloading Exposure with that identity semantics was considered and rejected in the RFC — it would break the dataset-hash contract that result-cache and provenance depend on (a bulk row's identity is its dataset, not any one address). This ADR lands the Asset entity and its portfolio junction as schema only, per docs/plan/phase-8-foundations-orsa.md 8.5 — no generation service, geocoding, or UI ships here (tracked separately, see Consequences).

Decision¶

Asset — one row per real-world insured location, org-scoped and RLS-covered like every other domain table (migration 0022 pattern, FORCE ROW LEVEL SECURITY with the standard exact-match org_isolation policy). Columns: address (required, Text), normalized_address (nullable — populated by a future normalization/geocoding step, out of scope here per the RFC's B11), geom (nullable POINT, geometry_type POINT/srid 4326, populated by that same future geocoding step — Asset can exist and be assigned to a portfolio before it is geocoded), attributes (JSONB, non-null default {}, open-shape for construction/floors/use — precedent: Scenario.inputs), value + unit (mirrors Exposure.value/value_unit), and deleted_at (soft-delete, precedent: Scenario/Organization).
portfolio_assets — a new junction (portfolio_id, asset_id, weight) added alongside the existing portfolio_exposures (migration 0026), not a replacement. No existing portfolio_exposures row is touched, repointed, or migrated by this change; a portfolio may carry rows in both junctions simultaneously while asset-based workflows are adopted incrementally, and callers pick the junction that matches what they're aggregating (bulk exposure rows vs. identity-shaped assets).
portfolio_assets.org_id is denormalized from both parents, not just inherited via FK join, so its RLS policy is a direct exact-match check — the same shape as every other RLS policy in the codebase, mirroring org_risk_cells' single-parent version of this pattern (migration 0055 / ADR-040). Two composite foreign keys — one to portfolios(id, org_id), one to assets(id, org_id) — make it a DB-rejected error, not an application-trusted invariant, for a junction row's org_id to ever disagree with either parent's own org_id. portfolios gains a new uq_portfolios_id_org unique constraint solely to back its half of that pair; portfolio_exposures and its existing constraints are untouched.
Data-handling note: addresses are personal data. The schema is kept erasure-friendly on purpose. No table outside this migration holds a live foreign key to assets.id — portfolio_assets is the only referencing table and cascades on delete. A future Asset -> Exposure generation service (8.5 continuation, not built here) will copy Asset attributes into freshly-created Exposure rows at generation time rather than hold a live reference back to the Asset, exactly like ImpactResult.provenance is a JSONB snapshot rather than a live reference. That means an Asset row can always be hard-deleted (a real DELETE, not just setting deleted_at) to serve a right-to-erasure request without ever breaking ImpactResult provenance, now or after the generation service lands. deleted_at is offered alongside for the ordinary "remove from active use, keep an audit trail" case — the two are independent; callers choose per situation. The formal GDPR/processor posture (DPA, retention policy, subject-access-request tooling) is deferred until an external user exists; this note only keeps the schema from painting us into a corner before then.

Alternatives Considered¶

Evolve Exposure into Asset. Fewer tables, one less join for generation. Rejected per the RFC: it would overload a bulk, dataset-hash-identified compute row with per-address identity and lifecycle semantics, breaking the dataset_sha256 contract that result-cache and provenance depend on.
Repoint portfolio_exposures at Asset instead of adding a new junction. Would avoid a second junction table, but silently changes what every existing portfolio aggregates over and requires a data migration with no clear mapping (an exposure row is not an asset). Rejected — the issue's own grounding note flags this FK as not-to-be-repointed; a new, additive junction has zero blast radius on existing portfolios.
Join-based RLS policy on portfolio_assets (EXISTS (SELECT 1 FROM portfolios ... )) instead of a denormalized org_id column. Fewer columns, no composite FKs. Rejected for the same reason org_risk_cells rejected it (ADR-040): a regression in the join can't leak cross-org rows if there's no join to regress — the direct column is one less thing a future policy change can get wrong.

Rationale¶

Keeping Asset identity-shaped and Exposure bulk-shaped means the answer layer's future Asset -> Exposure generation service has a clean, one-directional seam to write across, instead of two tables competing for the same identity.
Denormalizing org_id onto portfolio_assets and enforcing it with two composite FKs reuses a pattern the codebase already has (org_risk_cells) rather than inventing a bespoke join-based policy for this table alone.
Deciding the erasure story at schema time — no live FK into assets.id from anything outside portfolio_assets — costs nothing now and avoids a much harder retrofit later, once real addresses exist in the database and an erasure request has to actually be honored under time pressure.

Consequences / Follow-up¶

This ADR covers schema only (issue #373). One-directional Asset -> Exposure generation (with its dataset_sha256 recipe for generated batches), geocoding, score history, and any UI are separate, later items under phase-8 8.5's continuation — assets and portfolio_assets stay empty until that generation service lands.
The formal GDPR/processor posture referenced in the data-handling note above is explicitly deferred; revisit before onboarding any external (non-internal-demo) user.
If a future access pattern needs a third parent on portfolio_assets (e.g. score-scheme-versioned membership), reconsider whether the two-composite-FK shape still holds or whether a lighter single-anchor version (mirroring org_risk_cells) is a better fit at that point.

ADR-042: CLIMADA as an Out-of-Process Sidecar Engine¶

Date: 2026-07-27 Status: Accepted Phase: 8 (Foundations + Showcase)

Context¶

Engine plurality is a product commitment, and the most valuable second engine to offer is CLIMADA — it is the reference implementation practitioners already trust, so "the same scenario through both engines, and here is the divergence" is the credibility argument the platform is built to make. ADR-042 was left reserved when #377 landed the engine registry and X-Engine dispatch; this decision fills it with the first non-default engine to use that seam.

The obstacle is ADR-024: the backbone — core service and worker — must never import CLIMADA. That ban exists for good reasons (dependency weight, install fragility, the RISK WISE coupling lesson) and this ADR does not weaken it.

Decision¶

Run CLIMADA in a separate container and reach it across a process boundary.

docker/climada-sidecar/ builds an image with CLIMADA installed. Its server.py is the only runtime code in this repository permitted to import climada (alongside the pre-existing tests/parity/sidecar_runner.py). It exposes GET /health and POST /v1/impact, stdlib-only so nothing can conflict with CLIMADA's dependency pins.
src/climate_lama/worker/models/climada_adapter.py implements ModelInterface as an HTTP client. It lives inside the backbone package and is therefore bound by ADR-024 exactly like every other module there: it imports no CLIMADA, and CLIMADA is never added to pyproject.toml.
It registers under the climada selector through the existing registry — a factory plus an availability probe. The registry needed no change; #377 was designed so an out-of-process engine is an ordinary entry.

Supporting choices:

Availability is a configuration check, not a live health probe. resolve_engine_name runs at the request boundary on every engine-selecting call; a network round-trip there would add latency to every request and could hang one. More importantly, an actively-probing version would reject an entire multi-engine comparison the moment the sidecar blipped. Leaving reachability to the job means a sidecar outage surfaces as one named entry in divergence.failed_engines while the other engines still compare.
CSR-encoded intensity on the wire. A dense hazard matrix would be n_events × n_centroids JSON numbers; the CSR triplet is proportional to stored entries instead, which is what makes a real dataset fit in a request.
Pinned return periods. The adapter sends the default engine's own default return periods so both engines report losses at the same points and the agreement view compares like with like rather than interpolating.
Engine identity is asserted by the backbone, not accepted from the sidecar's response, so provenance always names the engine that was dispatched.
Private only. The image is built inside the compose stack under an opt-in climada profile and is never pushed to any registry. In prod this is a deliberate deviation from ADR-038's "pull a pinned tag, never build in prod": there is no tag to pull because there is nothing published.

Failure modes map to stable codes, with no fallback to another engine:

Condition	Error	Code
Sidecar not configured here	`EngineUnavailableError`	`E_ENGINE_UNAVAILABLE`
Configured but unreachable / timed out	`EngineUnavailableError`	`E_ENGINE_UNAVAILABLE`
Reached, calculation failed or body unusable	`EngineExecutionError`	`E_ENGINE_EXECUTION` (HTTP 502)

Consequences¶

X-Engine: climada runs a real CLIMADA calculation without a single CLIMADA import in the backbone, and the two-engine agreement view becomes real.
The worker image stays exactly as light as it was.
The ban is now enforced mechanically, not by convention: a test parses every module under src/climate_lama/ with ast and fails on any CLIMADA import, including the dynamic importlib loophole and a CLIMADA entry in pyproject.toml.
Cost: an HTTP hop and JSON (de)serialisation per job, and a large image that must be built on each host. Both are accepted — the sidecar is opt-in and the comparison is not the hot path.
Cost: CLIMADA is pinned in two places (the sidecar image and the parity harness image). They must be bumped together, or a parity result stops describing what the sidecar actually computes.
Deferred: cost-benefit through CLIMADA. The adapter inherits the base ModelInterface behaviour and fails fast rather than pretending to support work it does not.

ADR-043: Versioned Score Schemes + Per-Hazard Banding Precedence¶

Date: 2026-07-27 Status: Accepted Phase: 8 (Foundations + Showcase)

Number provisional, assigned in merge order per ADR-041's note. The phase-8 plan (docs/plan/phase-8-foundations-orsa.md) reserves ADR-039 for the report-generator choice (8.2) and ADR-042 for the geocoding adapter (8.7); neither has merged, so those numbers are left free rather than renumbering this one. ADR-043 is the number the plan reserves for 8.6, which is this decision. Rollup semantics (asset / portfolio / admin unit) were listed under the same plan bullet but are not decided here — see Consequences.

Context¶

The answer plane landed by ADR-040 stores continuous metrics per H3 cell, and the hazard COGs hold finer-than-r8 truth. Neither is an answer a non-specialist can act on: "0.62 m of flood depth at the 100-year return period" needs a scale before it becomes "7/10, red". The ratified RFC (docs/plan/exploration/04-answer-layer-rfc.md section D) calls for a seeded, versioned banding catalog — the same shape as the impact_functions catalog seeded by migrations 0004-0006 — with scores treated as presentation over metrics, never as a replacement for them.

Two constraints shaped the design. First, credibility: banded scores without external sign-off invite the "unrealistic outputs" objection already raised in stakeholder discussions. The ratified ship-and-polish decision is that v1 thresholds are self-set from published ordinal scales with citations, shipped, and polished iteratively — external actuarial/scientific review is planned polish, explicitly not a blocking gate, with an advisory note to revisit before the first paid insurer engagement. Second, spatial correctness: amendment A3.3 of the RFC corrected the banding input per hazard after a code-grounded refuter pass — an r8 hex is ~0.74 km² and spans floodplain and hillside, so a cell aggregate is not a coarser version of the address answer for flood or wildfire, it is a different answer.

Decision¶

score_schemes is an org-less, seeded, versioned catalog. One row per (hazard_type, metric, version), seeded by data migration 0057 with one v1 scheme per HazardType. Attribution columns (citation NOT NULL, source_url, license, notes) mirror impact_functions, except that citation is required here: a band with no stated basis is precisely the objection the catalog exists to answer. Unlike impact_functions the table carries no org_id and no RLS — precedent admin_boundaries and the reference plane of migration 0055. A per-org override would make two customers' "8/10" incomparable, which is the opposite of what an ordinal score is for.
Ten bands, fixed RAG tiering. Every scheme defines exactly ten half-open intervals [lower, upper) with unbounded tails, scores 1-10 ascending, and a fixed score→RAG map (1-3 green, 4-6 amber, 7-10 red) so "red" means the same severity tier whatever the peril. Band count is a DB CHECK (jsonb_array_length(bands) = 10); contiguity, ordering, tail openness, and RAG tiering are validated in core/scoring.py on load, so no malformed scheme can reach a banding call.
Per-hazard banding precedence (A3.3), encoded in code, not convention. ScoreMetricSource has exactly two members and _BAND_SOURCE_BY_HAZARD is exhaustive over HazardType: river flood and wildfire band from the COG point intensity at the coordinate; storm Europe and tropical cyclone band from a cell metric (their fields are smooth at hex scale). A scheme whose declared metric_source disagrees with its hazard's rule is rejected at load. There is no cross-source fallback: when the required input is missing the resolver raises ScoreMetricUnavailableError rather than substituting the other source, because a hex aggregate silently standing in for a point read answers a different question. Cell-derived bands remain authoritative for area rollups regardless of hazard.
A scheme change is a new version row; re-banding is a recompute. Thresholds are never edited in place — a revision inserts a row with a bumped version (uniqueness on (hazard_type, metric, version) enforces this), the superseded row stays queryable for audit, and no stored metric is touched. The active scheme is derived (highest version per hazard/metric) rather than flagged, so seeding is the whole deployment step. An identical metric value banding differently under v1 and v2 is the intended, auditable behaviour.
Every banded output carries its explainability contract. BandedScore.to_payload() always emits scheme_id, scheme_version, scheme_ref, citation, and the metric/metric_value/metric_unit the band was computed from. A band is therefore always re-derivable and always attributable — the drawer content is a property of the payload shape, not something a caller remembers to add.
Feature flag score_bands_enabled, default False. core/scoring.band_if_enabled() is the only function in the module intended for a serving path; with the flag off it short-circuits and returns None before computing anything. band() and resolve_metric_value() are documented as pure kernels for tests, rollups and offline work — they carry no flag check by design. The flag gates exposure only — it never changes stored data, so flipping it on or off is reversible with no migration.
v1 thresholds are self-set, and say so. Each seeded row's citation states what informed it and notes states what a reviewer should challenge first. Three of four rows are anchored on values already verifiable inside this repository (the Lüthi et al. 2021 wildfire brightness-temperature thresholds and the Klawa & Ulbrich 2003 ~20 m/s windstorm damage threshold seeded by migrations 0005/0006); the tropical-cyclone row is the only one using published boundaries verbatim (Saffir-Simpson categories 1-5 map onto scores 6-10). Nothing claims to be lifted from a source it was not.

Alternatives Considered¶

Store the band on the cell alongside its metrics. One read instead of a read plus a lookup. Rejected: it makes a scheme revision a data migration over every cell ever written, and it destroys the audit story — you could no longer show which scheme produced a historical score. Bands are cheap to recompute; storing them buys nothing and costs the versioning property that makes the catalog defensible.
A single metric_source of "best available" with point→cell fallback. Fewer errors on the serving path. Rejected as exactly the bug A3.3 was filed to prevent: a flood band computed from a hex aggregate when the COG read fails is not a degraded answer, it is a confidently wrong one. Failing loudly is the only option that keeps the number meaningful.
Per-org score schemes (RLS-covered, like impact_functions). Lets a customer tune bands to their own appetite. Rejected for v1: it forfeits cross-org comparability and turns every support conversation into "which scale were you on?". Revisit only if a customer's own scale becomes a procurement requirement — at which point it is an override layer on top of the reference catalog, not a replacement for it.
Gate on external actuarial review before shipping anything. Rejected at RFC ratification: it blocks the whole answer layer on an external dependency with no committed date. The feature flag is the compromise — the code lands and is tested, nothing reaches a user until someone opts in.

Rationale¶

Reusing the impact_functions seeded-catalog pattern means the score catalog inherits an already-proven deployment story (data migration, no admin UI needed) and an already-familiar attribution shape.
Encoding A3.3 as a two-member enum plus an exhaustive hazard map — rather than as a comment or a convention — makes the precedence rule a thing that fails a test when violated, which is the only form of "encoded" worth having. Adding a fifth HazardType now forces a deliberate classification.
Deriving the active scheme from max(version) avoids an is_active flag and the class of bug where two rows are both active, or none is.
Requiring citation at the schema level costs one NOT NULL and removes the possibility of shipping an unattributable band under deadline pressure.

Consequences / Follow-up¶

Rollup semantics are not decided here. The phase-8 8.6 bullet pairs score schemes with asset/portfolio/admin-unit rollups; this ADR covers only the scheme catalog and the banding function (issue #375 scope). Rollup functions, the lookup endpoint (8.8/B10), and any UI are separate items; when they land they should extend rather than restate this ADR.
Nothing calls band_if_enabled() yet — the module ships with no serving caller by design, and the flag stays off. The first caller (the point lookup read path) inherits the contract in decision 5.
Advisory note, on the record: obtain external actuarial/scientific review of the v1 thresholds before the first paid insurer engagement. Until then score_bands_enabled stays off in any environment an external party can see.
Cell-metric schemes name the key they read out of a cell's metrics JSONB (max_wind_gust_ms, max_sustained_wind_ms). The surface writer that populates those cells does not exist yet; when it lands, the metric keys it writes and the keys these schemes read must be reconciled in the same PR.
The RAG tiering and ten-band count are deliberately global constants. If a hazard ever genuinely needs a different granularity, that is a new ADR, not a per-row column — comparability across hazards is the whole point.
What an absent input means is decided separately. This ADR says which input a hazard bands from and refuses to substitute one source for another; it says nothing about a reading that is missing. That is ADR-074: for a count indicator an absent cell is the value zero, so band 1 of those ladders is reachable rather than dead by construction.

ADR-044: Geocoding Adapter + Provider¶

Date: 2026-07-27 Status: Accepted Phase: 8

Context¶

Phase 8.7 introduces the platform's first address-based entry primitive: a caller types a place name and gets back coordinates. Before this ADR, Climate-Lama had no geocoding capability anywhere — no code in core/, no service in either compose file. Two constraints shaped the choice:

Architecture rule 3 (no tight coupling to third-party SDKs): a geocoding provider is exactly the kind of dependency that must sit behind an adapter — a customer or self-hoster may have their own preferred provider, and the backbone should not hard-wire one in.
Data sovereignty for a self-hosted platform. A commercial geocoder (Google, Mapbox, HERE) means every address a user types — potentially the address of an insured asset — leaves the deployment as a third-party API call. For a platform whose pitch to a regional insurer is "your data does not leave your infrastructure," that egress is a hard sell before any pricing conversation even starts.

Decision¶

Self-hosted Nominatim (mediagis/nominatim Docker image) is the v1 provider, with a Greece OSM extract, matching the platform's existing Greece-first showcase scope (ADR-014). Nominatim geocodes entirely against locally-imported OpenStreetMap data — no address ever leaves the deployment.

The interface (`core/geocoding.py`)¶

GeocodingProvider is a Protocol with two methods, geocode(text, *, limit) and reverse(lat, lon), both returning (lists of) GeocodeCandidate — a frozen dataclass of lat, lon, confidence, display_name, low_confidence, place_id. Every caller depends on this interface, never on a provider directly. Two orchestration functions, geocode_address() and reverse_geocode(), sit in front of the interface and own the low-confidence policy (below) — that policy is business logic, so it lives here, not in any adapter.

NominatimGeocoder — also in core/geocoding.py, mirroring how TitilerPointReader (issue #382) lives beside the PointIntensityReader protocol it implements — is the only class in the backbone that imports an HTTP client aimed at a geocoding provider. Swapping in a per-instance Google/Mapbox/HERE adapter later (a real, anticipated need — some self-hosters will prefer a commercial provider's coverage or will not want to run Nominatim's import pipeline) means writing one new class here; no caller changes.

The low-confidence fallback contract¶

A geocode legitimately has no confident answer: a typo, a place outside the imported extract, a genuinely ambiguous query. That is a normal 200, never an error — RFC A3.6's "no answer is not an error" principle, applied to search instead of a hazard card. Every GeocodeCandidate carries its raw confidence (Nominatim's 0..1 importance score) and a low_confidence flag, computed in geocode_address()/reverse_geocode() against Settings.geocoder_low_confidence_threshold (default 0.4) — never by the adapter, so the cutoff is identical regardless of which provider answers. A caller — the UI, later — sees low_confidence: true as its signal to let the user fall back to a manual pin-drop rather than trust the match; the raw confidence is always included too, for a caller that wants its own cutoff.

The provider itself failing is different, and does raise. An unreachable, timed-out, or unparseable Nominatim response — or the geocoder being disabled altogether (Settings.geocoder_enabled = False, the default) — raises GeocodeUnavailableError (E_GEOCODE_UNAVAILABLE, HTTP 503). That status is the explicit, machine-readable signal that geocoding is unavailable right now, as opposed to "no match for this query" — a caller should not retry the same query, it should fall back to pin-drop.

The pin-drop path needs no geocoder¶

GET /v1/risk/lookup (issue #382) takes lat/lon directly and does not import core/geocoding.py or anything from api/v1/geocode.py — verified by a repo-grep test (tests/test_api/test_geocode.py::test_risk_lookup_module_does_not_import_geocoding). A geocoder outage, a disabled geocoder, or the geocoding compose profile never having been started all degrade address search only; a caller who already has a coordinate — the pin-drop UI flow — is entirely unaffected.

The API surface¶

GET /v1/geocode?q=<text>&limit=<1..10> and GET /v1/geocode/reverse?lat&lon, both behind require_role(Role.VIEWER) and a dedicated geocode_rate_limit (30/minute default, separate from login_rate_limit since this proxies a service with real per-request DB cost). api/v1/geocode.py is HTTP-only: it validates the query, rate-limits, and calls core/geocoding.py — it never imports an HTTP client itself.

Deployment: Nominatim in both compose files, opt-in¶

nominatim is added to both docker-compose.yml and docker-compose.prod.yml, gated behind an opt-in geocoding Compose profile (docker compose --profile geocoding up) — the same pattern already used for the monitoring profile. Nothing depends_on it. This was a deliberate call beyond the issue's literal ask: a country-sized OSM import is a real, one-off cost (bandwidth, RAM, wall-clock — see the compose file's comment for the numbers), and phase 8.7 does not yet wire geocoding into anything user-facing (no UI, no Asset geocoding call-site — both are later phase-8 items). Making it mandatory would tax every docker compose up, including the demoable-MVP-slice walk, for a capability nothing else uses yet. GEOCODER_ENABLED/GEOCODER_BASE_URL are still wired into the api service's environment unconditionally in both files, so bringing the profile up is the only step needed to make GET /v1/geocode work — the API side never needs a redeploy.

Alternatives Considered¶

A commercial geocoder (Google Maps, Mapbox, HERE) as the v1 default. Rejected: every query is an address, which is exactly the kind of user data a self-hosted, insurer-facing platform should not be sending to a third party by default. The adapter interface leaves this fully available as a per-instance choice — an operator who prefers it writes one adapter class and reconfigures geocoder_base_url/swaps the class construction, same shape as ADR-036's per-org allowlist model.
A third-party OSM-backed hosted service (Photon, LocationIQ, OpenCage). Rejected for v1: still an egress dependency, and self-hosting the same OSM data Nominatim would use is not meaningfully harder than standing up a proxy in front of someone else's Nominatim. Revisit if operating the import pipeline proves to be a real support burden.
Skip a fallback/low-confidence flag; return raw results only. Rejected: the issue's explicit requirement is that a UI can degrade to pin-drop, which needs a stated, provider-independent signal — a raw importance score is Nominatim-specific and not guaranteed comparable across a future provider.
Make the nominatim compose service mandatory (no profile gate). Rejected per the deployment rationale above — the cost/benefit does not clear the bar yet for something nothing downstream consumes. Revisit when 8.10's UI address-search moment or the Asset geocoding call-site lands and actually depends on it running.

Rationale¶

Putting the Protocol, the orchestration functions, and the concrete adapter in one file (core/geocoding.py) mirrors the TitilerPointReader/PointIntensityReader precedent in core/lookup_service.py exactly — the codebase already has one accepted shape for "a provider-agnostic interface plus its one production adapter," and a second, differently organized shape for the same problem would be a maintenance surprise, not an improvement.
Computing low_confidence in the orchestration layer rather than the adapter keeps the business decision (the cutoff) independent of how any given provider expresses its own confidence — the same discipline ADR-043 applies to score-banding precedence.
An opt-in Compose profile for a one-off, resource-heavy import step has a direct precedent in this same file (monitoring) — reusing an established pattern here costs nothing and keeps docker-compose.yml legible.

Follow-up¶

Wire the UI's address/pin search (phase 8.10, showcase moment 1) once the UI repo issue exists — this ADR ships the backbone half only.
The Asset geocoding call-site (turning an address into a stored geocoded point) is explicitly out of scope here per the issue and lands with the UI slice.
A per-instance commercial-provider adapter (alternative 1) is unscheduled — build one only when an operator actually needs it.

ADR-045: Phase B Ingest Scale — Out-of-Core Aggregation, Grid-Definition Centroids, COG Staging¶

Date: 2026-07-30 Status: Accepted Phase: 9 (Data foundation)

Context¶

436 (`78b0301`, PR #451) replaced the ingest chunk pipeline's dense reads with a sparse-first¶

design and met its ≤1 GiB RSS bar at the JRC river-flood pack's measured 2% occupancy (102.6M-cell Greece clip, 2,054,580 kept cells). #445 (6a84d00, PR #452) closed the gap that bound left open — a dense-occupancy or larger-extent hazard could blow the same budget — by windowing oversized mode-2 files and adding a fail-fast ceiling. Both PRs' own review comments explicitly deferred three structural questions to this ADR rather than solving them:

aggregate_and_commit._union_pixel_indices still builds the dataset-wide union of kept pixel indices with np.union1d(union, idx) per chunk (aggregate_and_commit.py:152) — an O(n log n) full concatenate-and-sort copy every call. #436's own "Out of scope" section named #446 as the destination for "out-of-core aggregation" before #436 was even built; #445's "Out of scope" section named the "Phase-B ADR" again as the real fix behind the interim ceiling; and PR #452 (implementing #445), in review, reconfirmed the item was still untouched: "Out of scope and deliberately untouched: the np.union1d rework in aggregate_and_commit.py (deferred to #446)." Three separate points in the chain named this ADR as the destination without anyone deciding what happens when it arrives — that is the pattern this ADR exists to break.
hazard_centroids stores one Postgres row per nonzero (union) pixel — 2.05M rows for the Greek pack alone, and the row count grows with dataset extent × occupancy, an axis Postgres is not the right tool to bound.
write_chunk's _stream_to_disk (write_chunk.py:360) downloads the entire staged object per chunk before reading a window from it. #445's mode-2 windowing multiplies that cost: at the production 256 MiB river_flood budget (hazard_ingest_chunk_size_bytes_per_haz_type), the bbox-clipped Greece extent (~102.6M cells) plans into ~19–20 windows, each of which re-downloads the whole ~22 GB raw per-RP object, so what was one full-file download becomes ~19. (The raw 5.73B-cell file is not the operative figure here — it never reaches subdivision, because _assert_within_max_cells rejects it against the 1e9 ceiling first.) Filed as #455, whose body names this ADR as the place the COG-staging half of its fix should be decided.

The scale these decisions must hold up against: the full EU JRC mosaic is 5.73B cells — 56× the 102.6M-cell Greece clip that #436/#445 were measured against. Phase 9 (docs/plan/phase-9-data-foundation.md) gates pan-EU full-resolution expansion (#447) on this ADR's out-of-core decision. Its Stream B3 previously carried "out-of-core decisions per

446" as a placeholder; the same change that adds this ADR updates that bullet to point at¶

ADR-045's outcome.

The hazard_ingest_max_grid_cells ceiling #445 shipped (default 1_000_000_000, src/climate_lama/config.py:227) is the interim stance for item 1, and it is worth stating plainly how loose it is. The ceiling bounds grid cells, not kept cells — it fires on width × height alone, deliberately occupancy-independent, before any pixel is read. So its bite depends entirely on occupancy: at the JRC pack's measured ~2% a 1e9-cell grid yields only ~20M kept cells (~160 MB of union), but at dense worst-case occupancy the same ceiling admits 1e9 kept cells — ~7.5 GiB for the uint64 union array alone (1e9 × 8 B), before union1d's transient concatenate/sort copies. #445's own PR description says as much in its "Assumptions worth revisiting" section — the ceiling is "not calibrated to keep every dataset under #436's 1 GiB aggregate bound in the dense-worst-case," and a bound that actually held that line would need to sit around 40–50M cells, which would reject the current JRC ingest outright (~102.6M cells post-clip). The shipped default is roughly 20–25× looser than the number it nominally stands in for — it is a "don't silently OOM before continent scale" tripwire, not a memory guarantee. (For calibration, and to keep this distinct from the ceiling above: #445 separately measured the real per-chunk read working set at 33.8 B/px against a 36 B/px shipped constant, and split the per-haz-type chunk budget 1/4 strip-read / 3/4 kept-triplet-output. That is an already-shipped, already-correct bound on one write_chunk task's own memory — orthogonal to the dataset-wide aggregate ceiling this ADR addresses.)

Decision¶

Each of the three topics below gets an explicit status. None of them are implemented by this ADR — that is out of scope by the issue that produced it (#446); this document records the direction and the follow-up, nothing else executes.

1. Out-of-core aggregation — Decided; implemented 2026-08-05 (#595)¶

Target design: aggregate_and_commit moves from "accumulate the whole union in one process" to a streaming, two-pass, disk-backed aggregation:

Pass 1 (union): instead of repeated np.union1d growing one in-memory array, each chunk's kept-pixel-index set is written to a per-chunk shard file (already sorted, since the sparse reader emits row-major indices per strip); the union is produced by a k-way merge of sorted shard files (bounded working set — one buffered read window per shard, not the whole union) instead of a repeated concatenate-and-sort.
Pass 2 (CSR assembly): the existing per-row triplet emission (_CsrAccumulator) keeps its shape, but accumulates into disk-backed shards instead of process-resident Python lists, with a final on-disk (or streaming) concatenation into the persisted .npz — matching the shape scipy's csr_matrix already expects (data/indices/indptr), so the downstream .npz consumer contract is unchanged.

This removes the "whole-dataset nnz must fit in one process's RAM" ceiling that both np.union1d and the in-memory _CsrAccumulator currently impose, and is what would let a continental-scale (5.73B-cell) mosaic ingest at all.

Interim stance, unchanged by this ADR: the #445 hazard_ingest_max_grid_cells fail-fast ceiling stays the production gate until the streaming rewrite ships. This ADR does not raise, lower, or otherwise touch that config value — changing it is implementation. It only records, per the Context section above, that the shipped value is a loose tripwire, not a tight bound, so a future implementer of this decision should not treat "under 1e9 cells" as "safe."

Follow-up: tracked under Phase 9 Stream B3 (docs/plan/phase-9-data-foundation.md), whose placeholder bullet this change updates to name ADR-045 — this ADR's decision is that input. Not filed as a new standalone issue here: Phase 9 has not yet run /phase-split, and B3 is the right-sized bucket for this work once it does.

Shipped (#595, 2026-08-05) as described above, with two implementation notes worth recording. (a) The spill/merge primitives — SpillDirectory, SortedRun, merge_sorted_runs — live in core/ingest/base.py, not in the chord callback, because a non-GeoTIFF adapter needs the same shape. (b) The .npz writer had to reproduce one thing scipy.sparse.csr_matrix used to do for free: reconciling indices and indptr onto a single index dtype. Streaming the arrays out directly bypasses that reconciliation, so _indptr_dtype re-derives it explicitly — without it the persisted artifact would have silently switched its indptr member from int32 to int64 for every dataset. The #445 hazard_ingest_max_grid_cells ceiling is deliberately left untouched: it still gates the per-chunk read path, which this change does not address.

2. Grid-definition centroid storage — Decided; implemented 2026-08-06 (#596, ADR-057)¶

Target design: replace the per-pixel hazard_centroids row with:

A grid definition (affine transform, width, height, CRS, nodata) stored once per hazard dataset — the same fields #442's conditional grid stamping (grid_transform/grid_width/ grid_height on hazard_datasets) already introduced, generalized to be the primary centroid representation rather than a membership-mode side channel.
A pixel-index array — the uint64 array the sparse pipeline already produces as pixel_idx in the chunk NPZ format — persisted alongside the intensity .npz in MinIO, ordered to match the CSR column ordinals (array_index) exactly as today. (Narrowing it to uint32 below 2³² cells is an open implementation choice, not something that exists today; the only narrowing shipped is _column_index_dtype's int32 for CSR column ordinals, a different array.)
PostgreSQL gains a dataset-level footprint geometry (one row, one geometry, per hazard_datasets row) for spatial dataset discovery ("which datasets cover this point/bbox"), and drops per-centroid geometry. Note this is new schema: HazardDataset has no geometry column today — the only Geometry column in models/hazard.py is HazardCentroid.geometry — so it needs a column plus a migration.

This makes a hazard dataset's Postgres footprint O(1) in row count regardless of pixel count, instead of O(kept cells) — the property that makes a continental mosaic (tens to hundreds of millions of kept cells even at low occupancy) a Postgres-scale problem as much as a memory one.

Cross-repo engine-adapter impact (climate-lama-engine) — spelled out, not hand-waved:

Verified against src/climate_lama/worker/models/engine_adapter.py (the sole file allowed to import climate_lama_engine.*, per ADR-024): ModelInterface.calculate_impact's conversion helpers, _build_hazard/_build_exposures, take flat centroid_lat/centroid_lon arrays (np.float64, shape (n_centroids,)) and a per-exposure-point centroid_idx array of pre-assigned indices into them (engine_adapter.py:176–177, 195) — the module's own docstring states the contract explicitly: the worker passes "pre-processed numpy arrays loaded from PostGIS and MinIO — not raw files or database connections," and "This separation keeps the engine pure (arrays in, arrays out)." The engine has no concept of a grid, a pixel index, or a Postgres row — it is already storage-agnostic arrays-in/arrays-out, the same property ADR-042 relies on for the CLIMADA sidecar. Conclusion: climate-lama-engine's public contract does not need to change. No sub-ADR or engine-repo work item follows from this decision.

What does have to change is entirely on the backbone side of that boundary:

worker/tasks.py (two call sites, tasks.py:409-410 and tasks.py:1097-1098) currently builds centroid_lat/centroid_lon with SELECT lat, lon FROM hazard_centroids ORDER BY array_index. Under grid-definition storage there is no such row to select — the arrays must instead be computed from the grid definition + pixel-index array via the same vectorized affine arithmetic core/ingest/base.py's _pixel_centers already uses at ingest time. This is a rewrite of that call site, not the engine.
core/centroid_assignment.py's cell-membership mode (#442) currently does a SQL LEFT JOIN keyed on hazard_centroids.pixel_index. With no per-pixel Postgres row, that join has nothing to join against. This needs a redesign — e.g., loading the pixel-index array into a session-scoped temp table to keep the SQL join shape, or computing membership in Python via searchsorted against the loaded array (the same technique aggregate_and_commit already uses) and bypassing SQL for this step. This ADR does not choose between those — that choice is implementation — but flags it as the concrete, non-optional redesign item this decision creates. #450 (out-of-grid exposures blocking readiness on a grid-defined dataset) touches the same code path and should be resolved with this redesign in mind, not independently of it.
db/repositories/hazard_repository.py::create_centroids_bulk (the ADR-022 unnest bulk insert) and the array_index-ordered centroid read path lose their reason to exist at per-pixel granularity; they are replaced by a much smaller per-dataset grid-definition write/read plus an object-store array read.

Follow-up: tracked under Phase 9 Stream B3 alongside topic 1 (both touch aggregate_and_commit and the centroid persistence path together); the core/centroid_assignment.py redesign should be split out as its own issue when B3 is scoped, informed by #450.

Shipped (#596, 2026-08-06) — see ADR-057 for what was actually built. Two notes against the target design above. (a) The core/centroid_assignment.py redesign this section flagged as "concrete and non-optional" picked neither of the two candidates verbatim: a dense grid needs no lookup at all (array_index == pixel_index), and only the sparse case joins the pixel-index array, as an unnest(...) WITH ORDINALITY bind rather than a temp table or a Python searchsorted loop. (b) Row deletion in prod is deliberately not part of it — the legacy join stays as the third source, so every un-backfilled dataset keeps working unchanged.

3. COG staging / remote windowed reads — Decided (direction); implementation deferred, tracked in #455¶

Target design: stage ingest sources as Cloud-Optimized GeoTIFFs (internally tiled, with overview pyramids) in MinIO, and switch chunk reads from "download the whole object, then read a window" to ranged/windowed reads against the staged object — a GDAL /vsicurl/-style (or MinIO presigned-URL range-GET) reader that pulls only the bytes covering a chunk's window, proportional to the window size rather than the file size. This directly targets the cost #455 measured: mode-2 windowing turns one full-file download per RP file into ~19 at the production 256 MiB budget, against raw per-RP files #445 itself sizes at ~5.73B cells (~22 GB float32 uncompressed).

Also folded into this decision, per #455's second finding: plan_chunks_for_descriptor now downloads every mode-2 file at plan time solely to read width/height — a new cost #445 introduced (pre-#445 mode-2 planning opened no files at all). The fix is to persist the raster metadata validate_source already computes (width/height/CRS/transform/nodata) onto the source row and have the planner read that instead of re-opening every file at plan time.

Status is explicitly "decided, not designed in implementation detail": this ADR picks ranged/windowed reads over "stage as COG, still download the whole object" as the target shape, and confirms #455 is the right place for it, but does not specify the read-path implementation (e.g., whether GDAL's native /vsicurl///vsis3/ virtual filesystem driver against MinIO is used directly, versus a hand-rolled range-GET reader) — that is implementation.

Follow-up: #455 (already open; its body names #446 — this ADR's originating issue — as the place the COG-staging decision belongs) is the tracked implementation issue for both halves of this decision.

Alternatives Considered¶

Topic 1 (out-of-core aggregation):

Raise hazard_ingest_max_grid_cells and accept the OOM risk it reintroduces above the real bound. Rejected — this is exactly the "roughly 20–25× looser than the bound it stands in for" gap already shipped; raising it further only widens that gap instead of closing it.
Lower the ceiling to the real ~40–50M-cell bound now, as a config-only change. Rejected for this ADR (it is a runtime/config change, out of scope for a docs-only issue) and rejected as sufficient even if implemented — it would reject the current production JRC Greece ingest (~102.6M cells) outright, which is a regression, not a fix.
Streaming, disk-backed two-pass aggregation (chosen direction). Removes the ceiling's reason to exist at the cost of implementation complexity (shard files, k-way merge, disk I/O during aggregation) and some wall-clock overhead versus an all-in-memory union on datasets that fit today. Accepted as the only option that scales to the 5.73B-cell mosaic without an unacceptably strict cell ceiling.

Topic 2 (grid-definition centroids):

Keep one Postgres row per pixel, add partitioning/sharding to cope with row count. Rejected — treats a representation problem (a dense grid does not need per-cell relational rows; it needs an array plus a transform) as a scaling problem, adding operational complexity (partition management, bulk-insert batching beyond the existing ADR-022 pattern) without addressing why the rows exist at pixel granularity in the first place.
Move centroid geometry to PostGIS raster (ST_Value over a stored raster band) instead of a grid definition + pixel-index array in MinIO. Considered — PostGIS raster support exists and would keep centroid lookups SQL-native. Not chosen as the recorded direction because it re-couples centroid storage to Postgres (the exact scaling axis this decision is trying to remove) and duplicates data the ingest pipeline already produces and stores in MinIO as the intensity NPZ's companion grid metadata. Worth revisiting only if the temp-table/Python- membership redesign (topic 2's engine-adapter section) proves harder than expected.
Grid definition + pixel-index array alongside the NPZ, dataset-level footprint only in PG (chosen direction). Matches the representation the sparse ingest pipeline (#436) already produces internally; makes Postgres row count O(datasets) instead of O(kept cells).

Topic 3 (COG staging):

Keep downloading whole objects, tolerate the ~19× multiplier as the cost of windowing. Rejected — #455 measured this as a real, unbounded-with-scale cost (proportional to raw file size / window size), not a one-time tax; it gets worse, not better, as datasets grow toward the 5.73B-cell mosaic this ADR is scoped against.
Pre-slice sources into per-window files at staging time instead of ranged reads against one COG. Rejected — multiplies the object count in MinIO by the chunk count per source, defeats the point of COGs' internal tiling (which already exists to make range reads cheap), and would need re-slicing whenever the chunk budget or CRS-driven window plan changes.
COG staging + ranged/windowed reads (chosen direction). Bytes transferred proportional to the window, not the file; matches the standard cloud-native raster access pattern (titiler/COG already used elsewhere in this stack per Phase 4's COG+titiler work).

Rationale¶

All three decisions are grounded in measurements from #436/#445 and their reviews, not projections: the 56× mosaic-to-clip ratio, the union1d cost flagged twice and never fixed, the shipped ceiling's own PR description admitting it is not a tight bound, and #455's measured ~19× download multiplier.
Deciding direction while deferring implementation is deliberate, not indecision: #446 is docs-only by its own issue scope, and each topic is large enough (memory model, storage schema, or I/O pattern) to warrant its own sized implementation issue rather than being smuggled into this ADR as code.
The engine-adapter analysis for topic 2 is a real finding, not a formality: confirming that climate-lama-engine's contract is unaffected means the grid-definition redesign is fully a backbone-side change, which changes its risk profile (no cross-repo version coordination needed) and where its follow-up issue should live (this repo, not the engine repo).
Naming #450/#453/#454/#455 here rather than re-describing their content keeps this ADR from silently diverging from the issue trackers that already own those specifics.

Follow-up¶

Topics 1 and 2 (out-of-core aggregation, grid-definition centroids): tracked under Phase 9 Stream B3 (docs/plan/phase-9-data-foundation.md), which already lists "out-of-core decisions per #446" and "activates #442" as work items; this ADR is the design input B3 needs before /phase-split sizes it into concrete issues. The core/centroid_assignment.py membership-join redesign should be split out as its own issue, scoped together with #450.
Topic 3 (COG staging): #455, already open, already references this ADR.
Related open issues this ADR does not resolve but is consistent with: #450 (out-of-grid exposures blocking readiness on a grid-defined dataset — touches the same pixel_index membership-join surface topic 2 redesigns, and should be read alongside it when Stream B3 is scoped), #453 (ingest bbox not persisted on the job row — resume re-plans against the raw file), #454 (plan_chunks signature change unsafe across a rolling deploy). #453 and #454 are planner/resume concerns, not centroid-storage ones — they are named here only so this ADR does not read as though the ingest-scale chain's open issues stop at #450/#455.

Related¶

Builds on #436 (PR #451) and #445 (PR #452), whose reviews are this ADR's primary source of measured facts.
Topic 2's engine-adapter conclusion reinforces ADR-024 (no CLIMADA in the backbone) and ADR-042 (CLIMADA sidecar): both rely on the same "engine takes storage-agnostic arrays" property this ADR confirms still holds.
Gates #447 (pan-EU full-resolution expansion), per docs/plan/phase-9-data-foundation.md Decision 1.

ADR-046: Publishing Is Gated on a Green CI Run for the Published Commit¶

Date: 2026-08-01 Status: Accepted Phase: 9 (Data foundation)

Context¶

Three workflows publish artifacts, and none of them checked whether the commit they were publishing had ever been tested:

release.yml — any v* tag push builds and pushes the backbone and worker images to GHCR, moving :latest for non-rc tags.
sdk-publish-test.yml — any sdk-v* tag push publishes to Test PyPI.
sdk-publish-prod.yml — a dispatch publishes to production PyPI.

Under the on-demand CI model (ci.yml and sdk-smoke.yml run only via dispatch or the ci label) trunk commits routinely carry no CI run at all, and CLAUDE.md's instruction to "dispatch ci.yml before tagging" was advice with no mechanism behind it. A tag pushed on a red or never-tested commit published anyway.

This was not hypothetical. When the July 2026 Actions billing lock cleared, the first dispatch against main (23ca0bb) failed on the Test job: the trunk had been red and unpublishable-in- principle for days while remaining fully publishable in practice.

The publish paths are also asymmetrically expensive to get wrong. A PyPI version can never be replaced once published, and GHCR :latest is what deployments pull.

Decision¶

Every publishing job is needs:-gated on a verify-ci job that requires a green CI run keyed on the exact commit being published. One script, scripts/verify_ci_green.sh, is the single implementation; it is called by all three workflows and by the /tag skill locally.

Three properties make the gate meaningful rather than ceremonial:

Job-level verification, not run-level. Every job in ci.yml and sdk-smoke.yml carries an if: guard, so adding any non-ci label to a PR produces a run that tested nothing while not reporting failure. The gate therefore requires named jobs (Lint, Type check, Test, End-to-end smoke) to have concluded success; skipped is not success. The opt-in slow job is deliberately not required.
Fail closed. No run, a run still in progress, or a run missing a required job all block the publish. The error names the exact dispatch commands that fix it.
Enforced in two places. Locally in /tag, which costs zero Actions minutes and prevents the bad tag from ever existing; and server-side in the workflows, so a tag pushed by any other route is still caught. The local check is convenience; the workflow job is the mechanism.

A --pr-fallback mode also accepts a green run keyed on the head SHA of the merged PR that produced the commit, which covers the --review + ci-label flow. It is weaker — that run tested the PR branch, not the merge result — and says so in its output.

Consequences¶

Releasing now requires dispatching ci.yml and sdk-smoke.yml on the release commit and waiting for green. Because runs are keyed by SHA, if main moves between dispatch and tag, the dispatch must be repeated on the new HEAD.
The gate costs ~1 billed minute per publish, against a per-release cost of two dispatches (~10-12 min) that the on-demand model already assumed were being run.
sdk-smoke.yml originally finished green with a notice when smoke credentials were absent (#329) — the live-stack steps were guarded individually, so the End-to-end smoke job passed having run nothing and the gate accepted it. Closed in #470: credential detection is now its own job (Detect smoke credentials) and End-to-end smoke is needs:-gated on its output, so a credential-less run leaves the required job skipped, which is not success. The gate fails closed if the secrets are ever rotated out. Cost: one extra ~1-minute job per smoke dispatch. The general invariant: only require jobs that cannot pass without doing their work.
This ports risk-assessment-tool's verify-tests pattern (its #252) and hardens it with the job-level check.

Related¶

Issue #468; follows the on-demand CI model documented in CLAUDE.md.
Depends on nothing in the compute path — this is release infrastructure only.

ADR-047: The Portfolio Rollup Scores Both Membership Planes¶

Date: 2026-08-03 Status: Accepted Phase: 8 (Foundations / ORSA) — closure of showcase moment 3

Context¶

A portfolio can hold membership rows in two junctions, and ADR-041 established that deliberately:

portfolio_exposures — the original, bulk, dataset-shaped plane. Every CSV/XLSX upload lands here: POST /v1/exposures writes exposures, then one POST /v1/portfolios/{id}/exposures per row.
portfolio_assets — the identity plane added by ADR-041, one row per real-world insured location with a lifecycle.

GET /v1/risk/portfolios/{id} (issue #383) read only the second. Nothing bridged the two, and no API surface creates Asset rows at all — the only Asset(...) construction in the repo is scripts/seed_demo.py.

The consequence was that the product's showcase moment 3 — "upload a portfolio CSV → per-asset scores + rollup" — did not work, and did not say so. Uploading into a new portfolio produced coverage: 0 of 0 with every hazard reporting no_members_to_aggregate; uploading into the seeded demo portfolio was worse, because the rollup returned its five unchanged seeded assets and looked like a success. Production on 2026-08-02 measured assets = 5, portfolio_assets = 5, exposures = 7631.

Two directions were on the table (issue #550):

(a) Asset-creating upload. The CSV upload also writes Asset + portfolio_assets; the asset plane becomes the scoreable primitive.
(b) Exposure-aware rollup. The rollup aggregates over the union of both planes.

Decision¶

(b). The portfolio rollup's scoreable membership is the union of portfolio_assets and portfolio_exposures. Both planes carry the only two things this rung needs — a point and a value — and neither is privileged in the aggregate.

Four properties make it honest rather than merely working:

Every scored member names its plane. worst_members[].member_kind is asset or exposure. The ids come from disjoint spaces, so only an asset member resolves against GET /v1/risk/assets/{id}. asset_id is retained as a compatibility alias of member_id for pre-#550 clients.
Coverage counts exposure members. coverage.considered is the union's size, and subject gains asset_member_count / exposure_member_count so a caller can tell an all-asset book from an all-upload one.
The cap is a budget over the union, not per plane. rollup_max_portfolio_members is spent on the asset plane first and the exposure plane fills the remainder, so the endpoint's worst case is unchanged. Membership costs four queries (two counts, two listings) regardless of portfolio size, and the per-hazard cell reads stay batched — the "query count is flat in member count" invariant survives.
Nothing is precomputed and nothing is written. This is a read-path change only; the two junctions stay separate tables and no migration was needed.

value_weighted.weighting changes from asset_value_times_membership_weight to member_value_times_membership_weight, because it no longer weights only assets.

Alternatives Considered¶

(a) Asset-creating upload. Rejected for now, not on the merits. It is the direction of Phase 9 Stream A (exposure_datasets as a first-class entity) and it is the better end state: one scoreable primitive, identity-shaped, with a lifecycle. But it is a write-path change to the product's most-used endpoint, it needs a policy for what an Asset generated from an anonymous CSV row even is (address is NOT NULL and is personal data under ADR-041), and it leaves every already-uploaded row — 7631 of them in production — still invisible until backfilled. Choosing it now would have deferred a working moment 3 behind a data-model decision that is not ready.

Score exposures through a point lookup per member. Rejected: one COG read per member on a serving path is precisely the unbounded synchronous work the architecture rules forbid, and it would have broken the flat-query-count invariant #383 asserts by test. The existing cell-batch path already scores from cell metrics, so exposure members ride it unchanged.

Cap + async precompute. Considered and not needed. The membership union adds two queries, not N; the existing cap already bounds the scored set. Precompute would be real work with no problem to solve yet.

Leave it and document the gap. Rejected: the seeded-portfolio case renders a plausible number computed from members the user did not upload, which is worse than an error.

Rationale¶

(b) is the smaller change and the one that makes the showcase demonstrable soonest — an explicit trade the orchestrator made for Phase 8 closure.
It is strictly additive on the wire: existing keys keep their meaning and asset_id still resolves for asset members.
It does not foreclose (a). When the asset plane becomes the primitive, this rung keeps working — an all-asset portfolio is just the union with an empty exposure side.
The alternative to unioning is pretending: a portfolio whose only membership is uploaded rows had a real risk profile and the API reported it as empty.

Follow-up¶

Issue #550 (this decision). Option (a) — asset-creating upload / first-class exposure_datasets — remains Phase 9 Stream A direction.
The Asset plane still has no creating API surface; that gap is what makes the union necessary rather than merely convenient.

ADR-048: Dataset Identity & Catalog Contract¶

Date: 2026-08-05 Status: Accepted Phase: 9 (Data foundation) — Stream A1

Context¶

Phase 9's catalog work — precedence between overlapping datasets (Stream A3A), the scenario-matrix resolver's provider-family pinning (Stream A3B-1/A3B-2), and the catalog.v1.json manifest producer (Stream A2) — all need full dataset identity before they can be built. hazard_datasets only partially carries it: source/license/citation/source_url/region/scenario/return_periods/supported_years plus the #442 grid-definition block. Missing: a machine-readable provider id, the upstream release version, the source data's native resolution, its native spatial extent, the temporal coverage of the upstream release, GCM/ensemble provenance for projected datasets, a data_type sub-classification (fluvial/pluvial/coastal, ...), and a catalog lifecycle status (active/superseded).

exposure_datasets does not exist yet as a table (exposures are rows today, not a dataset-level entity — see the 0029 migration comment referenced from ADR-030's era), so the identity contract has to be written once, for both halves, even though only the hazard half ships schema changes in this issue. The exposure half is implemented separately in Stream A1A-E against a new exposure_datasets table using this same contract.

Decision¶

One shared identity contract, split across two migrations. Both hazard_datasets and (later) exposure_datasets carry the same ten identity columns, all nullable with no backfill:

Column	Type	Meaning
`provider`	string	Machine-readable provider id (e.g. `"jrc"`, `"aqueduct"`) — distinct from the free-text `source` attribution label already on the table.
`upstream_version`	string	The provider's own release/version tag (e.g. `"v1.2"`).
`native_resolution`	float	The source data's native pixel resolution, degrees/pixel — distinct from `IngestRequest.resolution`, which is a target downsample applied at ingest read time.
`bbox`	JSONB	The dataset's native spatial extent `[min_lon, min_lat, max_lon, max_lat]` (EPSG:4326) — distinct from the ingest-time clip bbox already persisted on `ingest_jobs` (#453); mirrors that column's "verbatim float list, never queried spatially" design.
`temporal_coverage_start` / `temporal_coverage_end`	int	The inclusive year range the upstream release covers — distinct from `supported_years`, which lists only the discrete years actually ingested.
`gcm` / `ensemble`	string	Climate model and ensemble member driving a projected (non-baseline) dataset; NULL for observational/baseline data.
`data_type`	string	Hazard (or exposure) sub-classification within one `haz_type`, e.g. `"fluvial"` / `"pluvial"` / `"coastal"` for river flood.
`status`	string	Catalog lifecycle state, `"active"` / `"superseded"`. NULL is treated as active by any reader — this issue does not enforce the vocabulary at the DB level (no `CHECK`/enum), keeping the column cheap to widen later.

Three of these — provider, native_resolution, and status — are called out because Streams A3A/A3B-1/A3B-2 read them directly: provider pins the matrix resolver's provider family across scenario cells (a JRC baseline must resolve to JRC siblings, never silently substitute an Aqueduct cell), native_resolution feeds the ThinkHazard-style precedence rule (quality score → local-over-global → recency) when two datasets cover the same cell, and status lets a superseded dataset stay queryable for historical results without being offered to new selections.

This issue (#583) ships the hazard-side migration and ORM/schema updates only. exposure_datasets and its identity columns are Stream A1A-E's migration, against this same column contract — so the two tables stay structurally identical for anything that later needs to treat them uniformly (e.g. a shared precedence resolver).

Note (#900, 2026-08-14): what provider actually carries, and who reconciles it. The slugs above ("jrc", "aqueduct") were illustrative and are not what shipped — every producer in this repo stamps a human-readable producer name, and so do all of production's active rows. When #887 gave the approve→ingest handoff its own derivation of this column, one upstream distribution started landing under two different provider / upstream_version pairs depending on the path that ingested it. ADR-072 records the canonical value and the per-source override table that reconciles the two paths.

Alternatives Considered¶

A single wide dataset_identity table, foreign-keyed from both hazard_datasets and a future exposure_datasets. Rejected: normalizing identity out to its own table buys nothing here (identity is 1:1 with a dataset, never shared across datasets) and adds a join to every precedence-relevant read the whole point of Stream A3A/A3B is to make cheap.

A Postgres ENUM for status. Rejected for this issue: HazardType already demonstrates the enum pattern (hazard_type_enum) when a fixed, enforced vocabulary is wanted, but status's vocabulary (active/superseded, possibly more states as later precedence work lands) isn't settled yet, and altering a Postgres enum type is a heavier migration than widening a String column's app-level validation. Revisit once Stream A3A's precedence logic settles on the final state list.

Populate the columns from existing rows at migration time (e.g. source → provider, region → derived bbox). Rejected: source is a free-text attribution label (e.g. "JRC river flood dataset"), not a stable machine id, and there is no reliable programmatic mapping from it to a provider slug or a native bbox without re-reading each dataset's original files. Leaving the columns NULL for legacy rows is honest; a follow-up backfill (out of scope here) can populate them per-provider once A2's manifest writers exist to source the values from.

Follow-up¶

Stream A1A-E: exposure_datasets table + this same identity contract for exposures.
Stream A3A: precedence resolver reading provider/native_resolution/status.
Stream A3B-1/A3B-2: scenario-matrix resolver's provider-family pinning.
Stream D2: expose these fields over GET /v1/hazards (the HazardDatasetIdentity Pydantic schema added in #583 is the schema home for that work; not wired into the endpoint yet).

ADR-049: Centralized Object-Store Key Construction¶

Date: 2026-08-05 Status: Accepted Phase: 9 (Data foundation) — Stream A2 (storage design, points 3-4)

Context¶

ADR-029 froze the bucket layout for raw/, processed/, tiles/, and reports/ in Phase 3. Since then, ingest and compute picked up four more top-level prefixes that ADR-029 never documented — staging/, chunks/, hazards/, and results/ — and every one of them was constructed independently, as an inline f-string, at its own call site:

staging/{ingest_job_id}/{basename} — worker/ingest/stage_source.py
chunks/{ingest_job_id}/chunk-{chunk_idx:06d}.npz — worker/ingest/write_chunk.py and, separately, worker/ingest/aggregate_and_commit.py (which has to rebuild the same key to read every chunk back during the chord callback)
hazards/{dataset_id}/intensity.npz — worker/ingest/aggregate_and_commit.py (v2, sparse path) and, separately, core/ingest/base.py (v1, dense path)
results/{job_id}/eai_exp.npz — worker/tasks.py
processed/{org_id_or_public}/{dataset_id}/hazard.tif — worker/tasks.py
tiles/{dataset_id}/{z}/{x}/{y}.{ext} and tiles/{dataset_id}/ — core/tile_cache.py

Nothing enforced agreement between the two independent chunks/ and hazards/ implementations — they happened to match, but a future edit to either one alone would have silently forked the layout, orphaning objects the other side could no longer find. Stream A2A-2's manifest writers (follow-up work) need one place to source every key template from before they can be written correctly, and ADR-029 explicitly does not cover this — it defines raw/, processed/, tiles/, reports/ only.

Decision¶

One module, core/storage_keys.py, owns every key template used by the ingest and compute paths. It is pure string construction (no I/O, no MinIO import) so every function is a one-line unit test. Call sites that previously built keys inline now call into it; call sites that already had a locally-named helper (_staging_key, _chunk_npz_key, _processed_cog_key, TileCache.cache_key / .dataset_prefix) keep that name as a thin delegate, so existing imports and tests are unaffected — only the string-building logic moved.

This is a pure refactor: no key shape changed. Every key storage_keys.py emits is byte-for-byte identical to what the scattered call sites produced before, pinned by tests/test_core/test_storage_keys.py. No stored object is orphaned or needs a backfill.

The layout is now legalized in full, extending ADR-029's table:

{bucket}/
├── raw/            {source}/{data-type}/{version}/{dataset-id}/...   (ADR-029, unchanged)
├── staging/        {ingest_job_id}/{basename}
├── chunks/         {ingest_job_id}/chunk-{chunk_idx:06d}.npz
├── hazards/        {dataset_id}/intensity.npz  (+ source_manifest.json — ADR-066)
├── results/        {job_id}/eai_exp.npz
├── processed/      {org_id_or_public}/{dataset_id}/hazard.tif        (ADR-029, unchanged)
├── tiles/          {dataset_id}/b{bidx}/{z}/{x}/{y}.{ext}            (band segment: #831)
└── reports/        {org_id}/{subject_id}/{template}/...              (ADR-029, unchanged)

staging/, chunks/, hazards/, and results/ are intermediate/derived prefixes with no manifest.json obligation — ADR-029 invariant 2 (mandatory manifest at every leaf) applies only to the four prefixes it already governs. staging/ and chunks/ are working storage: cleaned up on success, retained on failure for forensic re-runs (see aggregate_and_commit.py).

reports/ keys are intentionally not in storage_keys.py — core/reports/storage.py already owns that prefix and pairs its key functions with the manifest-write-last I/O the ADR-029 invariant requires there; folding it in here would separate the key from the invariant that governs it.

One known pre-existing exception, left alone. worker/download_task.py / api/v1/admin/downloads.py write download-cache objects under a different raw/ shape (raw/{hostname}/{download_id}/{filename}, not ADR-029's raw/{source}/{data-type}/{version}/ {dataset-id}/). That predates this ADR, addresses a different concern (an opaque URL-fetch cache, not catalog-joined source data), and its own call site was not part of this refactor's scope. Flagging it here so a future storage-layout audit does not mistake it for a second, undocumented interpretation of ADR-029's raw/ prefix.

Alternatives Considered¶

Fold key construction into each key's own I/O module (e.g. a chunk_io.py next to write_chunk.py that owns both the key and the get/put calls), mirroring core/reports/storage.py. Rejected for the ingest/compute prefixes: chunks/ and hazards/ are each already written and read from two separate modules (write_chunk.py + aggregate_and_commit.py; aggregate_and_commit.py + core/ingest/base.py), so there is no single natural I/O owner to attach the key function to. A dedicated key-only module has one home per template regardless of how many readers/writers exist.

Rename the call sites' local helpers away instead of keeping them as delegates. Rejected: _chunk_npz_key and _processed_cog_key are imported directly by existing tests (tests/worker/ingest/test_end_to_end_mode1.py, tests/test_worker/test_convert_to_cog.py, others). Keeping them as one-line delegates centralizes the actual string logic without an unrelated test-churn diff.

Follow-up¶

Stream A2A-2: manifest writers for staging/, chunks/, hazards/, results/ build on storage_keys.py as their key source. Landed (issue #587): core/manifest.py + storage_keys.staging_manifest_key/.hazard_manifest_key, wired into aggregate_and_commit.py's commit step — the staging leaf (standing in for raw/ until the key-layout reconciliation below lands) and the hazards/{dataset_id}/ leaf each get a manifest.json (source, license, attribution, checksum, version, volume_bytes, timestamp). Backfilling manifests for objects ingested before #587 is explicitly out of scope — deferred to whichever follow-up executes the key-layout reconciliation. Amended by ADR-066 (#784): the source manifest moved off the staging leaf to hazards/{dataset_id}/source_manifest.json, because leaving a record of permanent value inside the prefix this ADR calls disposable working storage is what stopped the success path from emptying it. staging/ now holds nothing of record.
A future storage-layout audit should reconcile the download_task.py / downloads.py raw/ exception noted above — either fold it into storage_keys.py under a distinctly-named function, or document it as a permanently separate cache namespace.

ADR-050: Ingest Format Adapter ABC (ADR-027 Sub-ADR)¶

Date: 2026-08-05 Status: Accepted Phase: 9

Context¶

ADR-027 named "ingest formats" as one of five extension points needing a formal contract and recorded its status as "No ABC yet; concrete ingestors share a common pattern but lack a formal interface. Needs ABC," with a follow-up to land the ABC as its own sub-ADR. This is that sub-ADR.

The async hazard-ingest chord (stage_source → validate_source → plan_chunks → write_chunk × N → aggregate_and_commit, composed in worker/ingest/pipeline.py) only ever had one format to serve — GeoTIFF/COG — so its four preface/write tasks called GeoTIFF's reader functions (plan_chunks_for_descriptor, the per-chunk sparse-read helper, stage_local_sources) directly. That was safe while GeoTIFF was the only format, but it meant a second format (NetCDF, HDF5 — both named in docs/CLIMATE_LAMA.md's roadmap) could not register without editing chord internals: there was no seam to plug into, only a hardcoded call graph.

Decision¶

IngestAdapter, a new ABC in core/ingest/base.py, formalises the seam that already existed informally. It has four static methods, matching the four questions the chord already asked of any staged source, in the order it asks them: stage, validate, plan_chunks, read_chunk. Every method's signature is the signature a pre-existing GeoTIFF function already had — the ABC adds no new behaviour, it names an existing pattern.

GeoTIFFIngestAdapter (worker/ingest/adapters.py) is the concrete implementation: each method is a one-line delegate to the free function of the same name that already existed (stage_local_sources, validate_staged_source, plan_chunks_for_descriptor, the per-chunk sparse-read helper in write_chunk.py). None of those functions moved or changed — this is a pure refactor of how they're called, not of what they do, which is what keeps the change behavior-neutral and every existing ingest test passing unchanged.

Two-key registry — peril and format both resolve. BaseGeoTIFFIngestor (the existing peril-keyed base class every hazard-specific ingestor extends) gained a format_name: ClassVar[str] = "geotiff" class attribute. worker/ingest/adapters.py::resolve_ingest_adapter(haz_type) chains the two registries: HazardType → the registered BaseGeoTIFFIngestor subclass (core.ingest.INGESTOR_REGISTRY, peril-keyed, pre-existing) → that class's format_name → the concrete IngestAdapter (INGEST_ADAPTER_REGISTRY, format-keyed, new). A hazard type with no registered ingestor falls back to the sole registered format rather than raising — chunk planning doesn't require a peril-specific ingestor to exist, only the (separately-resolved) event_name hook does — mirroring the same tolerant .get() lookup assert_ingest_unit_compatible already uses against the identical peril registry.

Where each adapter lives, and why. The ABC lives in core/ingest/base.py: it is a pure interface, no I/O. The concrete GeoTIFFIngestAdapter and the registry live in the worker layer (worker/ingest/adapters.py), not in core/ingest/, because a concrete adapter's methods stream through MinIO and open Celery-worker-local staged files — worker responsibilities per this project's layering (core/ never depends on worker/). GeoTIFFIngestAdapter's method bodies import their target functions lazily, at call time rather than at module load, because plan_chunks.py / write_chunk.py / stage_source.py all import adapters.py at module scope to resolve an adapter; importing any of them back at module scope from adapters.py would cycle. This is the same lazy-import idiom the ingest chord already used elsewhere (e.g. stage_source._close_legacy_job's import of aggregate_and_commit).

Chord call sites now resolve through the contract:

plan_chunks.py's task and pipeline.py's resume path (_select_chunks_for_resume_impl) both resolve an adapter via resolve_ingest_adapter(haz_type) and call .plan_chunks(...) — both had haz_type in scope already.
write_chunk.py's task gained an optional haz_type: str | None = None parameter (both write_chunk.si(...) call sites in pipeline.py now pass it as a keyword, which doesn't disturb the positional-argument tests that pin the chord's dispatch contract) and resolves through the same function; omitted, it falls back to the sole registered format, so a message serialised before this parameter existed keeps working across a rolling deploy.
stage_source.py's task calls GeoTIFFIngestAdapter.stage(...) directly rather than through resolve_ingest_adapter — staging has no haz_type in scope, and is already fully format-agnostic (copy bytes, hash them; no raster parsing at all), so there is nothing format-specific left for the resolution step to remove.
aggregate_and_commit.py is unchanged: its only ingestor-registry use (INGESTOR_REGISTRY[haz_type] for the per-hazard event_name hook) was already going through the peril registry, not calling a GeoTIFF reader directly, and the NPZ/grid serialization it consumes (grid_from_npz, grid_to_npz_kwargs, serialize_sparse_chunk's wire format) is the format-neutral contract any future adapter's read_chunk must also produce.

A second format's write_chunk dispatch is not fully solved by this ADR. write_chunk today resolves by haz_type, and every haz_type in INGESTOR_REGISTRY maps to GeoTIFF. A genuinely second format would need a chunk to carry its own format marker (plan_chunks is the natural place to stamp one) so write_chunk doesn't have to assume "GeoTIFF" the moment a hazard's registered format changes underneath it. That is deliberately out of scope here — it has no consumer until a second format lands (out of scope for this issue; see its "Out of scope" section) — and is called out explicitly so it isn't mistaken for solved.

Alternatives Considered¶

Relocate the actual read/plan logic (window subdivision math, sparse-read strip budgeting) into core/ingest/base.py behind the adapter, instead of leaving it in worker/ingest/*.py and delegating to it. Rejected for this pass: that logic is deeply optimized, numerically sensitive (ADR-045's out-of-core aggregation, #436's sparse-first rewrite, #445's windowing) and tested at exactly its current location. Moving it would touch far more surface than the contract itself needs to land safely in one change and risks a behavior regression in code with no local Postgres to verify against in this environment. The thin-delegate adapter gets the same extensibility property (a new format registers without touching chord internals) without moving a single line of the existing numerical implementation.

Stamp every chunk with its format at plan time now, so write_chunk never needs the haz_type fallback. Deferred to whichever issue adds the second format: doing it now would add a key to the chunk dict that every existing test asserting exact chunk shape would need to absorb, for a capability with no consumer yet. Threading haz_type through write_chunk.si(...) instead gets the same peril → format resolution live today, additively (a new optional keyword argument), with no wire-shape change to the chunk dict itself.

Rationale¶

Landing the interface without moving the implementation is the lowest-risk way to satisfy ADR-027's follow-up: the chord's behavior — every existing ingest test's assertions — is provably unchanged because the functions those tests exercise directly were never touched.
The two-key registry (peril → format) reuses a pattern this codebase already trusts (INGESTOR_REGISTRY plus a tolerant .get() lookup) rather than inventing a new one.
Keeping the concrete adapter in the worker layer, not core.ingest, keeps the project's "core/ never depends on worker/" rule intact rather than special-casing it for this ABC.

Follow-up¶

The next format adapter (NetCDF/HDF5, tracked as B2A per the Phase 9 data-foundation plan) is the first real test of this contract: it should need no chord-internal changes beyond a new IngestAdapter implementation and a registry entry.
Give write_chunk a real per-chunk format signal (see "not fully solved" above) when that second format lands, instead of leaning on the haz_type fallback indefinitely.

ADR-051: NetCDF Ingest Format (ADR-050 Follow-up)¶

Date: 2026-08-05 Status: Accepted Phase: 9

Context¶

Every CDS/EWDS dataset in the Phase 9 data foundation — wildfire FWI, windstorm footprints, heatwave, drought — ships as NetCDF, and the backbone could read none of them: rasterio was the only raster dependency and no NetCDF code existed anywhere under src/climate_lama/. ADR-050 had just landed the IngestAdapter ABC precisely so a second source format could register without editing chord internals, and named NetCDF as the first real test of that contract. Phase 9 ratified decision 4 explicitly allows an xarray dependency for reading NetCDF/Zarr sources at ingest; storage stays npz+COG and Zarr adoption is deferred.

Decision¶

A NetCDF adapter, split along the layering ADR-050 established. core/ingest/netcdf.py owns every pure, local-file operation — declaring a product's layout, deriving the ingest grid from its coordinate variables, planning chunks, reading one chunk's kept cells into a SparseRasterRead. worker/ingest/adapters.py::NetCDFIngestAdapter owns the worker half: pulling the staged object onto local disk and resolving the per-haz-type byte budgets. This is the same split core/ingest/base.py already had for GeoTIFF, where rasterio reads of a local path are core and MinIO is not. stage is shared with GeoTIFF outright, because staging is a byte copy that has never parsed the source.

ADR-050's success criterion held. Registering the format is one IngestAdapter implementation plus one INGEST_ADAPTER_REGISTRY entry. No chord task's logic changed.

Layout is declared, never sniffed (NetCDFDatasetSpec): the variable to read, the names of its x/y dimensions, the CRS its coordinates are in, and — for a CDS-shaped projection file — the scenario and time dimensions its non-spatial axis is built from. A CDS download carries no grid_mapping variable at all, so a declared CRS is the only way to know what its latitude/longitude coordinates mean; when a file does declare one, it is checked against the registration rather than silently overriding it. NETCDF_DATASET_REGISTRY ships empty: a spec pins a real product's variable and dimension names, which is knowledge the pack that ingests it owns (Stream B4B/E), not knowledge this module can invent.

The event axis is scenario × year, not return period. One NetCDF file holds every slice, so chunks fan out over slices and row-windows of one file, rather than over one file per event as GeoTIFF mode 2 does. Each slice's ordinal is the event_offset every one of its row-chunks reports, so several windows of one scenario/year still aggregate onto a single intensity-matrix row — the same invariant mode 2 maintains for a return period. Declared supported_years must be present in the file in full, because that list is what reaches HazardDataset.supported_years: a partially-present declaration would advertise horizons the dataset cannot serve.

write_chunk now dispatches on a per-chunk format marker. ADR-050 flagged this as "not fully solved" and deferred it explicitly until a second format landed. It has landed, so plan_netcdf_chunks stamps format: "netcdf" on every chunk and resolve_adapter_for_chunk(chunk, haz_type) prefers that marker, falling back to the peril chain and then to DEFAULT_FORMAT. An unmarked chunk resolves exactly as it did before, so a message serialised pre-#593 keeps working; a chunk naming an unregistered format raises rather than being read by the wrong reader.

Deliberate limits, each a fail-fast rather than a silent approximation: EPSG:4326 only (every downstream contract is lon/lat); ascending longitude within [-180, 180] (an ERA5-style 0..360 grid needs a roll that would break the contiguity every windowed read relies on); regular grids only (curvilinear and rotated-pole are rejected); native resolution only (GeoTIFF's resolution down-sampling has no NetCDF equivalent here). Each rejection names the conversion or the registration change that would fix it.

Dependencies land in the worker extra, not the base install: NetCDF is only ever read at ingest time, which is chord work, and xarray drags pandas in — bloat the API image does not need. core/ingest/netcdf.py therefore imports xarray at call time, the same lazy idiom core/ingest/base.py uses for rasterio, so core stays importable without the extra.

Alternatives Considered¶

Put xarray/netCDF4 in the base dependencies, next to rasterio. Rejected: rasterio is used by API-path code, NetCDF reading is not, and the transitive pandas install is a real cost for an image that would never call it. The lazy import makes the worker-extra placement safe rather than merely cheaper.

Normalise 0..360 longitude grids by rolling them at read time. Rejected for this pass: a roll makes the read non-contiguous, which is exactly the property every windowed/stripped read depends on for its memory bound. Rejecting with a message naming the conversion is honest; silently rolling would trade a clear failure for an unbounded one.

Reuse GeoTIFF's integer mode field for NetCDF chunks. Rejected: mode 1/2 are GeoTIFF's own single-file/per-RP layouts and _read_chunk_pixels raises on anything else. A NetCDF chunk carries a format marker instead, so a mis-routed chunk fails loudly rather than being misinterpreted as a mode it does not implement.

Rationale¶

The format/peril split stays clean: this issue registers a format, not a peril. No HazardType maps to NetCDF yet, so nothing in production changes until a pack wires one.
Declaring layout per registration (rather than sniffing dimension names) means a source whose layout drifts fails validation instead of being ingested against the wrong axis.
Closing ADR-050's deferred per-chunk format signal now — while exactly one format could possibly be affected — is far cheaper than closing it once several perils resolve to different formats.

Follow-up¶

Register concrete product specs (variable/dimension names) as the CDS/EWDS packs land (Stream B4B/E); the registry is intentionally empty until then.
Revisit 0..360 longitude support if a target CDS product turns out to ship that convention.
Reprojecting NetCDF reads (non-4326 sources) if a target product is ever distributed projected.

ADR-054: catalog.v1.json Producer — Optional Identity, Deterministic Keys¶

Date: 2026-08-05 Status: Accepted Phase: 9 (Data foundation) — Stream A2

Context¶

The /v1/admin/catalog reader (core.catalog.CatalogClient, issue #320) and its admin routes have existed since issue #320 and are fully tested, but nothing produces the catalog.v1.json manifest they read — every deployment 404s. Issue #588 adds the producer: a Celery task that builds the manifest from hazard_datasets + exposure_datasets (ADR-048 identity, issues

583/#584) instead of a manually curated pre-ingest list.¶

Two problems fall out of that source-of-truth switch:

The original issue-#320 CatalogEntry schema requires region, year and sha256 on every entry. exposure_datasets (ADR-048, issue #584) has no columns for any of the three — a structural gap, not missing data — and hazard_datasets leaves its ADR-048 identity columns nullable with no backfill (ADR-048's explicit decision). Keeping these three required would mean the producer could never emit a single exposure entry, and would silently drop every un-backfilled legacy hazard row.
manifest_url (and, for a not-yet-COG'd hazard dataset or any exposure dataset, s3_url) must point somewhere, but neither table persists the raw source-file path ADR-029's raw/{source}/{data-type}/{version}/{dataset-id}/ layout describes — only the processed artifact (cog_path/npz_path), and exposure_datasets has no per-dataset object key at all (exposure data lives in Postgres exposures rows, not a blob).

Decision¶

Loosen exactly the three structurally-absent fields to optional; keep the rest of the issue-#320 contract as a hard requirement, gating entries on it. CatalogEntry.region, .year and .sha256 become | None. name, type, source, license, s3_url and manifest_url stay required — the producer skips (logs a warning, does not fabricate) any dataset missing provider/source or license, since an unattributed entry is not something the catalog's request-ingest flow should offer. The ten ADR-048 columns are added to CatalogEntry as new optional fields, additive and backward-compatible with the existing issue-#320 fixture.

Deterministic ADR-029 keys, real artifacts preferred. Both manifest_url and s3_url are built from raw/{provider-or-source}/{data-type}/{upstream_version-or-"v1"}/{dataset-id}/... — always derivable, so every dataset that passes the attribution gate gets a valid entry regardless of ingest/COG state. For a hazard dataset, s3_url prefers the real processed artifact (cog_path, else npz_path) over the synthetic path, since that is genuinely where its data sits; a not-yet-COG'd hazard dataset and every exposure dataset (no artifact column at all) fall back to the synthetic key. manifest_url for a dataset without a real ADR-029 raw/.../manifest.json (issue #587, the sibling manifest-writer, has not landed for it) still resolves deterministically — the catalog listing does not require the leaf manifest to exist; only GET /v1/admin/catalog/{name}/manifest does, and it already surfaces a 404 for a missing manifest via CatalogUnavailableError("missing", ...).

Regeneration: best-effort enqueue from worker.tasks.convert_to_cog/build_dataset_cog once a hazard dataset's cog_path lands (the closest existing "this dataset just became catalog-worthy" signal without reaching into the ingest chord itself), plus a nightly Celery beat sweep as the backstop — the only mechanism that covers exposure datasets, which have no analogous per-dataset processing step to hook. Any registered Celery task is already operator-dispatchable on demand; no new HTTP route was added (out of scope per issue #588).

Alternatives Considered¶

Backfill region/year/sha256 with heuristics (e.g. region from bbox, year from created_at) instead of leaving them optional. Rejected for the same reason ADR-048 rejected backfilling its own columns: a derived guess presented as the real attribution field is worse than an honest null — a future precedence/UI reader cannot tell "value is null" from "we guessed and got it wrong."

Persist a raw_key column on both tables instead of deriving a synthetic path. Rejected as out of scope for issue #588 (Alembic revisions are centrally assigned per the phase-9 build plan, and this issue was not allocated one) — the deterministic derivation needs no schema change and is sufficient for the catalog listing to be correct; it does not claim the synthetic key names an object that already exists in storage.

Skip exposure datasets from the catalog entirely (only emit hazard entries) rather than loosen the schema. Rejected: the issue's explicit grounding is "generating catalog.v1.json from hazard_datasets + exposure_datasets" — omitting one table unconditionally would not satisfy that, whereas loosening exactly the fields the table structurally lacks lets exposure entries appear whenever they clear the (unchanged) attribution gate.

Follow-up¶

Issue #587 (manifest writers): once it lands, manifest_url for real datasets will actually resolve to a written manifest.json; this producer's deterministic path already matches that target layout, so no change is needed here when #587 ships.
A precedence/backfill pass (Stream A3A) may eventually populate provider/region/etc. on legacy rows, at which point more entries pass the attribution gate — no schema change needed.
Wiring an exposure-dataset commit hook (mirroring convert_to_cog's enqueue) if the nightly sweep's latency proves too coarse in practice.

ADR-055: Exposure Ingest Is a Sibling Contract, Not a Hazard Adapter¶

Date: 2026-08-05 Status: Accepted Phase: 9

Context¶

ADR-050 landed IngestAdapter, the format seam the async hazard ingest chord dispatches through, and ADR-051 registered NetCDF against it. Exposure ingest had no equivalent: the one raster→exposure path in the codebase lived inline in scripts/ingest_historical_catalog.py's ingest-exposure subcommand — a windowed read of the GHSL built-up-volume grid, an in-Python block aggregation, a rendered CSV, and a call into the CSV upload service. Three consequences:

Not registered. A second exposure raster (WorldPop population, a national asset-value grid) meant a second script, not a registration.
Not streamed. src.read(1, window=window) materialised the whole Greece window before aggregating, then built a CSV of every point in memory, then handed those bytes to a parser with a 50 MB ceiling. The GHSL pack survives that at ~7 km effective resolution; a 100 m grid does not. This is the RISK WISE monolithic-load failure mode the project bans.
Identity by accident. #584 gave exposure_datasets the ADR-048 identity columns, but the script path could only fill the four attribution ones — native_resolution and bbox were left NULL even though the raster it just read knows both.

Decision¶

Exposure ingest gets its own ABC, ExposureIngestAdapter (core/ingest/exposure.py), as a sibling of IngestAdapter rather than an implementation of it. The two problems are not the same shape:

	hazard (`IngestAdapter`)	exposure (`ExposureIngestAdapter`)
execution	async Celery chord, fanned out, resumable	synchronous, inside the caller's DB session
verbs	`stage` / `validate` / `plan_chunks` / `read_chunk`	`describe` / `iter_batches`
keyed by	`HazardType` + return periods	nothing peril-shaped
output	sparse intensity matrix + centroids	point rows + one catalog row

Forcing exposure through the hazard contract would mean either stubbing four chord verbs on every exposure adapter (stage into object storage that exposure never uses, a plan_chunks whose plan nothing resumes) or widening the hazard signatures with exposure-only optionals. Two honest interfaces that share vocabulary — format_name, a declared-not-guessed layout, fail-fast on an unsupported CRS, a format-keyed registry — beat one interface that fits neither. The registry (EXPOSURE_INGEST_ADAPTER_REGISTRY) lives in core/, not worker/, which is the concrete consequence of the synchronous/no-object-storage row above: exposure adapters only do local reads, so ADR-050's reason for putting INGEST_ADAPTER_REGISTRY in the worker layer does not apply here.

RasterExposureAdapter (core/ingest/raster_exposure.py, registered as "raster") is the first implementation: a single-band EPSG:4326 value raster, read in block-aligned strips sized against a byte budget, aggregated agg_factor × agg_factor per block, emitting one point per non-empty block at the block's cell-centre. Peak memory is one strip plus one batch, set by the budget and the raster's width — a 100 m national grid and a 1 km one cost the same working set. Aggregation semantics are otherwise preserved bit-for-bit from the script; the parity is pinned by a test that keeps a verbatim copy of the old implementation as its oracle.

core.exposure_service.ingest_exposure_from_source owns persistence: name-idempotency check first (so a duplicate never reads the source at all), then the exposure_datasets identity row — including the native_resolution and bbox the adapter measured — then one bulk insert per streamed batch via the new ExposureRepository.insert_bulk, an executemany that materialises no ORM instances. It returns counts, never rows: returning a hundred thousand Exposure objects would undo the memory bound the path exists for.

Two deliberate behaviour changes from the script it replaces, both fail-loud-instead-of-wrong:

A non-EPSG:4326 raster is rejected. The script wrote the affine transform's x/y straight into lat/lon, which is silently garbage for a projected source. Same stance as ADR-051.
NaN cells count as no-data (zero) rather than poisoning their block. Previously one NaN turned its block's sum into NaN, which then failed the > 0 test and silently discarded every other cell's value in that block.

The CSV/XLSX upload path (create_exposures_from_upload) is untouched: it is interactive, size-capped, and returns the rows it wrote, all of which are correct for an upload and wrong for an ingest.

Consequences¶

A new exposure source format registers by implementing two methods, not by writing a script.
scripts/ingest_historical_catalog.py's ingest-exposure is now a thin caller: download the cached archive, build an ExposureIdentity from the manifest, call the service. Its exposure_grid_rows/exposure_csv_bytes helpers are gone.
Exposure datasets ingested this way carry real native_resolution/bbox, so an ADR-048 precedence resolver can treat hazard and exposure datasets uniformly.
The CSV round-trip is gone from the raster path, so the 50 MB upload ceiling no longer bounds how large an exposure raster can be ingested.
Two ABCs to keep coherent instead of one. Accepted: they are deliberately allowed to diverge, and the alternative was one ABC that lies about half its callers.

Alternatives Considered¶

Make RasterExposureAdapter implement IngestAdapter. Rejected: see the table above. Three of the four verbs would be stubs, and plan_chunks/read_chunk both take a HazardType.
Keep the logic in the script and just add streaming. Rejected: it leaves the second exposure pack (WorldPop, Stream E) with no path but a third script, which is the problem #594 exists to close.
Route exposure ingest through the Celery chord for symmetry. Rejected: exposure ingest has no fan-out to justify — the expensive part is a windowed read that streams fine in one process — and it would put a job-status surface in front of a call that today returns synchronously to a script.
Reuse create_bulk for the writes. Rejected: it builds and retains one ORM object per row, which is exactly the unbounded growth the streaming read avoids.

Follow-up¶

Register a vector/point-file exposure adapter (GeoPackage, Parquet) when a pack needs one.
Reprojecting reads for projected exposure rasters, if a target pack ships one.
An API surface over ingest_exposure_from_source if exposure ingest ever needs to be operator-triggered rather than script-triggered.

ADR-056: Dataset Precedence — Quality, Then Locality, Then Recency¶

Date: 2026-08-06 Status: Accepted Phase: 9 (Data foundation) — Stream A3A

Context¶

Until now, when two hazard datasets covered the same peril, core/lookup_service.pick_dataset resolved the tie with the newest row that has a COG. That was defensible when the catalog held one dataset per peril. Phase 9 exists to end that: the exit criterion is "≥2 hazards offer ≥2 user-selectable datasets", and the moment a coarse global raster is ingested after a fine national one, insertion order silently serves the worse answer — with no way for a caller to override it and no statement in the response of what was picked or why.

Phase 9's design (docs/plan/phase-9-data-foundation.md, storage design pt 5) names the replacement: a ThinkHazard-style precedence rule of quality score → local-over-global → recency. ADR-048 shipped the columns it reads (native_resolution, provider, status, the native bbox, …) but deliberately left them nullable with no backfill, so the rule must work on a catalog where most rows still declare nothing.

Decision¶

One shared, pure module — core/dataset_precedence.py — implements the rule, and the four /v1/risk/* rungs accept an optional dataset_id that overrides it.

Candidates are ordered by the first stage that separates them:

Stage	Term	Rule
0	Lifecycle gate	`status == "superseded"` sorts last, never dropped
1	Quality score	0–100: `native_resolution` (60, bucketed) + `provider` (20) + `upstream_version` (10) + `license` (10)
2	Local over global	native `bbox` ≥ 50% of the globe is global; a `region` label alone is local; neither is unknown, ranking between the two. Two locals tie-break on the smaller extent
3	Recency	newest `created_at`; ties keep input order (the sort is stable)

Four decisions inside that table are the load-bearing ones:

Resolution is bucketed, not continuous. 90 m versus 1 km is a real quality difference; 0.0083° versus 0.0084° is noise. A continuous score would invent precision the catalog does not have, and would make the ordering hostage to a provider's rounding.
Resolution outweighs the whole attribution block (60 > 40). The finest dataset with no paperwork still describes the ground better than a fully documented coarse one. But an un-backfilled row scores 0 and loses to anything that declares itself — which is the only pressure ADR-048's no-backfill decision leaves in the system.
Unknown never guesses. Unknown resolution scores zero rather than a default; unknown locality ranks between local and global; an unrecorded extent sorts as if it covered everything. A malformed bbox is read as absent, never raised: a precedence decision is not the place to fail a lookup over catalog metadata drift.
Superseded is ranked, not filtered. A peril whose only dataset has been retired still answers, and an explicit dataset_id still reaches it — ADR-048's stated intent that a superseded dataset stay queryable for historical results while being withheld from new selections.

Answerability is decided outside the rule. pick_dataset ranks the COG-bearing candidates as their own pool first and only falls back to the COG-less ones, so a finer dataset whose raster conversion has not run cannot shadow the sibling that can actually answer a point query. That keeps cog_path — a hazard-only column — out of the shared module, which is typed against a structural DatasetIdentity protocol so exposure_datasets (ADR-048 ships it the same ten columns) can share the rule unchanged.

A pin scopes exactly one peril. One dataset_id cannot describe a four-hazard card, so it applies to its own peril and every other peril still resolves by precedence. It is validated once, in core/lookup_service.resolve_selected_dataset — exists, org-visible (the repository is org-scoped, so another org's dataset is reported identically to a non-existent one, and the id cannot be used to probe), and consistent with the request's hazards. Resolution is by id rather than out of the scenario/region/horizon-filtered list: silently dropping a pick that falls outside those filters would return a different dataset's numbers under the caller's id.

Every response names the dataset and the rule. provenance.dataset gains provider, upstream_version, status and a selection block (quality_score, locality_rank, superseded); the rollup rungs render the same block at the top level. Naming the winner without naming the rule leaves a user unable to tell a deliberate pick from an accident of insertion order — which is precisely the state this ADR replaces.

The admin-unit aggregate pins exactly one generation. With a dataset pinned it runs legacy_fallback=False and, on an empty result, retries against legacy (NULL-dataset) rows only. Leaving the fallback on would union two dataset generations into one mean and count shared ground twice; widening to another dataset's surface would make the dataset named in the payload a lie.

Alternatives Considered¶

A stored quality_score column. Rejected: a persisted score is a snapshot that goes stale the moment the rule changes, needs a backfill for every existing row, and would have to be recomputed on every identity edit. Deriving it makes the rule a code change with tests, not a data migration.

Demote the seeded demo fixture (SEEDED_FIXTURE_SOURCE) inside precedence. Rejected here:

644 already keeps the demo fixture out of scenario resolution, and `source` is not an ADR-048¶

identity column — folding a hazard-only, demo-only marker into the shared rule would couple the future exposure resolver to it. Revisit if a seeded fixture is ever observed winning a real selection.

Let dataset_id scope a whole multi-peril request. Rejected: it reads as convenient and is incoherent — a portfolio spans perils, and pinning a flood dataset would either scope the wind surfaces to an id they cannot match (silently empty) or be ignored for them (silently partial). One id, one peril, stated.

Default the portfolio rung to a per-peril precedence pick. Rejected: a portfolio spans regions and scenarios, so a per-peril default would scope a whole book to one raster's footprint. Only an explicit pick narrows that rung; without one it reads every surface for the key, as before.

Follow-up¶

Stream A3B-2 (#591): the scenario-matrix resolver pins the provider family across cells using the same identity columns; the matrix's cell-level tie-breaks should call this module rather than restate it.
Stream D3B (#610): the SDK's dataset selector passes dataset_id through to these endpoints.
ui#100/#101: the dataset picker renders the selection block so a user sees why a default was chosen before overriding it.

ADR-057: Retiring `hazard_centroids` Row Writes (ADR-045 Topic 2 Implementation)¶

Date: 2026-08-06 Status: Accepted Phase: 9 (Data foundation) — Stream B3, issue #596

Context¶

ADR-045 topic 2 decided the direction — replace the per-pixel hazard_centroids row with a grid definition on the dataset plus the pixel-index array already in the intensity NPZ — and deferred the implementation. docs/plan/grid-native-hazard-geometry.md measured what the deferral costs: prod Postgres is 4.32 GB, of which hazard_centroids is 4294 MB, across 7,182,016 rows for 8 datasets. Every other table combined is ~15 MB.

The read side has already moved. worker/tasks._resolve_centroid_geometry (#559), core/centroid_geometry.resolve_point_centroids (#560) and core/admin_aggregation (#561) all derive coordinates from the v2 artifact's (grid, pixel_idx) pair and only fall back to the table for a v1 artifact. What remained was the write side — two sites, both via HazardRepository.create_centroids_bulk — and the one consumer that could not be moved by inspection: core/centroid_assignment's cell-membership mode, which resolved an exposure's array_index with LEFT JOIN hazard_centroids ON pixel_index.

That join is the whole problem. array_index is the CSR column ordinal, and for a sparse dataset (only nonzero cells stored) it is not equal to the row-major pixel_index — so "just do the arithmetic" is only correct for a dense grid. ADR-045 named two candidate redesigns (a session-scoped temp table, or searchsorted in Python) and explicitly declined to choose between them.

Decision¶

Membership resolves array_index from one of three sources, chosen per dataset at assignment time. The decision is made once, in _resolve_index_source, and drives which of six pre-compiled statements (2 scopes × 3 sources) runs:

GRID — dense grid-native. The artifact stores every cell, so array_index == pixel_index identically. The cells CTE already computes that guarded key, so the statement writes it straight through: no join at all. This is the shape all 8 production datasets have (the plan doc verified pixel_index == array_index for all 7,182,016 rows).
PIXEL_MAP — sparse grid-native. The mapping is the ascending pixel_idx union the v2 NPZ already carries (array_index i is union entry i). It is joined in as unnest(CAST(:pixel_index_map AS bigint[])) WITH ORDINALITY — a bind, not a table.
CENTROID_ROWS — legacy. A v1 artifact, or any dataset whose NPZ carries no grid geometry, keeps the original join unchanged. This is the answer to "what happens to non-grid/legacy datasets": keep the rows and the existing paths, do not backfill as part of this change. Those datasets still have their rows — nothing has deleted them — so the path is exact, not approximate.

Both write sites stop writing rows. The chord's commit step (_persist_events, formerly _persist_centroids_and_events) persists events only. The synchronous BaseGeoTIFFIngestor. ingest path becomes grid-native instead: it stamps the grid definition and writes a v2 artifact with an identity pixel_idx, rather than a v1 artifact plus a row per pixel.

hazard_datasets.footprint (migration 0075, typed ops) stores the dataset's EPSG:4326 extent once, as the grid envelope derived from the eight numbers that define the grid. It replaces the MIN/MAX(lon, lat) FROM hazard_centroids aggregate in two places: the KNN assignment scope's spatial pre-filter and the catalog preview's bbox. The grid envelope rather than the kept cells' tighter one is deliberate — it is O(1) to derive (no scan of a dataset-sized array, which #595 just made out-of-core), it is a superset so it never prunes an exposure membership would have matched, and it is the exact envelope the membership pre-filter already uses. COALESCE(footprint, <the old aggregate>) keeps pre-#596 datasets correct.

A grid dataset with neither v2 geometry nor legacy rows is an error, not a fallback. Silently taking the CENTROID_ROWS path against an empty table would not fail: it would LEFT-JOIN every exposure to a sentinel and report a complete, successful assignment whose every cell computes to zero impact. A transient artifact read failure must not be indistinguishable from "this hazard affects nothing", so _require_legacy_rows raises.

Consequences¶

A hazard dataset's Postgres footprint is O(1) in rows regardless of pixel count. Ingest no longer performs millions of batched INSERTs, and the DB stops growing per dataset.
Cell-membership assignment now reads the dataset's intensity NPZ (once per (hazard, exposure-dataset) pair, behind the existing idempotence check and advisory lock) to recover pixel_idx. For the dense case that read only confirms density; making that confirmation free would need a stored cell count, which is not worth a column today.
create_centroids_bulk and preview_dataset survive as legacy-only surfaces: still read/written by tests, fixtures and any future backfill or repair script, but no production ingest path calls them. Both say so in their docstrings.
The unnest bind for the sparse case sends the union over the wire per assignment call (~21 MB at the measured 2.6M-cell Greek pack). If that ever becomes the bottleneck, ADR-045's other candidate — a session-scoped temp table populated in batches — is the escalation; it was not chosen now because it adds DDL and transaction-lifetime semantics for a cost nothing has yet measured.

Alternatives Considered¶

Widen the CSR's column space to the whole grid so array_index == pixel_index always. Rejected. It would make membership pure arithmetic for every dataset, but the compute path derives centroid_lat/centroid_lon for every column — at Greek-pack scale that is 102.6M coordinate pairs (~1.6 GB) per job instead of 2.05M. It moves the cost from Postgres into the worker's heap, which is the exact ceiling ADR-045 topic 1 just removed.
Store the pixel-index union in a Postgres array column on hazard_datasets. Rejected: it keeps everything in SQL and is O(1) in rows, but it re-couples the array to Postgres — the scaling axis this decision exists to remove — and contradicts ADR-045's recorded direction that the array lives beside the NPZ in object storage.
Backfill legacy datasets as part of this change, then delete the rows. Rejected as scope: deleting rows in prod is an ops decision with its own rollback story (ADR-045 topic 2's own follow-up, and item 8 of docs/plan/grid-native-hazard-geometry.md, both gate it on verification against prod). This change is write-path-only and leaves every existing row readable, so it is reversible by reverting code alone.

Follow-up¶

Ops, not code: deleting the ~4.29 GB of existing hazard_centroids rows in prod, after the grid backfill (scripts/backfill_grid_geometry.py) confirms every dataset's v2 artifact reproduces its stored lat/lon bit-exactly. Until then the legacy path is load-bearing.
Dropping the table itself is item 8 of docs/plan/grid-native-hazard-geometry.md and stays gated on that verification.

ADR-058: Coastal Flood Ships as an Indicator Peril Pending a Coastal Curve¶

Date: 2026-08-06 Status: Accepted Phase: 9

Context¶

Issue #602 wires coastal_flood — the second computable-shaped peril of the phase-9 expansion, and relevant to Greece's long coastline — following the mechanical checklist in docs/architecture/peril-wiring.md (ADR-024/#600). The enum member and its hazard_type_enum value already landed in wave 1 (#599, migration 0069). What remained: an ingestor, a LayerSpec, and a posture decision on whether coastal flood ships computable or indicator (core/eai_eligibility.py, #600).

The only depth-damage curves seeded on this platform are the Huizinga et al. (2017) JRC functions (impact_function_seeder.py), and those are explicitly derived and attributed for fluvial river flooding — the same publication's applicability to coastal/surge-driven damage has not been verified from the published methodology spreadsheet, and that verification is issue #621's job (the Huizinga re-derivation), not this one's. Separately, climate-lama-engine does not yet recognise the peril's CF two-letter code (climate-lama-engine#30), so the peril could not run through EngineAdapter even if a curve existed.

Decision¶

Coastal flood ships as an indicator peril in #602: ingestor + LayerSpec + score scheme, but no seeded impact function. Concretely:

core/ingest/coastal_flood.py — CoastalFloodIngestor(BaseGeoTIFFIngestor), intensity_unit = "m", registered in INGESTOR_REGISTRY. Its module docstring records why no curve is seeded (honest-framing rule) rather than silently reusing the fluvial river-flood curve, which would misrepresent a different damage mechanism as the same one.
models/layer_spec.py gains a HazardType.COASTAL_FLOOD raster entry ("GnBu" colormap, distinct from river flood's "Blues" so both flood layers are visually distinguishable together on the map).
core/scoring.py's _BAND_SOURCE_BY_HAZARD classifies coastal flood as POINT_INTENSITY — same reasoning as river flood: inundation depth is spatially discontinuous at sub-cell scale.
Migration 0074 seeds exactly one score_schemes row (self-set v1 depth bands, mirroring the river-flood scheme's breakpoints per migration 0057's established convention) — mandatory, not optional, per the peril-wiring checklist's indicator-peril row: the band is this peril's only user-visible output until a coastal curve is seeded.
No entry is added to impact_function_seeder.py's _BUILTIN_FUNCTIONS / _BUILTIN_EXPOSURE_TYPES. That absence is what core/eai_eligibility.py reads as NO_SEEDED_CURVE — the posture is derived, never declared (#600), so this ADR does not introduce a new mechanism, only a new peril that exercises the existing one.
worker/models/engine_adapter.py's supported_hazards list stays unchanged (no coastal_flood entry) — documented in place, not silently omitted — since the peril cannot reach the engine either way today.
tests/hazard_wiring.py's UNWIRED_HAZARD_TYPES drops its COASTAL_FLOOD entry.

Consequences¶

Coastal-flood datasets ingest, tile, and band today; they never produce an expected annual impact until both #621 seeds a coastal-specific curve and climate-lama-engine#30 lands.
The score band is genuinely the whole user-visible output for this peril in the interim — its v1 thresholds carry the same self-set caveat as every other seeded scheme (ADR-043) and are planned-polish, not blocking.
A future coastal curve (#621) requires no ingest or gate change — only a _BUILTIN_FUNCTIONS entry whose intensity_unit matches this ingestor's "m", at which point every coastal-flood dataset flips to computable with no migration or backfill (the property #600's derivation exists to buy).

Alternatives Considered¶

Seed the fluvial Huizinga curve against coastal flood to make it computable now. Rejected: the curve is calibrated and attributed for river flooding; reusing it for a surge/wave damage mechanism it was never validated against would produce a confidently wrong number rather than an honestly absent one — exactly what the peril-wiring checklist's honest-framing rule exists to prevent.
Defer the score scheme until a curve is seeded. Rejected: per the peril-wiring checklist, a score scheme is mandatory for an indicator peril — it is the only output the peril has, not optional polish to add later.

Follow-up¶

621 (Huizinga re-derivation, Opus·max) resolves the curve question one way or the other:¶
seed a coastal-specific curve, or record explicitly that none exists.
~~climate-lama-engine#30 (already filed) tracks the CF/EQ engine code gap.~~

Update (climate-lama-engine#30, 2026-08-10). There was no engine code gap. The engine never dispatched on haz_type — it is a free-form tag, and the peril-specific physics lives entirely in the intensity matrix and the MDD/PAA curve — so CF would always have computed. climate-lama-engine 0.5.0 published a documented hazard-code vocabulary (HAZARD_CODES) that confirms CF verbatim against climada_petals.hazard.coastal_flood.HAZ_TYPE, with tests running a full ImpactCalc over it.

That collapses the "both #621 and engine#30" condition in Consequences above to one condition — the curve. And #621 has itself since closed (2026-08-06) without producing a coastal curve: it re-derived the fluvial Huizinga set and left coastal without one, which is the "or confirms none exists" branch its own Follow-up anticipated. Sourcing a coastal curve from elsewhere — or recording permanently that none exists at acceptable provenance — is now tracked in #833. Until then _BUILTIN_FUNCTIONS seeds five perils and no coastal entry, so classify_eai still derives NO_SEEDED_CURVE.

supported_hazards still omits coastal_flood, but the reason has changed from "the engine cannot" to "no curve is seeded" — listing it would advertise compute the platform cannot deliver, so the omission is now load-bearing on its own and is pinned by test_adapter_does_not_claim_coastal_flood.

This document should be updated as new decisions are made.

ADR-059: Huizinga Flood Curves Re-Derived From the Published JRC Source; No Coastal Variant Exists¶

Date: 2026-08-06 Status: Accepted Phase: 9

Context¶

Issue #621 (phase-9 Stream E, vulnerability pack E13). Two questions, one investigation:

The license boundary. core/impact_function_seeder.py seeded its river-flood curves under a comment naming a CLIMADA river-flood module as their origin, while attributing them in the database to Huizinga et al. (2017) with a CC-BY-4.0 license string. The coefficients were taken from a GPL-3.0 codebase and labelled with a permissive citation that did not produce them. Under ADR-024 this backbone never depends on CLIMADA; a value copied out of it is the same dependency as an import, only unauditable.
The coastal question. ADR-058 shipped coastal_flood as an indicator peril explicitly pending this issue's verdict on whether the JRC source supports a defensible coastal depth-damage curve.

The authoritative sources were retrieved from the JRC Publications Repository (record JRC105688) and hashed: the report PDF global_flood_depth-damage_functions__10042017.pdf (sha256 f1e9f1b1…) and the accompanying spreadsheet copy_of_global_flood_depth-damage_functions__30102017.xlsx (sha256 5b944b59…).

Decision¶

1. Curves are re-derived from the published spreadsheet, mechanically and re-runnably. scripts/packs/derive_huizinga_flood_curves.py reads sheet Damage functions, column C (EUROPE), and writes scripts/packs/huizinga_2017_europe_damage_functions.json. The seeder holds the same values with per-class cell citations, and tests/test_core/test_huizinga_provenance.py plus the script's --check mode fail on any drift between the two. No coefficient in this repo is hand-transcribed.

Europe is a special case worth recording: it has no numeric table in the report body (Tables 3-1..3-20 cover the other five continents), appearing only as Appendix F figures, because the European functions came from a separate earlier study (Huizinga, 2007). The spreadsheet is therefore the sole authoritative numeric source for Europe — and the later one, its Info sheet recording "European damage functions updated in October 2017" against the report's April 2017 date.

2. MDD carries the whole published factor; PAA is 1.0. The source publishes one fractional curve per damage class — the Info sheet defines it as "the share of asset that is damaged at a given flood depth" — and no MDD/PAA decomposition. Since this platform computes damage as MDD × PAA, pinning PAA to unity reproduces the published function exactly; any other split would invent structure the source does not contain. The superseded row used a graded PAA with no basis in its citation.

3. The license string is corrected. The report is not Creative Commons. Its own reuse notice (PDF inside cover, the page carrying the JRC105688 / EUR 28552 EN identifiers) reads: "© European Union, 2017. The reuse of the document is authorised, provided the source is acknowledged and the original meaning or message of the texts are not distorted." That is the European Commission reuse regime (Commission Decision 2011/833/EU). Reuse with attribution is explicitly permitted, so the re-derivation is sound — but the previous CC-BY-4.0 claim was unsupported and is replaced.

4. The population curve is removed, not re-derived. Huizinga publishes six damage classes — residential, commerce, industry, transport, infrastructure (roads), agriculture — and none of them is population. JRC flood Europe population cited a document containing no counterpart to it. There is nothing to re-derive it from, so it is deleted rather than left in place under a citation that does not support it. River flood is consequently no longer EAI-eligible for ExposureType.POPULATION; that fails closed, which is the correct direction for a risk platform, and no demo path used it.

5. Only the three building classes are seeded. JRC flood Europe buildings (residential, default), … commerce and … industry. Transport, infrastructure-roads and agriculture are extracted and fully cited in HUIZINGA_EUROPE_DAMAGE_FACTORS but not seeded: seeding them would silently flip river flood to EAI-eligible for ExposureType.INFRASTRUCTURE, and agriculture has no ExposureType member at all. Wiring them is a product decision, not a provenance one.

The coastal verdict — ADR-058's open question, answered: NO¶

The published JRC source does not support a coastal depth-damage curve, and none is seeded. The evidence is structural rather than a matter of judgement:

The Damage functions sheet is indexed by damage class × continent only. There is no flood-type dimension anywhere in it — no river/coastal split, no surge variant.
The report never distinguishes the two. Across all 114 pages the strings salt, sea water, saline, brackish and storm surge do not occur at all. coastal occurs only in the JRC's framing of its own remit, one aside about comparing flooded coastal cities, and bibliography entries — never as a curve, parameter or adjustment.
Salinity is named exactly once as a flood parameter others have considered (§3.2.6, agriculture, citing Brémond et al. 2013), immediately followed by the study's statement that flood depth is the parameter it uses. The damage mechanisms that make coastal flooding different — salt-water contact, wave action, flow velocity, inundation duration — are outside the model by construction.

Reusing the fluvial curve for coastal flood would therefore not be a documented approximation; it would be an undocumented one, producing a confidently wrong number where the platform currently produces an honestly absent one. ADR-058 stands unchanged: coastal_flood remains an indicator peril. Sourcing a coastal curve is new work against a different publication, not an extraction from this one.

Consequences¶

Every seeded flood coefficient now traces to a published, non-GPL document, with the spreadsheet cell range recorded per damage class and the source files hashed.
The provenance claim is enforced by tests, not asserted in prose: a drifted coefficient, a reinstated CC-BY string, a re-added population curve, or a CLIMADA module named as a value source all fail the suite.
Migration 0077 rewrites the buildings row and deletes the population row for orgs seeded before this change. New curves reach existing orgs via POST /v1/impact-functions/seed, per the back-population precedent migration 0030 records.
ADR-058's follow-up is discharged: #602 can treat the coastal-curve question as closed (negative) rather than pending.

Alternatives Considered¶

Keep the existing coefficients and just fix the citation. Rejected: that is the exact defect — a GPL-derived value wearing a permissive label. The values had to change.
Take the values from CLIMADA and "verify" them against the spreadsheet. Rejected: the direction of derivation is the whole point. Reading the published source first is what makes the result independently reproducible; back-checking a GPL copy is not.
Derive a coastal curve by adjusting the fluvial one (e.g. a salt-water uplift factor). Rejected: any such factor would be invented. The source contains no coastal data to calibrate against, and a plausible-looking fabricated curve is worse than an absent one.
Seed all six damage classes. Rejected for now — see decision 5; it changes EAI eligibility surface, which is a product call outside this issue.

ADR-060: Earthquake Ships Computable but Screening-Grade, on ESRM20's Published PGA Vulnerability Functions¶

Date: 2026-08-06 Status: Accepted Phase: 9

Context¶

Issue #604 (phase-9 Stream C1, ratified decision 3) wires earthquake, the largest remaining gap in the Greek peril mix. Two questions had to be answered before any code was written, and the issue's own grounding flagged the first as the primary design risk.

1. Curve shape. This platform stores a vulnerability curve as continuous mdd_x/mdd_y and paa_x/paa_y arrays against a physical intensity. ESRM20's headline vulnerability product is fragility, keyed by GEM building taxonomy and by spectral acceleration Sa(T), and its own risk calculations convolve that fragility with a hazard curve inside OpenQuake. The phase-9 plan's recommendation was that "the EMS-98/macroseismic form of the ESRM20 curves is the cleaner v1 match to raster intensities" — i.e. band the raster into macroseismic degrees and apply a damage-per-degree function.

2. Posture. Coastal flood (ADR-058) and heatwave (#603) both shipped as indicator perils because no defensible curve existed. Whether earthquake should join them, or ship a number behind a label, was open.

The sources were retrieved at pinned release tags and hashed:

esrm20_vulnerability_total-repl-cost.xlsx @ v2.1, sha256 31caa497… — the European Building Vulnerability Data Repository (Romão et al. 2021, Zenodo doi:10.5281/zenodo.4062410).
Exposure_Model_Greece_Res.csv @ v1.0, sha256 1104b73d… — the ESRM20 European Exposure Model's Greek residential file.

Both repositories carry an explicit LICENSE file reading "This work is licensed under the Creative Commons Attribution 4.0 International License." That was read, not assumed —

621 (ADR-059) had just found a `CC-BY-4.0` string in this same module that no source¶

supported.

Decision¶

1. Take ESRM20's published PGA vulnerability sheet verbatim. Do not adopt a macroseismic form. The workbook publishes four intensity-measure sheets — PGA, SA(0.3s), SA(0.6s), SA(1.0s) — each holding 511 building classes' mean loss ratio of total replacement cost across 50 intensity levels. The PGA sheet is a direct, published, continuous loss-ratio-against-g function: exactly this platform's curve shape, and exactly the quantity ESHM20's return-period hazard grids carry.

The recommendation in the plan was written before that sheet was known to exist. It is superseded on its own logic: the reason to prefer a macroseismic form was to avoid an unsourced conversion between a raster intensity and the curve's native axis, and the PGA sheet removes that conversion entirely. Adopting EMS-98 would have added one — a PGA-to-macroseismic-intensity conversion, which EMS-98 itself does not publish (the scale classifies shaking from observed effects and states no ground-motion equivalences; every such correspondence in circulation comes from separately published conversion equations). Choosing the macroseismic route would therefore have meant importing an unsourced numeric relationship to avoid one that never existed.

2. Earthquake is computable, and its EAI is screening-grade. The seeded curves consume the g the ingestor declares, so core/eai_eligibility.py derives computable with no special case. But this platform reads a PGA raster at a point and applies one of four representative published class curves; ESRM20 convolves taxonomy-keyed fragility with Sa(T) over a per-asset building-class portfolio. That is a real simplification and the number must say so.

3. Grade is a second axis, declared where posture is derived. EaiGrade (full / screening) is a per-peril table in core/eai_eligibility.py, surfaced as eai_grade and eai_grade_note on the score card, the rollups and GET /v1/hazards beside the existing indicator flag. It is declared rather than derived because the simplification is a property of the method, not of any recorded unit — there is nothing in the data to derive it from. Keeping the axes separate is what makes the pending C3/OpenQuake verdict (#366) cheap: downgrading earthquake to indicator-only means deleting its seeded curves and letting the posture derivation do the rest; confirming it as full-grade means deleting one dictionary entry. Neither is a rewire.

4. Four curves, selected by published Greek replacement-cost share. This platform holds no GEM taxonomy on its exposure records, so a per-asset class lookup is impossible and 511 curves would be noise. The four seeded classes are the ESRM20 counterparts of the largest Greek residential building classes by total replacement cost, computed from the published exposure model and recorded in the extract (6.21%, 5.04%, 4.26%, 3.17%). The exposure model's GEM taxonomy strings and the workbook's column headers are different notations with no published crosswalk, so each reading is written out in SELECTED_CLASSES — this is a selection of which published column to seed, never an arithmetic transformation of a published number.

5. Two derivation choices, both stated, nothing else touched. PAA is pinned to 1.0 at every level, because ESRM20 publishes one fractional loss-ratio curve per class and no MDD/PAA decomposition — carrying the whole published ratio in MDD reproduces the published function exactly, and any other split would be invented (identical reasoning to ADR-059). And a (0 g, 0) knot is prepended: the published axis starts at 0.05 g with several classes already non-zero there, so without an anchor the interpolation would report a loss at zero ground motion. Every knot from 0.05 g upward is verbatim.

6. Banding cites what it actually uses. Migration 0078's PGA band edges are self-set, informed by the shape of the USGS ShakeMap perceived-shaking ladder — the same honest posture as migrations 0057/0073/0074/0076. EMS-98 is deliberately not cited there, for the reason in decision 1: it publishes no PGA values, so attributing numeric edges to it would be exactly the unsupported attribution #621 was opened to repair.

Consequences¶

Earthquake appears in the catalogue with an ingestor, a LayerSpec, a score scheme, seeded curves and a screening-grade label. tests/hazard_wiring.py no longer excuses it, so every coverage guard in the suite now holds it to full wiring.
Every seeded earthquake coefficient traces to a hashed, CC-BY-4.0 published workbook. scripts/packs/derive_esrm20_earthquake_curves.py --check and tests/test_core/test_esrm20_provenance.py fail on drift, on a reinstated CLIMADA attribution, on a license string the source does not carry, and on an added knot beyond the documented zero anchor.
~~climate-lama-engine still does not recognise the EQ code (climate-lama-engine#30), so an impact job fails at the adapter with an unknown-hazard error rather than routing to a heavier runtime (ADR-024). Ingest, catalogue, banding and the score card all work meanwhile. Adding "earthquake" to EngineAdapter.supported_hazards is the single change that turns the EAI path on once the engine learns the code.~~ Done — see Update below.
The UI must render eai_grade as a label wherever it shows an earthquake EAI. The field is additive, so an un-updated UI degrades to showing an unlabelled number — which is the one degradation this ADR is trying to prevent, and is why the label also travels inside each curve's citation string.

Update (climate-lama-engine#30, 2026-08-10). The earthquake EAI path is live: EngineAdapter.supported_hazards now lists "earthquake", pinned by test_adapter_supports_earthquake, and the backbone pins climate-lama-engine>=0.5.0.

The premise of the struck bullet was wrong in an instructive way. The engine never dispatched on haz_type at all — it is a free-form tag carried through to the result, and the peril-specific physics lives entirely in the intensity matrix and the MDD/PAA curve. So EQ was never rejected by the engine; it was rejected by this repo's own supported_hazards list, which had been written on the assumption of an engine-side gate that did not exist. What 0.5.0 actually shipped is a documented vocabulary (HAZARD_CODES) plus tests proving EQ and CF compute — evidence for the claim, not a lifted restriction. EQ has no CLIMADA hazard class to match, so it was adopted after collision-checking against CLIMADA's ten codes.

The screening-grade caveat is untouched: it is a property of the data and the one-curve-per- class simplification, not of the compute route, and still travels on every result via EaiGrade and the curve citation string. #366 (OpenQuake) remains the thing that would remove it.

Alternatives Considered¶

Adopt the EMS-98/macroseismic curve form, as the phase-9 plan recommended. Rejected on the plan's own reasoning once the published PGA sheet was found: it would introduce an unsourced PGA-to-intensity conversion to avoid a conversion that the PGA sheet makes unnecessary. See decision 1.
Ship earthquake as an indicator peril, like coastal flood and heatwave. Rejected: those two are indicators because no defensible curve exists. Here one does, published and directly consumable. Suppressing a real number to avoid labelling it would be a different kind of dishonesty, and would leave the screening-grade mechanism unbuilt for the next peril that needs it.
Compute an exposure-weighted portfolio curve for Greece by combining ESRM20 class curves with the exposure model's replacement-cost weights. Rejected: it needs a class-by-class crosswalk that ESRM20 does not publish, with judgement calls on lateral-force-coefficient buckets and height bands baked into every resulting coefficient. The arithmetic would be sound and the inputs would not be — the failure mode ADR-059 exists to prevent. A selection of published columns keeps every number verbatim and every judgement visible.
Seed all 511 published classes. Rejected: nothing in this platform can choose between them, since exposure records carry no GEM taxonomy. It would be a larger surface with no more information.
Wait for OpenQuake (#366) and ship nothing now. Rejected per the plan's ratified decision 3: the Greek peril mix is the gap, the label carries the caveat honestly, and the OpenQuake scoping spike remains the anti-indefinite-deferral anchor.

ADR-061: (Peril × Exposure Type) Is Enforced for Built-In Curves, Reported for Everything Else¶

Date: 2026-08-06 Status: Accepted Phase: 9

Context¶

Issue #690, a follow-up to #621 (ADR-059). Since #600 the platform has treated EAI eligibility as a (peril × exposure type) question and reported the matrix on GET /v1/hazards under eai_eligibility.by_exposure_type. Enforcement never followed the reporting, and the two drifted into contradicting each other:

GET /v1/hazards reported by_exposure_type.population = false and .infrastructure = false for every river-flood dataset — correctly, because _BUILTIN_EXPOSURE_TYPES keys every seeded JRC flood curve to BUILDINGS.
core/compute_service.validate_impact_refs gated only on the dataset-level verdict with exposure_type=None — the union over every exposure type — and never compared the chosen curve against the exposure data at all.

So a caller could submit a POPULATION exposure dataset with the JRC flood Europe buildings curve and the job succeeded, returning a monetised "loss" produced by multiplying a building depth-damage fraction by a head count. The platform advertised the combination as ineligible and then computed it anyway.

The union behaviour was deliberate and documented: validate_impact_refs's docstring blessed it in as many words — "river flood against infrastructure exposure is served by the seeded buildings curve". That was written to avoid breaking calculations that ran, and it is wrong on its own terms. Those calculations ran; they did not mean anything. A silent misapplication is strictly worse than a silent absence: the absence is visible as a refusal, while the misapplication arrives as a number with a currency symbol on it.

Two product questions had to be closed before the API change could be scoped, and both were closed against seeding new curves:

River flood × POPULATION stays unsupported. The published literature measures mortality (dose-response functions whose output is deaths, with the better-specified members additionally requiring flow velocity and rise rate) or exposure/displacement counts — never an asset-style depth-damage fraction. Seeding any of those as an mdd_y against a USD-defaulted exposure would yield a monetised "population damage" figure with no defensible basis: exactly the failure #621 corrected, with an invented shape substituted for a copied one.
The remaining three Huizinga classes stay unseeded. ExposureType has one INFRASTRUCTURE bucket against three distinct published classes (transport, roads, agriculture — and agriculture is not infrastructure at all). Seeding would force either an arbitrary "the" infrastructure curve or a taxonomy expansion. The extracted, cited values stay in scripts/packs/huizinga_2017_europe_damage_functions.json at zero cost; a taxonomy-expansion issue is the prerequisite, and curves follow it, not the reverse.

Decision¶

1. The compute gate refuses a built-in curve aimed at the wrong exposure type. validate_impact_refs now raises ExposureTypeCurveMismatchError → 422 E_EXPOSURE_TYPE_NOT_EAI_ELIGIBLE when the exposure dataset's exposure_type is not the one the chosen built-in curve was calibrated against. The details name the curve, both exposure types, the peril, and the exposure types the peril's built-in curves do cover.

This reverses the tolerance the previous docstring blessed. It is a behaviour change, not a bug fix: a submission that succeeded before now returns 422. The docstring was rewritten to say so rather than quietly deleted.

2. Unit consumability and curve applicability stay separate axes. The indicator check (classify_dataset_eai, exposure_type=None) is unchanged and still unions every exposure type — it asks "can anything consume this intensity?", which is not an exposure-type question. The new check is a second, independent comparison against the curve the job named. Merging them was rejected; see Alternatives.

3. User-authored curves are exempt. impact_functions has no exposure_type column. A curve's calibrated exposure type is known only via the name-keyed impact_function_seeder.builtin_exposure_type(), which returns None for anything not in _BUILTIN_FUNCTIONS. A user-authored curve is therefore never refused: its author asserted its applicability, and blocking it would refuse the one path that legitimately extends the platform beyond what ships seeded.

4. Org-wide runs are exempt, and this is a recorded gap. exposure_dataset_id is optional; without it the job spans every exposure in the org and there is no single exposure type to compare against. Pre-#584 exposure "datasets" — a bare shared UUID on Exposure.dataset_id with no exposure_datasets catalog row — are likewise unchecked, since they carry no recorded exposure type.

5. The matrix carries its reason. eai_eligibility.by_exposure_type widened from {"population": false} to {"population": {"eligible": false, "reason": "no_seeded_curve_for_exposure_type", "note": "…"}}, and IndicatorReason.NO_SEEDED_CURVE_FOR_EXPOSURE_TYPE was added to distinguish "this peril has no curve at all" (heatwave) from "this peril has curves, just not for this exposure type" (river flood × population). The new reason is reachable only when classify_eai is called with a narrowed exposure_type; every gate passes exposure_type=None, so the ingest and compute refusals keep the reasons they already carried.

No new curves, coefficients, ExposureType members, or migrations are introduced by this decision.

Alternatives Considered¶

Narrow the existing union gate — pass the exposure dataset's type into classify_dataset_eai at the gate. Rejected: it would also refuse a job using a user-authored curve for that exposure type, which is the case that must stay allowed, and it conflates unit-consumability with curve-applicability — two axes core/eai_eligibility.py deliberately keeps apart.
Warn instead of refuse — let the job run and flag the mismatch on the result. Rejected: it reproduces the silent-absence failure this decision exists to close, one layer further in. A caveat attached to a number that should not exist is still a number.
Refuse org-wide runs too, via SELECT DISTINCT exposure_type over exposures. Rejected for v1: the column is unindexed, and one stray row of the wrong type would block an otherwise legitimate portfolio run. Recorded as the gap in decision 4.
Re-source a population curve from a non-JRC source. Rejected — see context, closed decision 1.
Add a per-(peril × exposure type) EaiGrade. Rejected: nothing here needs to label a computable result as screening-grade, so _SCREENING_GRADE_PERILS stays keyed by peril alone (ADR-060 is unaffected).

Rationale¶

The reporting/enforcement split is not a compromise — it is the shape of the two facts. What the platform knows about a built-in curve is a published calibration, which is strong enough to refuse on. What it knows about a user-authored curve is that someone uploaded it saying it applies, which is strong enough to permit. Enforcing the axis where the knowledge is and reporting it everywhere else is what keeps the refusal honest without making the platform closed.

The API-shape change (bool → object) is breaking for any consumer parsing those values as bools. climate-lama-ui @ origin/main (2464ba7) references neither eai_eligibility nor by_exposure_type, so no UI change was required; had a consumer existed, the reason would have gone into a sibling field rather than widening in place.

Follow-up¶

Refusing the org-wide (exposure_dataset_id = null) case needs either an index on exposures.exposure_type or a per-dataset pre-aggregation. Not scoped.
Seeding the three remaining Huizinga classes (transport, roads, agriculture) is blocked on an ExposureType taxonomy expansion, which is its own issue. The extracted values are already committed in scripts/packs/huizinga_2017_europe_damage_functions.json.
Population risk, if wanted, is an indicator (people exposed above a threshold), not an EAI. Separate work.

ADR-062: LISCOAST Recorded as Permanently Out of Scope for Hazard Ingest¶

Date: 2026-08-07 Status: Accepted Phase: 9

Context¶

Issue #708, a follow-up to #613 (PR #705). #613 closed as completed, but only the Aqueduct coastal half of its stated scope shipped. The JRC LISCOAST half was evaluated and found un-ingestable with the platform's current formats, and the finding was recorded rather than buried in a merged PR body.

The build behind #613 downloaded the real JRC LISCOAST "Global Extreme Sea Level projections" archive whole (13,351,354 bytes, measured via both a live HTTP HEAD and the downloaded body) and inspected it directly. The issue's premise that LISCOAST is "likely NetCDF" does not hold: the archive is 24 CSVs of irregularly-spaced coastal transect points (latitude,longitude,valueMedian,value5th,value95th), not a raster. Neither ingest format this platform has can consume that — the GeoTIFF path requires a regular raster grid, and the NetCDF path (#593/ADR-051) also requires a regular grid. Scattered point geometry has no ingest route today. The finding is recorded in-repo as LISCOAST_EVALUATED_SOURCE in scripts/packs/coastal_flood.py, mirroring the shape the scenario manifest already uses to document its wildfire/windstorm source gaps.

Two options were open:

(a) Add a point→grid ingest path. Irregular coastal transect points interpolated onto a regular grid before ingest. A genuinely new ingest capability, not a pack change — it would need an interpolation-method choice, a new point/vector IngestAdapter, fixture tests, and a fresh licence read per ADR-059 before any data could be seeded. It would also unlock any other point-published hazard source, not just this one.
(b) Record LISCOAST as permanently out of scope and rely on Aqueduct coastal alone for the coastal-flood peril.

Decision¶

(b) is taken. LISCOAST is recorded as permanently out of scope for hazard ingest. LISCOAST_EVALUATED_SOURCE in scripts/packs/coastal_flood.py is updated to state the decision is final and cite this ADR. The volume-budget ledger note for #613 (docs/data/volume-budget.md) is updated to match.

No point→grid ingest path, interpolation step, or point/vector IngestAdapter is built by this decision. No LISCOAST data is seeded.

Alternatives Considered¶

(a) Build a point→grid ingest path now. Rejected: disproportionate to a decision issue. Coastal flood already has working data from Aqueduct (#613 shipped), so nothing is blocked on this choice — the new capability would advance no user-visible slice today. It also bundles three separate decisions (interpolation method, adapter shape, licence re-read) into what should be a scoped, reviewable piece of work in its own right if it is ever needed.
Leave the gap undocumented (rely on LISCOAST_EVALUATED_SOURCE's existing prose alone). Rejected: the prior wording left the door open as an active question rather than a closed one, which is what #708 exists to resolve. A reader hitting the module later should not have to re-litigate the choice.

Rationale¶

Coastal flood ships computable today on Aqueduct coastal alone (#613, ADR-058). Building a new Stream-B ingest capability to unblock nothing is not a good use of scope for what is otherwise a two-line decision issue. The decision is reversible: if a point-published hazard source later becomes load-bearing for this platform, this ADR can be superseded and option (a) built then, informed by whatever that future source actually needs rather than by LISCOAST's transect-point shape specifically.

Follow-up¶

None. If a point-published hazard source becomes load-bearing, open a new issue that supersedes this ADR rather than reopening #708.

ADR-063: LitPop May Be Sourced From the CLIMADA Data API as an Opt-In Second Source¶

Date: 2026-08-08 Status: Accepted Phase: 9

Context¶

Issue #618 shipped scripts/packs/litpop_exposure.py with exactly one source: ETH Zurich's Research Collection release of LitPop (doi:10.3929/ethz-b-000331316), a per-country CSV at 30 arcsec (~1 km) on 2014 asset values. The pack has no default download URL by design — ops must resolve the per-country file in a browser and pass it via --download-url.

That requirement is the standing blocker on OPS-3b in docs/plan/phase-9-ops-plan.md: the LitPop pack is the only wave in the Phase 9 ops plan that cannot run without owner input.

Three facts, all read live on 2026-08-08, reframed the problem:

research-collection.ethz.ch returns HTTP 403, body: "Access Restricted — due to a high volume of automated traffic (scraping), access … is temporarily restricted from your location / your provider." This is an IP-level anti-scraping control, not the JS-rendering failure the pack's docstring and the volume-budget ledger previously recorded. It is an explicit access-control measure by the publisher.
DataCite (api.datacite.org/dois/10.3929/ethz-b-000331316) serves the item's own registered metadata: rightsList = "Creative Commons Attribution 4.0 International", rightsIdentifier cc-by-4.0 on the SPDX scheme, publisher ETH Zurich, publicationYear 2019, and formats ['text/plain','text/csv','application/x-tar','16.63 GB']. That closes the licence caveat #618 recorded as unconfirmed — the record is registered by ETH Zurich as publisher, so it is the item's own rights field. It also shows the whole release is 16.63 GB, which makes the ledger's ≤0.02 GB per-country estimate unverified.
The CLIMADA Data API (climada.ethz.ch/data-api/v1/dataset/, files on data.iac.ethz.ch) is a plain, public, credential-free REST file server publishing the same authors' LitPop as per-country HDF5. A live Greece query (?data_type=litpop&country_iso3alpha=GRC) returns 3 active v3 datasets, each declaring license: "Attribution 4.0 International (CC-BY-4.0)". The canonical Lit×Pop product (LitPop_150arcsec_GRC, exponents (1,1), fin_mode pc, uuid d9840877-2347-4247-9fc9-ad914cc53fd0) is 1 678 480 bytes; it was downloaded and its declared md5:06f9bd0a29cfc0d1faa2a9acf3f103f8 verified byte-exact. Its own description records reference_year 2018, gpw_version 4.11, climada_version v2.2.0.

The API release is not the same product: 150 arcsec (~4.6 km) versus 30 arcsec, and a 2018 value epoch versus 2014.

Decision¶

Add the CLIMADA Data API as a second, explicitly-selected source. Do not remove or silently replace the ETH Research Collection source.

scripts/packs/litpop_exposure.py gains --source {eth-rc,climada-api}, defaulting to eth-rc, so nothing that already ran changes meaning. Specifically:

--source eth-rc (default) still requires --download-url and still has no default URL constant. The 30-arcsec Research Collection release remains the canonical provenance.
--source climada-api resolves the dataset live against the API (never a hardcoded UUID), refuses to proceed on zero or multiple matches, downloads the file, and verifies the declared md5 and byte size — including on a cache hit.
The API release is catalogued under a different dataset name ("LitPop -- Greece default asset-value exposure (150 arcsec, 2018)"), because the name is the idempotency key. The two releases coexist; they are never merged into one row.
Its identity records its own truth: res_arcsec: 150, exponents: "(1,1)", fin_mode: "pc", reference_year: 2018, gpw_version: "4.11", upstream_version "2018", unit "USD_2018", plus an explicit note that this is a coarser, screening-grade grid than the 30-arcsec release, and the resolved uuid / file URL / md5 so the exact artefact is reproducible.
The HDF5 is read with h5py against its numeric blocks only. It is a pandas HDFStore fixed-format frame whose block1_values is a pickled shapely geometry array; pandas.read_hdf would unpickle it, i.e. execute arbitrary code out of a downloaded artefact. The pack never touches block1_* — latitude and longitude are already plain float columns, so the geometry is redundant.
h5py lands in a new optional packs extra (BSD-3-Clause, on ADR-026's allowlist), not in the base dependencies and not in the api image. The convert subcommand exists so the HDF5→CSV step can run off-box and the CSV be handed in via docker cp + --source-file.

ADR-024 is not breached. ADR-024 fences CLIMADA's code and packaging out of this platform — no import climada, no CLIMADA in any image, no fallback compute path. Fetching CC-BY-4.0 data files over HTTPS from data.iac.ethz.ch loads no GPL code into any process and adds no dependency. The distinction is code versus data, and only the code side is fenced.

Alternatives Considered¶

Replace the ETH Research Collection source with the Data API. Rejected: the two are different products (30 vs 150 arcsec, 2014 vs 2018). Swapping them silently would put a coarser, newer grid under the older release's attribution — precisely the class of mis-attribution ADR-059 exists to prevent. The RC release stays canonical and default.
Install CLIMADA and generate LitPop locally. Rejected twice over. It is unnecessary for downloading — the Data API is plain REST needing no client library — and insufficient for generating: climada/entity/exposures/litpop/gpw_population.py raises FileNotFoundError instructing the user to manually download GPW v4.11 from SEDAC ("Free NASA Earthdata login required"). It would swap one human gate for another while breaching ADR-024 to do it.
Evade the Research Collection's 403. Rejected outright. The 403 is an explicit access-control measure the publisher chose; working around it (proxies, spoofed agents, rate games) is circumvention, not engineering. A human resolving the URL in a browser is the supported path, and the Data API is the supported no-human-input path.
Leave OPS-3b blocked. Rejected: a credential-free, licence-clear, checksum-verified source for the same quantity from the same authors exists, and the ops plan has exactly one wave that cannot run unattended. Recording the coarser grid honestly costs less than the block.

Rationale¶

The blocker was never "we need CLIMADA", it was "we need the file". The Data API supplies the file with no code, no credentials, and a publisher-declared licence that matches DataCite's.
Two sources, two names, two identities is the honest encoding of two different products. One source silently changing shape is how a 4.6 km screening grid ends up presented as 1 km truth.
Verifying the declared md5 (and size) — on cache hits too — makes the artefact reproducible and turns a truncated or tampered download into a hard failure instead of a quiet one.
Reading the numeric HDF5 blocks directly removes a genuine remote-code-execution primitive. A pandas HDF read of a downloaded file is gated only on the file server staying honest.

Consequences¶

The catalog can hold two LitPop rows for the same region. Anything comparing exposure datasets must read properties.res_arcsec / reference_year rather than assume one LitPop.
The API release is screening-grade at ~4.6 km. Downstream impact numbers computed on it are coarser than the RC release's; the grid note in properties says so on every row.
pyproject.toml gains a packs extra. It is mirrored into dev so the HDF5 tests actually execute in CI (which installs .[dev,worker]) instead of skipping themselves. The api image is unchanged — no rebuild or redeploy is needed to use --source-file.
The exact per-country CSV URL and size at the Research Collection remain unresolved, and the ledger's ≤0.02 GB figure remains an unverified estimate. This ADR does not close that.

Follow-up¶

Related: ADR-024 (CLIMADA isolation — code and packaging, not data), ADR-026 (dependency policy; h5py is BSD-3-Clause, on the allowlist), ADR-059 (never fabricate provenance), issue #618, tracking issue #735, and docs/plan/phase-9-ops-plan.md § OPS-3b.

ADR-064: NetCDF Product Layouts Are Declared in `src/`, Not Registered by Packs¶

Date: 2026-08-08 Status: Accepted Phase: 9

Context¶

ADR-051 landed NetCDF as the second ingest source format. A NetCDF source's layout — the variable to read, its x/y dimension names, the CRS its coordinates are in, and any scenario/time axes — is declared, never guessed (core/ingest/netcdf.py), and layouts live in NETCDF_DATASET_REGISTRY, a module-level dict populated by register_netcdf_dataset().

That registry was deliberately left empty at import, on the reasoning that a layout pins a real product's variable names, which is knowledge the pack that ingests the product owns. Every layout was therefore registered by a scripts/packs/* module at its own import.

NETCDF_DATASET_REGISTRY is in-process module state. The packs run in the api container (./dc exec -T api python scripts/packs/...); the ingest chord runs in the worker container. No module under src/climate_lama/ imports scripts.packs, and celery_app.include lists only climate_lama.worker.*. So the worker's copy of the registry was always empty, and match_dataset_spec — the first thing NetCDFIngestAdapter.validate and .plan_chunks call — always raised InvalidSourceError("no NetCDF dataset layouts are registered").

This is the second of the two worker-side blockers on #761. It is a structural defect, not a missing call: there is no place a pack could put a registration call that the worker process would execute.

Decision¶

Move the layout declarations into src/climate_lama/core/ingest/netcdf_layouts.py and register them as an import side effect. Packs import the declared spec; they no longer author one.

core/ingest/__init__.py imports netcdf_layouts, and every worker task module reaches climate_lama.core.ingest through worker/ingest/adapters.py — so the worker populates the registry by construction, the same way the api container does. This is exactly how INGESTOR_REGISTRY (peril-keyed, core/ingest/unit_compatibility.py) has always worked, and it has never had this problem.

The consequence that decided it: a layout is product knowledge, not job knowledge. One spec describes a product family — every file that CDS dataset ever delivers — not one retrieval. Two jobs ingesting the same product must agree about its variable name, and match_dataset_spec explicitly refuses to resolve an ambiguous match by guessing. Knowledge two processes must share belongs in versioned code that both import.

register_netcdf_dataset() stays public and unchanged: a layout this repo does not declare can still be registered at runtime, and re-registering a declared name with a different spec still raises — correctly, because the worker would not see the difference.

Options rejected¶

1. Import scripts.packs from the worker. Rejected outright: it violates this repo's rule that src/ never imports from scripts/, which exists so the deployable images do not depend on the operator-tooling tree. It would also make worker startup depend on script module side effects (the pack modules mutate sys.path at import and pull in scripts.ingest_scenario_hazards, which reaches the DB layer). Listed in #761 only to be rejected explicitly.

2. Persist the chosen layout in the job params / DB, alongside the existing per-job source_format override (#706). Rejected, but not on principle — it is the closest runner-up:

It adds a serialization surface for a frozen dataclass with Mapping and tuple fields, which must then be re-validated on the far side.
The spec must reach both plan_chunks and every write_chunk: read_netcdf_chunk calls resolve_spec(chunk["dataset"], path) and re-resolves by name from the registry, so the whole spec would have to be stamped onto every planned chunk dict, not just threaded once.
It introduces version skew (a job dispatched by old code, consumed by a new worker) for data that is not job-scoped in the first place — see the "product knowledge" argument above.

Nothing here forecloses it. A per-job override could be layered on top later exactly as source_format was layered onto the peril chain, if a real product ever needs two layouts.

Consequences¶

The worker resolves NetCDF product layouts without importing anything from scripts/. The invariant is structural — an import, not a remembered setup call.
scripts/packs/wildfire_fwi_ewds.py, wildfire_fwi_c3s.py and windstorm_c3s_wisc.py now read their spec, spec name, scenario map, variable and nodata sentinel off the declaration instead of rebuilding them, so the api and worker containers cannot drift.
The wildfire packs' --variable flag can no longer change the ingest layout. It still selects the variable in the CDS/EWDS request; if it disagrees with the declared layout the pack now fails fast (assert_declared_variable) instead of registering a spec the worker will never see. This is a deliberate narrowing: a per-process override of shared product knowledge was never sound. Correcting a variable name is a change to netcdf_layouts.py — one file, both containers.
Every [ASSUMPTION] marker on these layouts is carried over verbatim. None of the variable names or scenario coordinate values has been confirmed against a real downloaded file; CDS/EWDS retrievals are authenticated and queue-based, so no live download ran in the build that made this decision.
A NetCDF ingest is still not end-to-end. This ADR and its sibling fix (making the chord's validate_source step adapter-aware) close both of #761's blockers, and #782 closed the next one — aggregate_and_commit now derives its event axis from the planned chunks, so a scenario × year source commits one intensity-matrix row and one hazard_events row per slice, in ADR-065's scenario × year event shape. Two further gaps were found and are now closed too: the hazard_datasets row the chord commits described a return-period GeoTIFF (return_periods=[0.0], a placeholder supported_years, a single-valued scenario), fixed by ADR-067 (#791); and a single-slice product — three of the four layouts here — had no representable event shape at all, fixed by ADR-068 (#793). No known code blocker remains, but layer 2 of scripts/packs/_wildfire_fwi_common.py::assert_netcdf_adapter_wired stays until a NetCDF ingest has actually run end to end, and docs/plan/phase-9-ops-plan.md's OPS-4a/4c rows still read "blocked".

Follow-up¶

Related: ADR-050 (the adapter seam), ADR-051 (NetCDF as the second format), issues #761, #706, #782, #791 and #793, and docs/plan/phase-9-ops-plan.md § OPS-4.

ADR-065: A Hazard Event Is Return-Period-Shaped XOR Scenario × Year-Shaped¶

Date: 2026-08-08 Status: Accepted Phase: 9

Context¶

ADR-051 declared, in as many words, that for a NetCDF source "the event axis is scenario × year, not return period". The read side implements it: plan_netcdf_chunks (core/ingest/netcdf.py) stamps scenario and year onto every planned chunk alongside its event_index.

The commit side never did. worker/ingest/aggregate_and_commit.py is return-period-shaped end to end — it sizes the intensity matrix with n_events=len(return_periods) and builds every hazard_events row from that same return_periods list, setting exceedance_freq = 1.0 / rp. hazard_events matched it: rp, exceedance_freq and marginal_freq were all nullable=False, so there was no column a scenario × year event could be written into. A NetCDF pack dispatching one source therefore produced n_events == 1 at return period 0.0, i.e. exceedance_freq = 1.0 / 0.0, and a genuinely multi-scenario source tripped _ChunkAccumulator.add's event-range invariant instead.

This is the third blocker on #761, distinct from the two that ADR-064 closed (layout registration in the worker process, and an adapter-aware validate_source). Those were about reaching aggregate_and_commit at all; this one is about what it is allowed to write when it gets there.

The prior question had to be settled first, because it is not a storage question: what frequency does a scenario × year event have? A projection for one (scenario, year) pair is a deterministic realization of a climate model run. It is not drawn from an annual exceedance distribution, and no frequency can be recovered from it.

Decision¶

hazard_events carries two mutually exclusive event shapes, and a named CHECK constraint enforces the exclusion. Migration 0079 adds scenario (String(64), nullable) and year (Integer, nullable), makes rp, exceedance_freq and marginal_freq nullable, and creates ck_hazard_events_event_shape:

return-period-shaped — rp, exceedance_freq, marginal_freq all NOT NULL and scenario, year both NULL; or
scenario × year-shaped — scenario, year both NOT NULL and all three frequency columns NULL.

A row carrying both, or neither, is rejected. event_name and event_index stay NOT NULL on both shapes: an event still needs a label and a row index into the intensity matrix whichever axis it sits on.

A scenario × year dataset is therefore indicator-only, structurally. core/eai_eligibility.py gains a fourth IndicatorReason, NO_EVENT_FREQUENCIES, reached when a caller supplies the dataset's EventAxis. It composes with the existing derivation rather than replacing it: the unit/curve axis (#600, #690) is evaluated exactly as before and keeps the reason field when it fails, while EaiVerdict.event_axis and its lacks_event_frequencies property carry the structural cause alongside. A wildfire/FWI scenario × year dataset is ineligible for both reasons and reports both. The haz_type-keyed eai_grade axis (#604, ADR-060) is untouched and orthogonal, as it already was to posture.

Alternatives Considered¶

1. A sentinel frequency — rp = -1, or marginal_freq = 1.0. Rejected, and it is the alternative the CHECK constraint exists to foreclose. Either sentinel keeps the schema unchanged and lets every existing read path run, which is exactly the danger: the engine would multiply impact by that frequency and return a number typed and labelled as an expected annual impact which is not one. marginal_freq = 1.0 is the worse of the two because it looks defensible — "this realization happens once a year" — and is silently a category error about what the model output is. A sentinel makes the invariant conventional; the constraint makes it structural, which is the difference between a rule a future contributor must know and one they cannot violate.

2. A separate hazard_projection_events table. Rejected. The two shapes share everything that makes an event an event here — org and dataset scoping, event_index as the intensity matrix row, event_name, the uq_hazard_events_dataset_idx uniqueness — and differ only in how the row is labelled on its non-spatial axis. A second table would fork every reader (worker/tasks.py's frequency assembly, the repository, the catalog) on a distinction that a nullable column plus a CHECK expresses in one place.

3. An is_indicator / event_axis column on hazard_datasets. Rejected for the reason core/eai_eligibility.py's module docstring already gives for the posture generally: it would be a second copy of an answer derivable from the event rows themselves, free to drift from them. models/hazard.py::dataset_event_axis derives it instead.

4. Make EAI eligibility read the axis off the dataset automatically. Rejected. It would mean classify_dataset_eai lazily loading a relationship — I/O inside a module that is deliberately a pure function of the values it is handed, and callable from sync and async contexts alike. The axis is passed in by the caller that already holds the events.

Rationale¶

EAI is an integral over an annual frequency distribution. The distribution is the input, not a formatting detail, so a dataset that has none cannot produce the output — and the honest way to say that in a relational schema is to make the frequency columns absent and forbid their absence from being patched over. This reuses semantics the platform already has rather than inventing new ones: #481 established indicator-only perils (no EAI on FWI, ever) and #690 added the eai_grade / eai_grade_note labelling that carries such a verdict onto every surface. What is new is only the cause: previously a dataset was indicator-only because of what its intensity meant, now it can also be indicator-only because of what its events lack.

Safety against existing data. Every pre-existing row satisfies the first branch of the XOR by construction — the three frequency columns were NOT NULL before 0079, and scenario/year are added by it, so they are NULL everywhere. Confirmed against production on 2026-08-08: all 83 hazard_events rows are return-period-shaped. No backfill, and the constraint validates on creation.

What this ADR does not do. It lands the decision and the schema only. aggregate_and_commit still sizes the matrix and writes events from len(return_periods); nothing yet writes a scenario × year row. Until that follow-up lands the new shape is reachable by construction but unreached in practice, and layer 2 of scripts/packs/_wildfire_fwi_common.py::assert_netcdf_adapter_wired stays.

Update (#782, same day). The commit side has landed: aggregate_and_commit derives its event axis from the planned chunks (derive_event_axis) and writes the scenario × year shape for a NetCDF source, so the new shape is now reached in practice. Two findings from that work: a single-slice NetCDF product — one 2-D field, no scenario dimension and no time dimension, which is three of the four layouts in core/ingest/netcdf_layouts.py — satisfies neither branch of the XOR, so the commit step refuses it with an InvalidSourceError rather than fabricate a frequency (#793); and the hazard_datasets row itself still describes a return-period GeoTIFF (#791). Layer 2 of the guard therefore still stays.

Update (#791, next day). The second of those findings is settled by ADR-067: the committed hazard_datasets row derives its own axis columns from the same slice layout the events come from. That ADR also answers the modelling question this one deferred — a dataset row is one ingested source, not one scenario — and revisits alternative 3 below: the reasoning that refused an event_axis column applies to a derivable classification, not to a resolution key the scenario matcher partitions candidates by in a single query. #793 (the single-slice shape) is still open, so layer 2 of the guard still stays.

Superseded in part (#793, same day) by ADR-068. The first finding is settled by widening the second branch of the XOR, not by adding a sentinel: the constraint is now between carrying an annual frequency and carrying none, and a frequency-less row records whichever of scenario/year its source declared — both, one, or neither. The two-shape summary at the top of this ADR's Decision therefore reads, from migration 0081 onward: return-period-shaped, or label-shaped. Everything this ADR forbade for the reason it was written stays forbidden — a frequency beside a label, and a partial frequency triple — and no sentinel frequency was introduced. Layer 2 of the pack guard still stays, now for the evidence gate rather than a code gap: no NetCDF ingest has ever run end to end.

The migration uses typed Alembic operations only (op.add_column, op.alter_column, op.create_check_constraint) per this repo's migration convention, and the constraint is named explicitly so downgrade() can drop it deterministically. downgrade() restores NOT NULL before dropping the new columns, so a database that already holds scenario × year rows fails the downgrade loudly rather than losing a dataset's events: the pre-0079 schema cannot represent them, and deleting them would leave a dataset with an intensity matrix and nothing to index it by.

Follow-up¶

Related: ADR-051 (which declared this axis), ADR-064 (the two sibling #761 blockers), ADR-024 (the engine boundary this frequency array is handed across), ADR-061 (the peril × exposure type axis this composes with), and issues #761, #782 — the aggregate_and_commit event-axis rework that will write the new shape — #481 and #690.

ADR-066: Source Provenance Is Dataset-Scoped; `staging/` Holds Nothing of Record¶

Date: 2026-08-09 Status: Accepted Phase: 9

Context¶

ADR-049 legalized staging/ as working storage — "cleaned up on success, retained on failure for forensic re-runs". Its follow-up (#587) then wrote a manifest.json at two ingest leaves, and put the source one at staging/{ingest_job_id}/manifest.json on the reasoning that staging/ is today's only landing spot for a source file's bytes, standing in for ADR-029's unbuilt raw/{source}/{data-type}/{version}/{dataset-id}/ zone.

Those two statements are incompatible, and the incompatibility was live in production. The success-path purge in aggregate_and_commit derives its delete list from the staged-source descriptor:

staging_keys = [f["key"] for f in (descriptor or {}).get("files", []) if f.get("key")]

The descriptor names what stage_source uploaded. It cannot name manifest.json, which _write_ingest_manifests wrote afterwards in the same function. So every succeeded ingest left a manifest-only directory behind — 16 of them, 794–922 bytes each, found under staging/ during the OPS-5b sweep (#784, #735). Telling those apart from genuine orphans cost a manual cross-reference against ingest_jobs.

The tempting one-line fix — add the manifest key to the delete list — is the one option that must not ship. The staged source bytes are already gone by then, so that file is the only surviving record of what produced the dataset: source, source_url, license, attribution, checksum_sha256, version, volume_bytes. The sibling hazards/{dataset_id}/manifest.json does not substitute for it — that leaf describes the committed intensity.npz, an artifact this platform produced, not the input it was derived from. Deleting the staging manifest would trade a few hundred bytes for the auditable-provenance commitment the platform is built on.

Decision¶

Source provenance moves to hazards/{dataset_id}/source_manifest.json, and the success path then purges the job's entire staging/ prefix.

hazards/{dataset_id}/
├── intensity.npz          # the artifact
├── manifest.json          # describes intensity.npz          (#587, unchanged)
└── source_manifest.json   # describes the source file(s)     (NEW — was staging/{job}/manifest.json)

Three parts:

Provenance is keyed by the thing it describes. A dataset's provenance belongs with the dataset, not with the transient job that happened to produce it. staging/ is addressed by ingest_job_id; answering "where did dataset X come from?" from a job-keyed key requires the ingest_jobs join that OPS-5b had to do by hand. hazards/{dataset_id}/source_manifest.json answers it with a single GET, and survives the ingest_jobs row being pruned. The manifest still carries ingest_job_id so the join back to the job is available, just no longer required.
The manifest now names its inputs. combined_source_checksum digests filenames plus digests, so once the staged bytes are deleted the combined value is neither reproducible nor interpretable on its own. The source manifest therefore carries a source_files list of {name, sha256, bytes} — the per-file record that used to be recoverable by listing the staging prefix. Object keys are deliberately not recorded: they would preserve a path that no longer resolves.
The purge enumerates by listing the prefix, not by walking the descriptor. A descriptor-derived list only covers what stage_source wrote, which is exactly how this defect arose; any future writer under staging/{job}/ would reintroduce it. The purge now lists staging/{ingest_job_id}/ and deletes what is there, so "the prefix is empty after success" holds by construction. When the listing call itself fails, it falls back to the descriptor keys plus the pre-ADR-066 manifest.json key — a listing outage must not strand source bytes.

Failure and cancel still retain staging/ in full. That is deliberate (#296): an operator re-runs a failed ingest from the staged bytes. The purge sits below every early return and below the except block, and both branches return or re-raise before reaching it.

No backfill. Datasets ingested before this ADR keep their provenance at staging/{ingest_job_id}/manifest.json; the 16 directories observed in prod are exactly those records and must not be swept as garbage. Relocating them is optional cleanup, not a correctness requirement, and stays out of scope here — as does the pre-#587 population that has no manifest at either key (ADR-049 already deferred that).

Alternatives Considered¶

Purge the staging manifest too, and stop there. One line, satisfies "staging/ is empty after success", and destroys the only surviving source-provenance record for every dataset ingested after it. Rejected outright: the observable symptom is 800 bytes per ingest, the cost of this fix is the audit trail.

Keep it; close the issue as working-as-intended, and make it operationally legible instead (a documented query or a scripts/ helper that separates provenance manifests from genuine orphans). Honest about the file's value, and the cheapest option in code. Rejected because it leaves an unbounded directory count under a prefix ADR-049 defines as disposable working storage, and pays for it with a permanent operator obligation: every future staging/ sweep has to re-learn the distinction. Tooling that exists to explain a layout is a signal to fix the layout.

Fold a source block into the existing hazards/{dataset_id}/manifest.json instead of adding a second file. One fewer object. Rejected on two counts: source is already a top-level scalar field in that manifest's schema (the provider identifier), so the nesting would collide with the shape build_manifest emits for every other leaf; and the two records have different lifecycles — the artifact manifest is rewritten by any re-commit of the same dataset, while the source record describes an upstream release that is supposed to be pinned literally (ADR-029 invariant 3). Separate keys keep the ADR-029 "manifest describes this leaf's primary data" contract intact for both.

Build ADR-029's real raw/ zone now and write provenance there. The correct end state, and explicitly the larger follow-up ADR-049 defers. Rejected as the vehicle for this fix: it is a key-layout migration with a backfill, gated on decisions this issue does not settle (does the raw zone hold bytes or only metadata? what pins {version} for a user upload?). This ADR is that reconciliation in miniature — it moves one record onto a dataset-scoped key and can be subsumed by the raw zone later without another provenance-losing step.

Rationale¶

The invariant this restores is a separation the codebase already claims: staging/ and chunks/ are working storage; hazards/ is where a dataset's durable record lives. #587 violated it by putting a record of permanent value inside the disposable prefix, and the purge bug was the symptom — a prefix that cannot be emptied is not working storage.

Consequences worth stating plainly:

The success path now issues one list_objects call per ingest. Negligible against a chord that just streamed a dataset twice.
storage_keys.staging_manifest_key is retained but no longer written; it exists so the purge can name the legacy key explicitly on the listing-failure path. Its docstring says so.
staging/ volume after this change is bounded by in-flight and failed jobs only, which is what the Phase 9 volume-budget ledger assumed all along.

Follow-up¶

The 3 genuinely orphaned staging/ directories from the OPS-5b sweep (no matching ingest_jobs row) remain a one-time manual delete under #735. The 16 manifest-only directories are pre-ADR-066 provenance — leave them, or relocate them, but do not sweep them.
The key-layout reconciliation that builds ADR-029's real raw/ zone (ADR-049 follow-up) subsumes this key when it lands.

Related: ADR-029 (the layout and its manifest invariants), ADR-049 (which legalized staging/ as working storage and deferred the raw zone), and issues #784, #735, #587 and #296.

ADR-067: One Dataset Row Per Ingested Source; the Scenario × Year Axis Is Summarised On It¶

Date: 2026-08-09 Status: Accepted Phase: 9

Context¶

ADR-065 settled how a hazard event is shaped, and #782 made the chord write that shape: a NetCDF ingest now commits one hazard_events row per scenario × year slice. The hazard_datasets row those events hang off was never revisited, and it still described a return-period GeoTIFF:

return_periods was stamped [0.0] — both NetCDF packs dispatch a single file with rp=0.0, so the catalog advertised a 0-year return period for an indicator dataset that has none;
supported_years came from a pack-declared placeholder (ProjectionsFwiSpec.horizon_year = 2098) rather than the file's own time axis, even though core/ingest/netcdf.py::summarize_netcdf computes n_slices, scenarios and supported_years — and worker/ingest/validate_source.py::descriptor_with_grid then dropped all three, because its _GRID_FIELDS filter copies only the spatial record;
scenario is a single String(64), while WILDFIRE_FWI_C3S_PROJECTIONS declares a scenario × time cube (3 scenarios × N horizons).

Nothing crashed. The ingest completed and committed slices that nothing downstream could see — which is why it needed its own issue (#791): the failure was silent.

The question that issue put is a modelling one with exactly two candidate answers: one dataset row per scenario, or one row plus a scenario dimension resolution can read. Today's shape supports neither.

Decision¶

A dataset row is one ingested source. The scenario × year axis lives on hazard_events (ADR-065) and is summarised on the dataset row. worker/ingest/aggregate_and_commit.py::derive_dataset_axis derives the row's axis columns from the same slice layout the events are derived from, and _finalize_dataset stamps them:

column	return-period source	scenario × year source
`return_periods`	the RP vector, unchanged	NULL — the source has no return-period axis
`scenarios` (new, migration `0080`)	not written	every scenario label the events cover, in row order
`supported_years`	not written	every horizon year the events cover, ascending
`scenario`	not written	the single label when there is exactly one, else NULL

Three properties make this a decision rather than a set of column writes:

The row and its events are derived from one axis. _is_return_period_axis is the single predicate selecting both the event shape (_event_records) and the row shape (derive_dataset_axis), and both run inside the same commit transaction. A row that says "return periods" while its events say "scenario × year" is not reachable.
The row's axis comes from the source, never from the caller. The values are read off the declared slice layout summarize_netcdf produced and descriptor_with_grid now carries onto ingest_jobs.staged_source (key slice_axis), with the committed event axis as a fallback for a descriptor persisted before that key existed. ProjectionsFwiSpec accordingly drops its horizon_year placeholder: a pack no longer states a horizon it cannot know.

Amendment (#827, 2026-08-11): the source includes its delivery container. ADR-071 established that for sis-heat-and-cold-spells the archive is the scenario axis: each member is one experiment and none of them carries a scenario dimension of its own. Read literally, property 2 then throws away a label the source did state — just in its member structure rather than on an axis — and prod measured exactly that: heatwave-euro-cordex-europe-ssp5-8.5 committed with scenario IS NULL, landing in the partition group resolve_matrix_cell_datasets gives NO_SCENARIO_KEY, which no request addresses. So the property is amended to read: the row's axis comes from the source — including the delivery container whose member structure the source layer resolved a label from — never from the caller.

The channel is deliberately narrow, and it is not the pack's scenario field. That field is free text with a default (heatwave_packs dispatches historical for the ECDE reanalysis), so a fallback keyed on it would stamp a label onto a horizon-agnostic baseline and flip is_horizon_agnostic_baseline false — #805's regression, and the reason core/compute_batch_service.py::resolve_matrix_cell_datasets rejected that fallback in the first place. The label that fills the column is RetrievedMember.scenario_label, non-None only when an ADR-071 scenario_labels mapping resolved it from the delivered archive, threaded on its own params key (member_scenario_label, distinct from scenario) and persisted by validate_source under MEMBER_SCENARIO_LABEL_KEY on ingest_jobs.staged_source — so derive_dataset_axis stays a pure function of persisted job state and a resumed or retried chord derives identically.

Three guards keep the amendment from reopening what property 2 exists to close:

it fills only when the source resolved no scenarios — neither its declared slice layout nor, for a descriptor persisted before that layout was carried, the committed event axis. The same union is what the refusal below tests, so "the file said something about its scenarios" has exactly one definition on both sides of the rule;

an axis carrying an event_label is never relabelled, so a WISC storm-catalogue id stays scenario IS NULL (#816);

a source that declares its own scenario axis and arrives with a member label is refused, never resolved in either direction — so the motivating failure, a caller-supplied label masking a multi-scenario cube committing as one row, stays structurally unreachable rather than merely unlikely.

This is a data-contract correction — catalog truth and scenario-keyed selection — not a compute unblock: ADR-069's frequency exclusion and #804's job refusal both still apply, so no heatwave matrix cell resolves because of it. 3. A multi-scenario cube does not claim to be one scenario. scenario is the key core.compute_batch_service.resolve_matrix_cell_datasets partitions candidates by. Stamping one of a cube's three labels there would let a matrix cell resolve to a dataset whose intensity matrix holds all three — the same class of silent wrongness this ADR exists to remove. NULL is the honest value; scenarios is where the set goes.

This generalises to a single-slice source (the case #793 owns): a source with no scenario dimension and no time dimension yields return_periods, scenarios, supported_years and scenario all NULL — the honest row for one undifferentiated 2-D field. #793 decides whether such a source's event can be written at all; the row shape above does not depend on that answer.

Update (#793, same day). ADR-068 answers that: it can. The event row applies exactly the principle this ADR applied to the dataset row — record what the source declared, NULL what it did not — and derive_dataset_axis needed no change for it.

Alternatives Considered¶

1. One dataset row per scenario. Rejected. It is the more familiar catalog shape — it is what the Aqueduct pack produces — and it would need no resolver change at all, since each row would carry a single scenario and its own supported_years. But it does not fit this source: one NetCDF retrieval is one file, one staged object, one chunk plan, one CSR intensity matrix and one intensity.npz. Splitting it into N dataset rows means either N ingests of the same file (N retrievals, N stagings, N artifacts of the same bytes) or one chord that commits N datasets — a chord that today creates exactly one placeholder row in ingest_hazard and threads one dataset_id through every step, _schedule_cog_build and the #587 manifests included. It also sits badly with ADR-065, decided one day earlier: if a dataset were per-scenario, hazard_events.scenario would hold the same value in every row of every dataset and carry no information.

2. Leave the scenario set on hazard_events only; add no column. Rejected, and this is the alternative that engages ADR-065's own alternative 3, which refused an event_axis column on hazard_datasets as "a second copy of an answer derivable from the event rows themselves, free to drift from them". That reasoning holds for a classification read by a caller that already has the events in hand — which is exactly what classify_dataset_eai is. It does not hold for a resolution key. HazardRepository.list_scenario_conditioned_candidates issues one query for a whole matrix and partitions the result in memory precisely so resolution is not per-cell I/O; deriving each candidate's scenario set from its events would put an event scan back inside that loop, per candidate. supported_years has been on this table since migration 0032 for the same reason. The drift the earlier ADR warns about is closed structurally rather than by convention: both columns are written only by _finalize_dataset, from the same derived axis, in the same transaction as the events.

3. Overload return_periods with a sentinel instead of NULL. Rejected for the reason ADR-065 rejected a sentinel frequency, one level up. [0.0] is what the row carried, and it is not a missing value — it is a false claim that reads as data everywhere it is displayed.

4. Keep scenario populated for a cube and let resolution match it. Rejected — see decision property 3. It would trade an unreachable dataset for a reachable and wrong one.

Consequences¶

Migration 0080 adds hazard_datasets.scenarios (text[], nullable), typed ops only, no backfill: every pre-ADR-065 row is single-scenario or scenario-less by construction, so scenario already carries everything a backfill would write.
A GeoTIFF ingest's dataset row is unchanged. _finalize_dataset writes the scenario/year columns only on the scenario × year branch, so a return-period ingest touches exactly the columns it always did — asserted directly in tests/worker/ingest/test_dataset_axis.py.
descriptor_with_grid now carries a second record. A GeoTIFF summary declares none of SLICE_LAYOUT_FIELDS, so a GeoTIFF descriptor is byte-identical; only a NetCDF entry gains a slice_axis key.
A scenario × year dataset is describable, not yet computable. Two gaps stay open and are deliberately out of this ADR's scope, because neither is a property of the dataset row: worker/tasks.py assembles its frequency vector as np.array([e.marginal_freq for e in events], dtype=np.float64), which is all-NaN for such a dataset; and core.compute_batch_service resolves a matrix cell to a whole dataset, not to one of its slices, so it has no way to say "this cell is row 4 of that cube". Until both land, a scenario × year cube resolves for no cell at all — scenario is NULL for it — which is the safe failure, and layer 2 of scripts/packs/_wildfire_fwi_common.py::assert_netcdf_adapter_wired refuses every NetCDF dispatch anyway. Filed as #797 so closing #793 — the last commit-side blocker — does not silently make a compute-side one reachable.
The pack guard's cited blocker moves from #791 to #793. It has now cited #706 → #761 → #782 → #791 → #793 in turn; ADAPTER_GAP_ISSUE_URL on trunk is the authority, not this list.
No NetCDF ingest has been run end to end. This decision was made by reading the chord, not by observing one — CDS/EWDS retrievals are authenticated and queue-based, and no credentialed download ran in the build that made it.

Superseded in part, 2026-08-10. The two consequences above are historical. A real NetCDF ingest has now run end to end in production (heatwave, windstorm and wildfire rows are live), and PR #825 deleted layer 2 of assert_netcdf_adapter_wired, its --first-end-to-end-run escape, and the ADAPTER_GAP_ISSUE_URL constant itself — so "ADAPTER_GAP_ISSUE_URL on trunk is the authority" no longer names anything that exists. The surviving guard performs the plan-time adapter-resolution check only. Read scripts/packs/_wildfire_fwi_common.py on trunk.

Follow-up¶

Related: ADR-065 (the event shape this summarises), ADR-064 (where a layout is declared), ADR-051 (which declared the axis), and issues #791, #782, #793, #761, #614, #615 and #616.

ADR-068: An Event Carries an Annual Frequency XOR Whatever Labels Its Source Declares¶

Date: 2026-08-09 Status: Accepted Phase: 9

Context¶

ADR-065 gave hazard_events two shapes behind ck_hazard_events_event_shape: a return-period row (rp + both frequencies, no labels) or a scenario × year row (both labels, no frequencies). A row carrying neither was rejected — that rejection is what made "a deterministic event has no annual frequency" structural rather than conventional.

It also made one very ordinary product unwritable. Three of the four layouts in core/ingest/netcdf_layouts.py — WILDFIRE_FWI_EWDS_HISTORICAL, WINDSTORM_C3S_FOOTPRINTS_HIST and WINDSTORM_WISC_SYNTHETIC — declare no scenario_dim and no time_dim, so core.ingest.netcdf._enumerate_slices yields exactly one slice with scenario=None, year=None. That slice has no return period and no (scenario, year) pair, so it satisfies neither branch, and aggregate_and_commit refused it with an InvalidSourceError (#793) — correctly, given the constraint, but after the authenticated retrieval, the staging and every chunk write.

The affected products are the platform's most straightforward NetCDF cases, not exotic ones: an FWI reanalysis field, a windstorm gust footprint. They are indicator perils (#481) whose value is "render this field on a map", which needs no frequency at all. Under ADR-065 they were permanently uningestable.

Decision¶

ck_hazard_events_event_shape (migration 0081) is now an exclusive-or between carrying an annual frequency and carrying none. A frequency-less row records whichever labels its source declared — both, one, or neither.

(rp IS NOT NULL AND exceedance_freq IS NOT NULL AND marginal_freq IS NOT NULL
 AND scenario IS NULL AND year IS NULL)
OR (rp IS NULL AND exceedance_freq IS NULL AND marginal_freq IS NULL)

source's declared axis	`scenario`	`year`	`event_name`
return periods (every GeoTIFF)	NULL	NULL	`RP-100` (unchanged)
`scenario_dim` + `time_dim` (a cube)	set	set	`ssp5-8.5@2050` (unchanged)
`scenario_dim` only	set	NULL	`ssp5-8.5`
`time_dim` only	NULL	set	`2050`
neither (one 2-D field)	NULL	NULL	`field`

Three things make this a decision rather than a relaxation:

Nothing ADR-065 was written to forbid became writable. A frequency beside a label is still rejected; a partial frequency triple is still rejected. There is no way to put a sentinel rp next to a scenario, which was that ADR's entire purpose. What was dropped is only the requirement that a frequency-less row carry two specific labels.
The widening has structural teeth. uq_hazard_events_dataset_single_field is a partial unique index on dataset_id over exactly the rows carrying neither a frequency nor a label, so a dataset holds at most one. "Unlabelled" is honest only for a source whose whole non-spatial axis is one field; a second such row can only mean a derivation lost labels it should have kept. aggregate_and_commit._slice_records refuses the same case one layer up, before any write. (The predicate must name the frequency columns: without them it would match every return-period row too — those carry no labels by the first branch — and cap every GeoTIFF dataset at one event.)
It is the rule ADR-067 already applies to the dataset row, applied to the event row. ADR-067 settled that a single-slice source's hazard_datasets row carries return_periods, scenarios, supported_years and scenario all NULL — record what the source declared, NULL what it did not. Its event now follows the same rule, and derive_dataset_axis needed no change at all, which is the check that the two halves really are one principle.

EventAxis stays two-valued. It answers the only question any reader puts to it — does this dataset carry an annual frequency distribution? — which core/eai_eligibility.py asks to reach IndicatorReason.NO_EVENT_FREQUENCIES. How far the source labelled its axis is carried by which columns are NULL, exactly as ADR-067 carries it for the dataset row, so an unlabelled dataset is EAI-ineligible for the same structural reason a cube is, with no new branch to forget.

Alternatives Considered¶

1. Keep the loud refusal; treat single-field products as out of scope. Rejected. It is the cheapest option and the issue lists it, but it permanently strands three of four declared layouts and both windstorm packs (#615), for products that are the easiest NetCDF cases the platform has. The refusal also sits at the worst possible point — after the retrieval, the staging and every chunk write — so keeping it would have meant moving it earlier as well, to a static check on the layout. That is real work spent to enshrine "this cannot be ingested".

2. Synthesise the missing labels at ingest time — write scenario="historical" and a year taken from the retrieval's reference date, both of which the dispatching pack knows (HistoricalFwiSpec.horizon_year, spec.scenario_label). Rejected on two grounds. It contradicts ADR-067 decision property 2 — the axis comes from the source, never from the caller — which is exactly why ProjectionsFwiSpec dropped its horizon_year placeholder one day earlier; reinstating a pack-declared horizon on the event axis would put back what #791 removed from the dataset row, one level down. And it cannot cover the products it is proposed for: WINDSTORM_WISC_SYNTHETIC is a synthetic event set with no calendar year in any sense, so its "year" would be pure invention. A fabricated label is the same category of error as a fabricated frequency — "a false claim that reads as data everywhere it is displayed", in ADR-067's words — just cheaper to overlook because nothing computes with it.

3. A third named shape, DETERMINISTIC_FIELD, admitted only when every axis column is NULL, with scenario-only and year-only still rejected. Rejected. It is tighter, and it would keep EventAxis a one-to-one mirror of the constraint's branches, but it refuses two shapes that are perfectly ordinary — a single-scenario product with a time axis is the most likely real shape of the EWDS reanalysis this issue is about, since WILDFIRE_FWI_EWDS_HISTORICAL's "no time dimension" is an [ASSUMPTION] about a file nobody has opened. Refusing shapes on the strength of an unverified assumption, when a real download could invalidate it next week, buys tightness with the wrong currency.

Update 2026-08-09 (#802): it did — within two days. The first real cems-fire-historical-v1 retrieval carries fwinx(valid_time, latitude, longitude), so WILDFIRE_FWI_EWDS_HISTORICAL now declares time_dim="valid_time" and commits a year-only row, exactly the shape alternative 3 would have refused. Correcting the layout was a one-line change to core/ingest/netcdf_layouts.py and needed no constraint or migration change.

4. A sentinel — rp = 0, or marginal_freq = 1.0, on the unlabelled row. Rejected, and restated here only because it is what the pre-#782 code actually did: both NetCDF packs dispatch a single file with rp=0.0, which persisted exceedance_freq = 1/0. ADR-065's rejection of sentinels is untouched by this ADR and is the reason its first branch is unchanged.

Consequences¶

Migration 0081 drops and re-creates the CHECK (typed ops: op.drop_constraint, op.create_check_constraint) and adds the partial unique index (op.create_index with postgresql_where). No backfill: the new predicate is strictly weaker, so every existing row passes, and the index can only collide on rows the old constraint made impossible. downgrade() restores 0079's predicate verbatim, which fails loudly on a database already holding an unlabelled or half-labelled row rather than deleting a dataset's only event — the posture 0079's own downgrade takes. tests/test_db/test_hazard_event_shape.py asserts 0081's frozen copies match the model and that its restore literal matches 0079's.
A GeoTIFF ingest is untouched. _return_period_records is byte-identical, the first branch of the constraint is unchanged, and the new index cannot match a return-period row.
aggregate_and_commit._scenario_year_records is renamed _slice_records: it no longer writes only the scenario × year shape.
Still not computable, and still guarded. A frequency-less dataset remains describable and not computable for the reasons ADR-067 records — #797, now enforced on the read side by ADR-069 — and layer 2 of scripts/packs/_wildfire_fwi_common.py::assert_netcdf_adapter_wired stays, now for a different reason than every previous time. ADAPTER_GAP_ISSUE_URL moves from #793 to #761, and the guard's message no longer names a code blocker: there is no known one left. What is left is that no NetCDF ingest has ever run end to end — no CDS/EWDS retrieval has been downloaded, staged, chunked and committed in any environment. The guard has cited #706 → #761 → #782 → #791 → #793 in turn, five successive "last" blockers each found by reading the next step of the chord; the honest reading of that sequence is that code review is not evidence. Removal is gated on a first successful retrieval.
This decision, too, was made by reading the chord rather than observing one. No credentialed download ran in the build that made it, and every layout it reasons about is still an [ASSUMPTION].

Follow-up¶

Related: ADR-065 (the shape this widens), ADR-067 (the same principle at the dataset row), ADR-064 (where a layout declares — or omits — its non-spatial dimensions), ADR-051 (which declared the axis), and issues #793, #791, #782, #789, #797, #761, #614 and #615.

ADR-069: A Frequency-Less Dataset Is Refused at Compute and Never Resolves a Matrix Cell¶

Date: 2026-08-09 Status: Accepted Phase: 9

Context¶

ADR-065, ADR-067 and ADR-068 between them made a frequency-less hazard dataset writable and describable: its hazard_events carry whatever labels the source declared and no frequency at all, and its hazard_datasets row NULLs return_periods to say so. Each of those ADRs states that such a dataset is not computable. None of them made the read side enforce it, and #797 found two places where the platform would have computed it anyway.

1 — the worker's frequency vector degraded to NaN instead of raising. worker/tasks.py built it identically in compute_impact and compute_cost_benefit:

frequency = np.array([e.marginal_freq for e in events], dtype=np.float64)

marginal_freq is NULL on every frequency-less event by construction, and np.array([None], dtype=np.float64) yields array([nan]) rather than raising. The engine integrates the NaN frequencies and returns a NaN EAI, stored and labelled as an expected annual impact — the exact failure ck_hazard_events_event_shape exists to foreclose, reappearing one layer down because the read side never learned about the new shape.

2 — the compute gate could not reach the reason that describes it. core/compute_service.py::validate_impact_refs did call classify_dataset_eai(hazard_ds), but without event_axis, so ADR-065's IndicatorReason.NO_EVENT_FREQUENCIES was structurally unreachable from the only gate that matters. FWI datasets were refused anyway — on the unit axis, for an unrelated reason — which is precisely why the gap was invisible: a peril with a seeded curve on a consumable unit, a river-flood or windstorm cube, would have passed.

3 — matrix resolution addresses a dataset, not a slice. core/compute_batch_service.py::resolve_matrix_cell_datasets returns one hazard_dataset_id per (scenario_label, horizon_year) cell. A cube holds every cell's slice in one intensity matrix, so there is no way to say "this cell is row 4". A multi-scenario cube did not resolve — but only because hazard_datasets.scenario is NULL for one and that column is the partition key. A single-scenario multi-year cube carries both a scenario and supported_years and resolved cells exactly, straight into problem 1.

Decision¶

A dataset whose events carry no annual frequency is refused at every compute entry point, and a matrix cell may not address one slice of a cube. A cube stays non-resolvable — now by rule, not by side effect.

Four parts, all read-side; no schema change and no migration.

validate_impact_refs passes the dataset's EventAxis into classify_dataset_eai, which is what makes NO_EVENT_FREQUENCIES reachable at all. It raises the new HazardDatasetNoEventFrequenciesError (422 E_HAZARD_DATASET_NO_EVENT_FREQUENCIES), separate from IndicatorDatasetNotComputableError because the two are different facts with different follow-ups: one says no curve consumes this unit (source a different intensity), the other says there is no distribution to integrate (there is nothing to source — read the score card). When both axes fail, classify_eai still gives the unit reason precedence, so an FWI dataset keeps the refusal and the error code it already had.
The axis costs one bounded read, not an event-set load. HazardRepository.get_event_axis selects a single row ordered marginal_freq DESC NULLS FIRST, so a frequency-less row surfaces whenever the dataset holds one and an arbitrary return-period row surfaces when it does not — exactly the discrimination models.hazard.dataset_event_axis needs, from one indexed row instead of N. The row is still passed through dataset_event_axis rather than read directly, so anything that satisfies neither shape answers SCENARIO_YEAR, the conservative side.
The worker refuses again, immediately before building the vector (_event_frequency_vector, now shared by both call sites). Not redundant with the gate: the worker also serves jobs the gate never saw — a replayed or resumed job, a batch cell dispatched internally, a dataset whose events changed after submission — and a NaN EAI is not a failure mode worth leaving to a single checkpoint.
Matrix resolution subtracts frequency-less candidates from the pool, via one further whole-matrix query (dataset_ids_without_event_frequencies), and names them in the cell's resolution_error rather than letting them vanish into "no dataset carries that scenario".

Why "a cube stays non-resolvable" rather than "a cell may address (dataset_id, event_index)":

Every consumer downstream of the cell takes a dataset id. compute_impact loads the dataset, its events and its whole intensity matrix; the result row, the cache key, the provenance record and the surface writer all key on hazard_dataset_id. Admitting a slice reference means threading an event index through all of them — and answering, at each, what a partial dataset means for a sha256, a cache hit and a stored result.
It would compute the wrong thing anyway. The slice would still carry no annual frequency. A matrix cell reports an EAI; a slice of a cube has none to report, whichever row is chosen. Slice addressing solves the addressing problem and leaves the arithmetic problem exactly where it was.
The rule is enforced against the event axis, not inferred from the dataset row. Reading return_periods IS NULL would have been free, but it is a summary column (ADR-067) that legacy rows predate; the events are where the axis actually lives, so that is what is read.

The honest reading of a scenario × year cube is that it is indicator data with a time axis — render it, score it, compare slices on a map — and the surfaces that do that (GET /v1/risk/lookup, the tiles, the catalog) already work off the same rows without touching a frequency. What is refused here is only the claim that an expected annual impact can be computed from it.

Alternatives Considered¶

1. Let a matrix cell address (dataset_id, event_index). Rejected on the two grounds above — it is a large refactor across every compute consumer, and it would still produce no valid EAI. Worth revisiting only if a cube ever needs to feed a non-EAI matrix (a scenario comparison of raw intensities), which is a different feature with a different result shape.

2. Fabricate a frequency for the deterministic slices — marginal_freq = 1.0, or 1/N over the slices. Rejected, and it is ADR-065's alternative 1 verbatim. It makes every number downstream a number, and none of them an expected annual impact.

3. Gate only at submission, not in the worker. Rejected. The worker is the last point before a NaN is written to impact_results as an EAI, and it serves jobs the API gate never inspected. A guard that only exists on the path someone remembered is the shape of this bug.

4. Filter frequency-less candidates inside list_scenario_conditioned_candidates. Tempting — it costs no extra query. Rejected because the exclusion then becomes invisible again: the cell's message would read "No dataset in this catalog carries scenario 'ssp2-4.5'" while a dataset carrying exactly that scenario sits in the catalog. #631 already paid for that lesson once.

5. Read hazard_datasets.return_periods IS NULL instead of the events. Rejected: it is a derived summary, written only by the post-ADR-067 commit step, and the column has been nullable since migration 0002. A legacy row that never recorded one would be misread as a cube and silently dropped from every matrix.

Consequences¶

No schema change, no migration, no backfill. Everything here is read-side. Alembic head is unmoved.
A return-period dataset's compute path is byte-identical — same frequency vector, same matrix resolution, same verdict — and is asserted so by tests on all three surfaces.
One extra bounded read per impact submission (get_event_axis) and one extra whole-matrix read per scenario matrix (dataset_ids_without_event_frequencies). Both are indexed on dataset_id; neither scales with the event set.
E_HAZARD_DATASET_NO_EVENT_FREQUENCIES joins the ADR-028 registry, maps to 422, and has an SDK exception (HazardDatasetNoEventFrequenciesError) so a client can branch on it rather than string-match.
Nothing observable changes today. No frequency-less dataset exists in any environment: no NetCDF ingest has ever run end to end, and layer 2 of scripts/packs/_wildfire_fwi_common.py::assert_netcdf_adapter_wired still refuses every dispatch. That is the point of landing this now — the guardrail must precede the data, not follow the first NaN EAI into a stored result.
Read from the code, not observed, exactly like the three ADRs above it.

Follow-up¶

Related: ADR-065 (which forbade the sentinel this enforces), ADR-067 (the dataset row's axis columns), ADR-068 (the event shape this reads), ADR-061 (the other axis the same gate enforces), and issues #797, #793, #791, #789 and #631.

ADR-070: The Raw Zone Is Transient by Design; the Source Manifest Is the Reproducibility Artifact¶

Date: 2026-08-10 Status: Accepted Phase: 9

Context¶

Phase 9's exit criterion 3 (docs/plan/phase-9-data-foundation.md) reads: "Every dataset in the catalog has: raw-zone copy + manifest, license + attribution recorded, working tiles, dataset-attributed lookups/results." It assumes the object store's raw/ prefix accumulates a retained copy of every source file the platform ingests.

It does not, for any of the ten Phase 9 packs (#611–#620), and never has. Verified live against the Hetzner bucket during the 2026-08-08 ops run (OPS-5d, recorded on #735 and in docs/data/volume-budget.md): the raw/ prefix holds exactly two objects — .keep and catalog.v1.json, ~0.03 MB — and the whole bucket measures 0.325 GB, against the volume ledger's raw-column sum of ≈24.64 GB. That gap is not drift and not a bug to chase; it is what the pipeline is built to do. Every pack streams its source download to local disk, clips or processes it there, ingests the result into DB rows or the processed zone (hazards/{dataset_id}/intensity.npz, processed/{org}/{dataset_id}/hazard.tif), then deletes the local download. No code path in the platform copies a source file into raw/.

Leaving the criterion as written has a concrete cost: the Full-Catalog walkthrough's step 6 audits it, so the phase exit would fail on a condition unmet by construction for every pack — inviting either a waived criterion (the Phase 8 failure mode) or a scramble to build raw archiving that nothing has asked for.

Separately, ADR-066 already moved source provenance to hazards/{dataset_id}/source_manifest.json and made a successful ingest purge staging/ wholesale — so the reproducibility record already exists, dataset-scoped and durable, independent of any retained bytes.

Decision¶

The raw zone is a transient staging concept, not an archive. Source bytes are fetched, processed and discarded; the platform retains the derived artifacts plus a manifest, not the input.

The per-dataset source manifest is the reproducibility artifact. hazards/{dataset_id}/source_manifest.json (ADR-066) — source URL/dataset id, request parameters, checksums, retrieval timestamp — is what makes an ingest reproducible. A re-fetch from the recorded source is the recovery path, not a restore from raw/.
Exit criterion 3 is amended to require source manifest + license + attribution recorded, working tiles, dataset-attributed lookups/results — dropping "raw-zone copy". The criterion text in docs/plan/phase-9-data-foundation.md is edited to match, so the Full-Catalog walkthrough audits something achievable.
The volume ledger's raw column measures transient peak local-disk headroom, not object-storage cost. docs/data/volume-budget.md's column heading and caption say so explicitly, and its Totals row must never be read as a storage bill.

Alternatives Considered¶

Implement raw archiving to satisfy the criterion as written. Rejected: it would add ≈24.6 GB of Phase-9-scope object storage — growing unbounded under the pan-EU expansion #447 contemplates — to retain bytes that are re-fetchable from a recorded, licensed public source. It also imports a licence problem: several packs' sources are redistribution-limited (see #620's unresolved Eurostat population-grid terms), and storing the raw file sits closer to redistribution than storing a derived clip.
Waive the criterion for Phase 9 and leave the text alone. Rejected: Phase 8 exited under a waived walkthrough criterion and carried four defects forward. A structurally unmeetable criterion should be corrected, not waived.
Keep raw for the packs that clip locally, skip it for windowed HTTP-range reads. Rejected as an inconsistent rule that would make "does this dataset have a raw copy?" unanswerable without knowing each pack's fetch strategy.

Rationale¶

Reproducibility is a provenance property, not a retention property. What a reviewer needs to re-derive a dataset is an exact, checked record of what was fetched and how — which the source manifest provides at ~1 KB per dataset instead of gigabytes. Retaining the bytes would prove only that the same bytes were retained.

The criterion was written before the packs existed; the packs then independently converged on stream-clip-delete, because that is what the phase doc's own Risks section pushes toward ("the raw zone is a cache with a budget line per pack, not a mirror of the internet"). The written record is what is out of date here, not the implementation.

Consequences¶

No code change, no schema change, no migration. This ADR records and ratifies existing behaviour; raw/ keeps its .keep and catalog.v1.json and stays otherwise empty.
Phase 9 exit criterion 3 is met by the manifest, and the Full-Catalog walkthrough's step 6 audits hazards/{dataset_id}/source_manifest.json rather than a raw/ copy.
Pre-ADR-066 datasets carry their provenance in staging/{ingest_job_id}/manifest.json instead — 16 such files remain in the bucket and are the only source provenance those datasets have. OPS-5b must not delete them.
The ledger's raw column and Totals row are re-labelled as transient peak local-disk headroom. The ops plan's disk-check steps still consume those figures correctly — headroom is exactly what they were checking.
Loss of a source upstream is a real, accepted risk. If a provider withdraws or silently revises a file, the platform holds the derived artifact and a manifest describing what was fetched, but not the original bytes to diff against. The manifest's checksum makes divergence detectable; it does not make the original recoverable.

What would be required to reverse this¶

If raw archiving is ever wanted, this ADR is superseded by a successor establishing, at minimum:

A licence review per pack confirming the source's terms permit retaining and re-serving the original file, not merely a derived product. #620's Eurostat population-grid terms are unresolved today and would have to be settled first.
A retention policy with a budget line — how long a raw object lives, what evicts it, and the object-storage cost at pan-EU scale rather than Greece-first.
A write path: the packs' fetch helpers (fetch_clip, fetch_fwi_netcdf, the pack-local downloaders) would each upload to raw/{source}/{data-type}/{version}/{dataset-id}/ before clipping, and the manifest would gain a pointer to that key.
A backfill decision for datasets already ingested — re-fetch from their manifests, or accept a mixed estate where only post-decision datasets have raw copies.

Follow-up¶

Related: ADR-066 (the source manifest this decision leans on), ADR-049 (the staging prefix and key convention), and ADR-029 (the bucket layout that defines the raw/ zone this ADR reclassifies). Tracking issue: #735. Ledger: docs/data/volume-budget.md. Amended criterion: docs/plan/phase-9-data-foundation.md.

ADR-071: A Delivered CDS Archive Is Unpacked in the Source Layer, One Ingest Per Member¶

Date: 2026-08-10 Status: Accepted Phase: 9

Context¶

Every Copernicus CDS retrieval this project has made came back as content_type: application/zip — four accepted jobs across three collections (#815), including one that selected a single experiment and still zipped its one member. The retrieval path wrote those bytes straight to a .nc destination, so the staged "NetCDF" file was an archive and core.ingest.netcdf._open_dataset would have rejected it. The EWDS cems-fire-historical-v1 retrieval (#802) came back as a bare application/netcdf, so zipping is a product behaviour, not a protocol one — an adapter cannot assume either shape.

Unpacking is not purely a decode step. For sis-heat-and-cold-spells the archive is the scenario axis: its two members are HWD_EU_health_rcp45_mean_v1.0.nc and HWD_EU_health_rcp85_mean_v1.0.nc, one per requested experiment, and neither carries a scenario dimension of its own. Whatever unpacks it therefore has to decide whether one archive becomes one dataset or one dataset per member.

Decision¶

A retrieval's delivered asset is unpacked in the shared source layer, detected by zipfile.is_zipfile, and a multi-member archive produces one ingest per member, each with its own scenario label.

Detection is on the bytes. Never on content_type (it varies per product) and never on the .nc suffix (the destination always has one today). climate_lama.ingest.sources.cds_archive holds the whole rule; a bare asset is returned untouched, with no copy and no rename, so the working EWDS path is unchanged by construction.
The unpack lives one layer above the byte transfer. CdsApiClient.download stays byte-faithful — it owns resumable Range transfers and the file:size check, both of which a rewrite would break — so CdsRetrieval performs the unpack, on both its download branch and its raw-zone cache-hit branch. Every consumer therefore gets it for free: CopernicusCdsSource.download/download_members and the packs' fetch_* helpers. The rejected alternative was a per-pack unpack, which three packs would have copy-pasted and which would have left the raw-zone cache-hit path (which never touches a pack's fetch code) still handing a zip to the ingest.
One member, one ingest. The alternative — one dataset for the whole archive — leaves the committed row's scenario NULL, which ADR-067 and #808 show is unaddressable by any matrix cell: the data would land in the catalog and be unreachable. Per-member datasets are also simply truthful — each member is a distinct scenario. A multi-member item's datasets are named {item_id}-{scenario_label}; a single-member item keeps the bare id, so nothing already catalogued is renamed.
Labels come from the request, not from a filename regex. A pack declares scenario_labels: request value ("rcp4_5", a published experiment enum member) → platform scenario label ("ssp2-4.5", a seeded row of the scenario_labels catalog). Matching normalises both sides to lower-case alphanumerics, which is what bridges the request's rcp4_5 to the file's rcp45 without pattern-matching the whole name. The RCP→SSP mapping is the forcing-level equivalence core/scenario_labels.py already records, not a new claim.

Note (#827, 2026-08-11): that label reaches the committed row. The first real heatwave ingest showed the label surviving every dispatch step and then being discarded at commit, because ADR-067 property 2 derived the row's scenario from the file alone and no member declares a scenario dimension. Property 2 is amended (see its own section) so that a label resolved here — from the delivered container's member structure — counts as source-derived, because point 3 below already says the archive is this product's scenario axis. RetrievedMember.scenario_label therefore rides its own params key member_scenario_label, kept distinct from the pack's free-text scenario all the way down: only a label with this provenance may fill hazard_datasets.scenario, which is what keeps a pack default off a #805 baseline row. A member label colliding with an in-file scenario axis refuses the ingest — consistent with point 5's "nothing is ever dropped, and nothing partial is ever written".

Nothing is ever dropped, and nothing partial is ever written. A member no request value matches, a member two values match, a multi-member archive with no mapping, an empty archive, and two members colliding on one output name are all refusals. Labels are resolved for every member before a single byte is extracted, and a pack resolves every dispatch before performing any, so an incomplete archive ingests none of its experiments rather than the subset that happened to be understood — the #803 failure mode.
Archive content is untrusted. A member whose name is absolute, carries a drive letter, uses a backslash separator, or contains a .. component is refused rather than quietly flattened: only the basename is ever used, so a traversal cannot succeed, but a member that attempted one is evidence the asset is not what the pack thinks it is.

Consequences¶

dest keeps the delivered bytes, so the raw-zone cache and the file:size check are unchanged; members are unpacked flat beside it as {dest.stem}__{member basename}, because every pack maps a staged file into the worker's view of the data dir by Path.name alone. Unpacking is idempotent — a member already present at its full size is not rewritten.
CdsRetrieval.run_bounded and CopernicusCdsSource.download now return the usable path, which for a CDS asset is the unpacked member rather than dest, and raise CdsArchiveError on a multi-member archive instead of picking one. run_bounded_members/download_members are the multi-member API.
New exception CdsArchiveError (subclass of CdsApiError, so existing handlers still catch it).
This does not unblock a NetCDF ingest on its own: assert_netcdf_adapter_wired's evidence gate (#761) still refuses every dispatch until one has run end to end, via the one-shot --first-end-to-end-run escape.

Follow-up¶

Related: ADR-067 (the committed row's axis columns, which a per-member ingest still derives from the file), ADR-064 (where a member's layout is declared). Issues: #815 (this decision), #808 (why a NULL scenario is unaddressable), #761 (the gate that still stands in front of the first real run).

ADR-072: One Upstream Distribution Has One Catalog Identity; Per-Source Overrides Reconcile the Ingest Paths¶

Date: 2026-08-14 Status: Accepted Phase: 9

Context¶

ADR-048 gave hazard_datasets a provider / upstream_version identity, and #887 gave the approve→ingest handoff a way to fill it: DatasetSource.attribution(metadata), derived strictly from what the provider itself publishes. Before that, every dataset the handoff created landed with its whole provenance NULL.

That derivation is the right default and it is also, on its own, insufficient. The same upstream distribution is reachable by two ingest paths, and for JRC EFAS river flood — the one distribution both paths actually ingest — they disagree (#900):

field	auto-ingest (#887)	hand-run pack (`scripts/packs/dataset_identity.py`)
`provider`	`JRC`	`Copernicus Emergency Management Service / European Commission Joint Research Centre`
`upstream_version`	`2024-03-20T11:33:56` (`metadata_modified`)	`v3.1.1`

Neither value is wrong. dcterms:publisher on the live package_show payload really is …/corporate-body/JRC, and v3.1.1 really is nowhere in that payload — the pack takes it from scripts/historical_catalog_manifest.json's hand-written source name, which is not reachable from source metadata at all. #887 declined to invent the long form, correctly.

What is wrong is two of them for one distribution. The UI's provenance chip renders upstream_version verbatim (ui#116/#117), so one row would read v3.1.1 and its sibling an ISO timestamp — two datasets, to a user. And the scenario-matrix resolver pins a provider family across cells (ADR-048) by string equality — core/compute_batch_service.py's candidate.provider == base_provider — so a split provider for one producer can empty the family a cell resolves through. (core.dataset_precedence reads the column too, but only scores whether it is set, so the split costs nothing there.)

Decision¶

The catalog's existing identity is canonical, and the auto-ingest path adopts it through a declared per-source override table.

Direction: auto-ingest adopts the pack's values, not the other way round. The catalog is already built out of the pack-side strings — 20 of production's 21 active hazard rows carry a provider from scripts/packs/dataset_identity.py (the wildfire EWDS row is the one that never emitted it), the JRC row among them — so this direction makes an auto-ingested row comparable with the row already in the catalog, which is the entire point of the exercise, and needs no data migration. The reverse direction would have to re-stamp those live rows and would put a catalogue-bookkeeping timestamp in a user-facing chip; metadata_modified moves when a curator edits a description, while v3.1.1 is the provider's own product release designation.
The override is data, not a branch. ATTRIBUTION_OVERRIDES in src/climate_lama/ingest/sources/attribution.py is a frozen tuple of AttributionOverride entries keyed by (source_id, item_id). Nothing inside any attribution() implementation knows the table exists: a source stays a faithful reader of its own provider. Adding a normalisation is a data edit; the mechanism never grows a per-provider if.
An override replaces exactly what it declares. Field by field: a field the entry is silent about keeps whatever the source stated, and an entry can never blank a stated value (an undeclared field is None, and None means "undeclared", never "clear it"). An item with no entry — the normal case — is returned untouched.
native_resolution is not declarable. The handoff measures it off the source raster's own transform while the clip has the file open (core.ingest.clip.RasterClip). A measurement beats a constant transcribed from a datasheet, and core.dataset_precedence buckets this value — finer wins — so a hand-written one silently reorders which dataset a matrix cell resolves to. The two agree for JRC EFAS anyway (measured 3 arc-seconds vs the pack's recorded 0.000833333), which is what makes leaving it measured free.
The seam is DatasetSource.resolved_attribution(item_id, metadata), in the ingest layer. attribution() remains the source-derivation hook a subclass overrides; resolved_attribution() is what an ingest caller uses, and it is the only thing worker/dataset_ingest.py calls. Putting it on the ABC rather than at the worker call site keeps the reconciliation in the layer that owns it and one call away from any future consumer. It is not a hard guarantee — attribution() stays public and callable — so the wiring itself is pinned by a test that drives the real ingest_approved_dataset task and fails if it ever calls attribution() again.
Every entry carries evidence. Same rule scripts/packs/dataset_identity.py and docs/data/volume-budget.md already enforce on their own numbers: each value names the manifest field, pack constant or delivered filename it came from. An override without traceable evidence is a fabrication with extra steps — and an override is more dangerous than a derivation, because it overwrites something the provider actually said.

Note on ADR-048's provider wording. That ADR's illustrative slugs ("jrc", "aqueduct") were never what shipped: every producer in this repo — packs and handoff alike — stamps a human-readable producer name, and 21/21 production rows carry one. This ADR records the long producer name as the canonical provider value, which is what the column already contains; the "machine-readable, distinct from the free-text source label" intent is unchanged, since source still carries the adapter id (jrc_flood).

Alternatives Considered¶

Normalise inside JrcFloodSource.attribution() — map the publisher URI to the long name there. Rejected: it is a branch, not data, and it is a lie about the payload. attribution() is defined as "what this provider states about this item", and a reader comparing the method against the live package_show response would find a string that appears nowhere in it. The next source needing the same treatment would copy the branch.

Have the backbone import scripts.packs.dataset_identity so there is literally one table. Rejected: scripts/ is not shipped in the worker image, and that module is deliberately dependency-free so the packs and the backfill tool stay cheap to import. The values are therefore restated in attribution.py and pinned against the pack table by tests/test_ingest/test_attribution_overrides.py — the same anti-drift arrangement tests/scripts/test_dataset_identity.py already uses for that module's own restated constants.

Flip the packs to the derived values instead. Rejected: it re-stamps 21 live catalog rows to gain nothing (they would still have to agree with something), and it degrades the user-facing version string from a release tag to a metadata edit timestamp.

Match overrides by name prefix, the way PackIdentity does. Rejected: a prefix rule would silently claim the next package an operator adds to dataset_source_jrc_flood_package_ids. An override is a statement about one distribution whose identity somebody verified, so it names its item ids explicitly. Matching is case-insensitive and space-trimmed only, and a duplicated or empty entry raises at import rather than resolving to whichever entry was listed first.

Consequences¶

A JRC EFAS approval now dispatches provider / upstream_version identical to the pack's, pinned by tests/test_worker/test_dataset_ingest_identity_parity.py at both ends: the params _create_ingest_jobs builds, and the params the whole ingest_approved_dataset task actually dispatches. Both compare against scripts.packs.dataset_identity.identity_for.
citation and source_url still differ in wording between the two paths (the pack cites the manifest's reference and collection URL; the source derives the dataset's own dcterms:bibliographicCitation and DOI). Both are honest, neither is an identity key, and neither is rendered as a version. They are left alone; the table can declare them the day that changes.
An auto-ingested row is still named by the provider's title ("River flood hazard maps for Europe…"), not by the pack's catalog name, so scripts/backfill_dataset_identity.py's name-prefix matcher will not claim it. That is harmless — the row now arrives with its identity already stamped, which is what the backfill exists to repair.
IngestAttribution moved from ingest/sources/base.py to ingest/sources/attribution.py (the override table cannot import base, and base re-exports it, so every existing import path is unchanged).

Follow-up¶

Related: ADR-048 (the identity columns and their readers), ADR-056 (which compares provider). Issues: #900 (this decision), #887 (the attribution seam it builds on, which deferred exactly this table), #864 (why the handoff states provenance at all).

ADR-073: Every Address-Level Band Reads the Point; a Resolved Cube Slice Is a Band Input¶

Date: 2026-08-15 Status: Accepted Phase: 9

Context¶

ADR-043 gave each peril a score_schemes ladder, and RFC amendment A3.3 gave each one a declared input: metric_source = 'point_intensity' (the hazard COG's value at the coordinate) or 'cell_metric' (a key out of a risk cell's metrics blob). core/scoring.py's _BAND_SOURCE_BY_HAZARD holds the per-hazard rule and BandingScheme.from_bands enforces it — a row whose declared source disagrees is rejected as metric_source_precedence.

A3.3 split the perils on physics: flood, wildfire, coastal and earthquake are spatially discontinuous at sub-cell scale (a floodplain edge, a fire perimeter, soft basin sediment against rock over a few hundred metres), so a hex aggregate is a different answer and the point read is mandatory. Windstorm, tropical cyclone and heatwave are smooth at hex scale, so A3.3 permitted them the cell aggregate, and issues #602/#603/#604 classified them that way.

#924 found that every one of those cell_metric ladders is dead in production — four rows after #928 split the heatwave ladder per definition (storm_europe/max_wind_gust_ms, tropical_cyclone/max_sustained_wind_ms, heatwave/heatwave_days_health, heatwave/heatwave_days_climatological). Two findings, and the second is the load-bearing one:

There is no write path for the key, and there cannot be one on the current architecture. core/surface_writer.py is the sole production writer of a cell's metrics, it writes exactly {eai, eai_density_km2, points}, and it is driven from an impact run's eai_exp. All three perils are indicator perils with no seeded damage curve, so core/compute_service.py refuses their impact runs outright. No run → no surface → no cell → no blob to write a key into. Prod confirms it: org_risk_surfaces holds river_flood and earthquake rows and nothing else.
Flipping metric_source alone would have changed nothing user-visible. _build_card takes its band value from _band_input, which accepted a sample only when sample.return_period == risk_lookup_band_return_period (100.0). These are all ADR-067 slice-axis datasets, committed with return_periods = None, so label_samples labels every sample return_period=None and no sample can ever match. point_intensity is therefore structurally None for exactly these perils, and the migration on its own would have swapped METRIC_UNAVAILABLE(cell_metric) for METRIC_UNAVAILABLE(point_intensity). The wildfire FWI ladder — already point_intensity — is dead for this same second reason.

Decision¶

Every wired peril bands its address-level card from the point intensity, and a resolved cube slice counts as a band input even though it carries no return period. Two halves, shipped together:

_BAND_SOURCE_BY_HAZARD classifies all seven wired perils as POINT_INTENSITY, and migration 0085 moves the four cell_metric rows to match. Neither half is optional: the map is what from_bands validates a row against, so a migration without the code change makes those rows unloadable and the card reports no_scheme_for_hazard instead of banding.
_band_input accepts the value of a resolved slice, with return_period = None. The guard is exactly "resolve_slice addressed a slice" — SliceAddress sets exactly one of band and unavailable_reason, so a non-None band is the signal — plus a structural single-sample check. It is never satisfied by select_point_bands' FIRST_BAND fallback, because banding that would answer a horizon=2050 question with the 1986 slice of a 1986-2085 cube, which is precisely defect #844.

A3.3 is narrowed, not contradicted. It makes a point read mandatory for discontinuous perils; it never forbids one for smooth perils. A pixel read is strictly more precise than the hex aggregate those three were entitled to, and it is the read that actually exists today.

CELL_METRIC stays a valid enum member. Rollups band from cell aggregates for every peril — band_rollup_if_enabled deliberately never calls resolve_metric_value — and a future peril whose only address-level input is genuinely a cell aggregate would be classified that way.

band_return_period may now be None on a banded card. It is provenance about the band, not a precondition for one. No API or SDK consumer reads it as a flag; it is serialized onto intensity.band_return_period and rendered as-is.

Alternatives Considered¶

(a) Teach surface_writer to write the per-hazard cell metric (e.g. heatwave_days) alongside the EAI keys. Rejected as structurally impossible for the peril this is mostly about. The writer was never the blocker — the compute gate is: with no seeded curve there is no run to hang a write off, and surface_writer refuses an empty build. Beyond that, its entire input is SurfacePoint(lat, lon, eai), which holds no intensity; the intensity that exists is centroid-indexed (n_events × n_centroids), exactly the indexing surface_writer forbids (the

370 trap); and a surface covers exposure locations only, so the result would be intensity sampled¶

at exposure points, not a hazard field. It would also require rebuilding every existing cell.

Superseded in part by ADR-075 (run-backed perils only). Each of the four grounds above was answered rather than overturned: the compute gate still holds — ADR-075's scope is exactly the perils that can run; the #370 rule still holds, because the intensity[event, centroid] join happens in the worker frame that already performs it for the engine and the value crosses into the writer positionally; and "exposure points, not a hazard field" is conceded and named — the run-backed metric is the hazard where the book is, and the unconditioned field remains alternative (c)'s job. Only "rebuild every cell" is genuinely deferred, to #943. This rejection stands for every indicator peril, which is still every peril ADR-073 was mostly about.

(c) Introduce a distinct indicator-peril surface path that builds cells from the hazard raster directly rather than from impact points. Not rejected — deferred, because it is a new subsystem and the card is answerable today without it. It is the honest fix for the rollup path, which is dead for all nine schemes and which metric_source cannot help (see Follow-up).

Flip metric_source and stop there. Rejected: finding 2 above. It would close the issue on a false green while every affected card kept reporting band_metric_unavailable.

Relax _band_input to accept any unlabelled sample. Rejected: that is #844 re-created. An unaddressed cube read is FIRST_BAND, and banding it reports one arbitrary year's value as the answer to a question about another.

Consequences¶

A heatwave, WISC-windstorm or wildfire-FWI card that addresses a horizon the cube actually carries now bands. One that does not still reports a reason, unchanged.
tropical_cyclone still cannot band, for an unrelated reason: no pack catalogues a TC dataset (scripts/packs/dataset_identity.py), so there is no COG to read. Its row is corrected here so it will band the day one lands, rather than being correct-looking and dead.
Migration 0085 touches metric_source only. No ladder, unit, version or citation moves, so nothing is re-banded and no version bump is warranted — metric_source says which input a ladder reads, not what a threshold means.
Migrations 0057 and 0084 still carry inline comments stating the old A3.3 classification. They are left as-written: a migration records the reasoning that applied when it ran, and 0085 is the revision that supersedes it.

Follow-up¶

Issues: #924 (this decision, and the _band_input half — it stays open for the criteria below), #932 (the rollup path: it is dead for all nine schemes including point_intensity ones, because band_rollup_if_enabled reads cell.metrics directly and metric_source is irrelevant to it — plus the invariant test pinning that some writer produces every scheme's metric), #933 (surface a specific unavailable reason rather than a generic band_metric_unavailable), #925 (heatwave band recalibration, untouched here). Related: ADR-043 (what a band is), ADR-067 (the slice axis this reads), ADR-074 (what an absent reading of a resolved slice means).

ADR-074: An Absent Cell of a Count-Indicator Peril Reads as Zero¶

Date: 2026-08-16 Status: Accepted Phase: 9

Context¶

ADR-073 made a resolved cube slice a band input, which is what finally lit the heat-wave ladder. Grounding #925 then found that most of the ladder still never fires, for a reason that is upstream of banding entirely: #935.

The platform is sparse-first (#436). core/ingest/base.py's keep-mask stores "cells worth storing: neither nodata nor zero", the CSR the COG writer densifies from therefore holds no zero cells at all, and the written COG carries nodata = 0.0. So titiler returns null for a pixel that recorded a zero — the two are the same byte by the time a card reads them, even though the source NetCDF distinguishes them (_FillValue is NaN over sea and mask).

For a continuous intensity that convention is exactly right. A flood depth of zero is not a shallower flood; it is dry land outside the footprint, and the river-flood raster legitimately has data at only ~2% of its pixels. Nothing useful can be said there, and a stated non-answer is the honest card.

For a count index it destroys information. "How many heat-wave days at this coordinate in 1985?" has the answer zero — a measurement, not a gap. Measured read-only on the prod ECDE grid (86 yearly bands, heatwave-ecde-europe), the share of the Europe grid holding a stored non-zero value is 4.8% for a 1971-2000 reference year, 25.5% for 2015-2025 and 36.0% for 2022-2025. So in a reference year roughly 95% of Europe answered no_bandable_reading where the truthful answer was Negligible, and band 1 of every count ladder was unreachable by construction — including the recalibrated v2 ladder of #925.

Decision¶

An absent reading of a count-indicator peril is the value 0, and 0 bands. Decided at the lookup layer, on the peril, with no re-ingest and no migration:

core/scoring.py gains _ABSENT_READS_AS_ZERO and absent_reads_as_zero() beside the existing _BAND_SOURCE_BY_HAZARD. heatwave is its only member today. Unlike the precedence table it is deliberately not exhaustive and does not raise on an unclassified peril: membership is the exception, non-membership is the default, so a new HazardType keeps the stated non-answer until someone decides otherwise.
_band_input substitutes 0.0 only on the resolved-slice branch. The guard is unchanged and still load-bearing: a slice resolve_slice actually addressed, narrowed to exactly one sample. A null there means titiler answered inside the raster's own extent and found nothing stored — a point outside the extent is an HTTP ≥400 that becomes cog_read_failed and never reaches this function, and a failed read leaves samples empty and never reaches it either.
The card states which it was. _band_input returns a third element, BandValueSource (read | absent_as_zero), carried on HazardCard.band_value_source and serialized as intensity.band_value_source beside intensity.band_value. band.metric_source keeps reporting point_intensity, because it names the input class the scheme bands from — a property of the ladder, not a claim that a pixel answered. band_value_source is the authoritative marker and the one a client must branch on.

The raw reading is not rewritten. intensity.band_value becomes 0.0 while intensity.by_return_period[…].value stays null. The convention supplies a band input; it does not invent a measurement, and a payload that claimed the raster returned zero would be a worse lie than the one this ADR removes.

Alternatives Considered¶

(a) Ratify the conflation and reword the reason ("no hazard signal at this location for this year"). Rejected: it leaves band 1 of every count ladder permanently unreachable and answers a question the platform's own convention already answers — the reason string would be explaining a non-answer to a question that has an answer.

(b) Write a real nodata sentinel and keep zero pixels in the COG. Rejected as not implementable as stated. The COG is densified from the sparse CSR and the CSR holds no zero cells: aggregate_and_commit.py filters row_values != 0 at CSR build, core/ingest/netcdf.py keeps only kept-mask cells per event, and base.py drops NaN and zero identically. (b) therefore means reversing #436 for these perils and re-ingesting every count-indicator dataset — and hazard_datasets.footprint is only the grid envelope, so no land mask survives ingest to help. It remains the only route to distinguishing sea from land (see Consequences) and is not ruled out forever; it is simply not what this question is worth today.

Key "count indicator" off metric_unit == "days". Rejected: that unit is free text on both the scheme row and the dataset row, so a typo would silently change what a card reports. The hazard-keyed table follows the _BAND_SOURCE_BY_HAZARD precedent and is checked by a test over the whole enum.

Apply the convention to every peril (river-flood dry land → Negligible). Coherent, and explicitly left open — but it reverses documented behaviour for the perils where absence really does mean "nothing to say", so it wants its own decision rather than riding in on this one.

Put the marker on BandedScore. Rejected: that dataclass is shared verbatim with every rollup rung, none of which has a reading to qualify, so the field would be permanently null in three payloads to serve one.

Consequences¶

A heat-wave card at a coordinate that stores nothing, for a horizon the cube carries, now returns band 1 / Negligible with intensity.band_value = 0.0, intensity.band_value_source = "absent_as_zero", a null raw sample and no band_unavailable_reason. This is a user-visible behaviour change on the peril, not only a new field.
The asset rung inherits it. GET /v1/risk/assets/{id} projects the address card verbatim (_hazard_from_card), so that peril arrives with a real band and aggregate_value = 0.0, and the asset's headline score moves accordingly. provenance.band_value_source is carried up beside band_source so the rung does not imply a pixel answered. Portfolio and admin-unit rungs read cell.metrics and are untouched (#932).
Sea and masked pixels inside the raster extent also band Negligible. The source's NaN land mask is discarded at ingest, so the card cannot tell "no heat-wave day here" from "this is the Aegean". Acceptable for a heat-wave-days indicator queried at an address, and recoverable only via alternative (b). Stated here rather than discovered later.
The #844 guard is untouched and outranks this: an unaddressed cube still yields no band input with the convention on, because there is no slice for the zero to be the answer of.
Nothing is re-ingested, no migration runs, no scheme version moves, and no SDK is regenerated (/v1/risk/lookup has no response model and the drift check watches errors.py only). Adding intensity.band_value_source is additive and does not bump schema_version.
no_bandable_reading (#933) becomes rare on heat-wave: it now means the horizon addressed no slice, not that the pixel was empty.

Follow-up¶

Issues: #935 (this decision), #925 (the band edges themselves — independent of this, and the reason band 1 being reachable matters), #944 (deferred indicator-peril rollup surfaces — any future heat-wave cell write must carry this decision rather than re-decide it). Related: ADR-043 (what a band is), ADR-073 (the resolved-slice branch this rides on), ADR-075 (the cell half — a cell with no valued point still gets no key, so this convention does not reach a rollup).

ADR-075: A Run-Backed Surface Carries Its Design-Return-Period Intensity¶

Date: 2026-08-16 Status: Accepted Phase: 9

Context¶

Both rollup rungs band from a cell's metrics blob — the portfolio rung in core/rollup_service.py's _score_members / _portfolio_hazard, the admin-unit rung through RiskSurfaceRepository.aggregate_cells_in_boundary's c.metrics ->> :metric. The only production writer of that blob, core/surface_writer.build_cells, wrote exactly {eai, eai_density_km2, points}. No seeded score scheme's metric was ever among them, so rollup banding was dead for all nine seeded schemes — and so were the numbers beside the bands, since aggregate_value and worst_value are derived from the same read.

This is not what ADR-073 fixed. ADR-073 moved every address-level ladder onto the point read; band_rollup_if_enabled deliberately never consults that precedence table, so a scheme's metric_source is irrelevant to the rollup path and a point_intensity ladder such as river_flood/flood_depth_m was exactly as dead as an indicator peril's. ADR-073 named the honest fix as its deferred alternative (c) — a distinct indicator-peril surface path that rasterizes a hazard COG into H3 cells — and left it unbuilt because it is a new subsystem.

Two facts reopened the question:

The impact worker already holds the answer, in the same frame that saves the surface. After the engine call, worker/tasks.compute_impact still has the run's (n_events × n_centroids) intensity matrix, the dataset's events in event_index order, and every exposure row with its assigned centroid. intensity[design_event, row.centroid_idx] is the design-return-period hazard at each exposure point, at the cost of densifying one CSR row.
The subsystem is not on the demoable MVP slice. It serves the P1 insurer's portfolio workbench (#367), which CLAUDE.md marks conditional pending a checkpoint expected autumn 2026.

Decision¶

A run-backed surface carries the design-return-period hazard intensity at each of its exposure points, aggregated per cell under the active rollup scheme's metric key. Concretely:

One design-return-period resolver. core/scoring.design_return_period_for(hazard_type, settings) is the single answer to "which slice of the distribution is a band of". Both address-level card consumers in core/lookup_service and the writer call it, so a card and the cell beneath it cannot band different return periods. Its per-hazard override table shipped empty at first — the seam existed, the divergence did not.

Addendum (#942, 2026-08-16). The table is no longer empty: HazardType.EARTHQUAKE: 476.0. Prod's ESHM20 dataset publishes nominal return periods [50, 476, 976, 2500, 5000] (scripts/packs/earthquake_eshm20.py:107-125) and no 100-year event, so the global default could never match and every earthquake card and cell banded band_metric_unavailable / metric_absent_from_cells. 476 is ESHM20's own nominal label for the ~475-year (10%-in-50-years) reference seismic design return period — an exact match against the pack's stated table, not a tolerance window against the live probability it is close to but not identical with. intensity.band_return_period is therefore hazard-varying in the payload (100 on a flood card, 476 on an earthquake card, None on a cube slice) — provenance-only, as ADR-073 already established for band_return_period being None. 2. Exact match, no nearest neighbour. A dataset publishing no event at the design return period yields no intensities and writes no key, exactly as the card reports no band. Prod's ESHM20 earthquake dataset publishes [50, 476, 976, 2500, 5000] — no 100-year event, so it stayed unbanded at every rung until #942's override above resolved its design return period to 476. 3. The estimator is stated, not implied. A cell's value is the mean over its valued exposure points; <metric>_points records how many those were. The boundary rung's mean_value is weighted by that count, so a one-asset hex cannot outvote a thousand-asset one; worst_value stays MAX over cell means and is "the worst cell average", not the worst point. A cell with no valued point gets no key, so the rungs' existing metric_absent_from_cells reason keeps meaning what it says. 4. A unit gate on the write. The key comes from rollup_metric_key_for_run — the rung's own hazard-only scheme selection, plus a check that the scheme's metric_unit matches the dataset's intensity_unit. Wildfire's hazard-only tie-break resolves to the Kelvin ladder, so without the gate an FWI run would file fire-weather indices under temperature thresholds. Rung-side narrowing by unit or index definition is out of scope here and belongs with (c). 5. The boundary aggregate dedupes surfaces. Its key predicates pin neither region nor the engine's identity, so several READY surfaces can satisfy them — production holds six for one river_flood/Greece/baseline/0 identity. A DISTINCT ON (h3_index) ... ORDER BY created_at DESC CTE now picks one surface per hex before aggregating, the same tie-break the point path already applied per coordinate. This was dormant only because no cell carried a value. 6. Scope: computable perils. River flood today; storm Europe and tropical cyclone the day a dataset is catalogued; coastal flood when #833 seeds a curve; earthquake when #942 lands. Indicator perils get nothing here — heat-wave (both ladders) and wildfire/FWI have no run to hang a write off, and remain ADR-073 alternative (c), filed as #944. 7. The exclusions are derived, not listed. core/scoring.rollup_metric_exclusion answers why a seeded scheme has no producible metric, from the same tables the compute gate uses — is_indicator_peril (no seeded curve) and seeded_units_for (the scheme's unit is not one a curve consumes) — so wiring a peril removes it from the excluded set with no edit anywhere. Only "the curve and the unit exist but no seed pack catalogues a dataset" is declared, as _NO_DATASET_SCHEMES (tropical_cyclone/max_sustained_wind_ms, wildfire/brightness_temperature_k): whether a pack exists is not a fact the shipped package holds, because scripts/ is not in the API or worker image.

This supersedes ADR-073's alternative (a) in part — for run-backed perils only. Each of its four rejection grounds is answered above rather than overturned, and it stands unchanged for every indicator peril. ADR-073 is stamped accordingly in place.

Alternatives Considered¶

Build ADR-073's alternative (c) now — rasterize the COG into H3 cells. Deferred to #944, not rejected: it is the only route for the indicator perils, and the only one that yields an unconditioned hazard field. But it is a multi-week subsystem (strip budgeting, nodata-weighted aggregation, a cell cap that is EAI-ranked and therefore meaningless for a hazard field, a region/slice/identity axis for surfaces with no run behind them) built for a persona the project's own plan marks conditional. Shipping it in preference to a field the run already computed is the failure mode the collaboration framework names: high issue throughput, zero demoable product.

Sample the COG at each existing cell's centroid, and backfill. Rejected on accuracy. An r8 hex is ~460 m across against 100 m flood pixels, so the centroid is routinely dry while the asset is in the floodplain — a different and worse estimator than the per-point intensity the run already consumed, and seeding one column from two estimators makes every stored value ambiguous.

A single ST_PointOnSurface(b.geom) read per boundary. Expressible — the boundary aggregate already computes ST_Centroid — and it costs one round trip against a 1200 ms budget, so an N-round-trip cost objection does not bind it. Rejected anyway: for river flood, whose JRC raster has real data at ~2% of pixels, the representative pixel is almost always nodata. A sample of one for a whole district.

Per-member point reads at the portfolio rung. Rejected: it breaks the stated "no COG reads at all" composition of the portfolio budget, which is pinned by a test that a 200-member portfolio must not issue 200× the queries of a 1-member one. Moot in any case once members' cells carry the metric.

A literal table of which schemes are excluded. Rejected: it goes stale by construction — #833 would have to hand-edit it — which is precisely how an exclusion list becomes a place to hide a regression. Replaced by the derived predicate in decision 7.

Per-hazard design return periods in this decision. Rejected as scope: changing earthquake's design return period changes what every earthquake card means, not just its cells. Split to #942, with the resolver seam left here so the change is one map entry.

Rationale¶

The rollup metric a user wants is "how bad is the hazard across this book", and the run already answered it for every point in the book. Deriving it from a second source — a COG read at a hex centroid, or at a boundary's representative point — would be more code, more latency, and a different number from the one the loss beside it was computed from.

Conditioning is the honest cost, and it is stated rather than hidden: a surface covers exposure locations, so this metric is the hazard where the book is. That is the same conditioning the cell's EAI already carries, and it is the right answer to the portfolio question. It is not the right answer to "how hot is this district", which is why (c) is deferred rather than cancelled.

Consequences¶

Both rollup rungs light up for river flood — the one peril with real production surfaces — with no read-path change, no titiler dependency, no new subsystem, and no migration. Earthquake, the only other peril with a prod surface, stays metric_absent_from_cells until #942.
The portfolio rung is untouched: no query-count change, no COG read, and its value-weighted mean over members is unchanged. Only the boundary rung's SQL changed, and only to dedupe surfaces and weight the mean.
Existing surfaces do not gain the metric until they are re-saved. Re-running a key overwrites its cells in place, so an ordinary re-run is the fix; a bulk backfill from stored results is #943.
Scheme resolution is coupled to read time. A cell is written under the key active at write time and read under the key active at read time, so bumping a scheme's version to a new metric, or seeding a second ladder that wins the hazard-only tie-break, darkens existing surfaces until they are re-saved. That is the same recurrence #943 addresses, and it is the price of not denormalising a scheme id onto every cell.
eai, eai_density_km2, points and any *_points key are now a reserved namespace in the cell blob; a scheme whose metric collided with one is refused at write time rather than silently overwriting the surface's own accounting.
provenance.aggregation on the admin-unit payload changes from "cell_mean" to "points_weighted_cell_mean", and gains surface_selection. Additive/renamed provenance only — no response model, no schema_version bump, no SDK regeneration (the drift check watches errors.py).
The demo seed writes its own flood_depth_m cells directly through replace_cells; once org-plane cells carry the metric the org plane answers first, so the demo rollup's number comes from the run rather than from the fixture. scripts/seed_demo.py is untouched here and is deleted by #892.

Follow-up¶

Issues: #932 (this decision), #942 (earthquake's design return period — until it lands, earthquake bands at no rung), #943 (backfilling the existing surfaces), #944 (ADR-073 alternative (c) — the indicator-peril surface path, still the only route for heat-wave and wildfire/FWI), #833 (a coastal-flood curve, which flips coastal into the producible set with no code change), #924 (the card-path half). Related: ADR-043 (what a band is), ADR-047 (the portfolio rung this does not touch), ADR-073 (superseded in part, above), ADR-074 (what an absent card reading means).