Skip to content

Phase 3 — Foundations

Status: Closed 2026-04-23. All 12 work items merged. See plan.md for the phase overview. Scope: Tier 1 items from the Phase 3+ vision draft. Predecessor: Phase 2. Successor: Phase 4 — Map stack.

Goal

Before building anything new on top of the Phase 2 production backbone, freeze the load-bearing architectural commitments (map library, dependency policy, adapter-by-design, error taxonomy, object-storage layout) as ADRs, stand up the docs surface that will carry them, and land the data-model primitives (Scenario, reproducibility metadata, open-data attribution, monitoring instrumentation) that every later phase depends on. Nothing user-visible ships in this phase beyond a docs site and a "save/load scenario" flow — it is deliberately foundational.

Entry conditions

  • Phase 2 closed (auth, RBAC, multi-tenancy, structlog, production Docker, UI repo split all merged).
  • climate-lama-engine pinned to a 0.3.x release on PyPI.
  • Backbone Postgres schema on latest Alembic head.

Out of scope (deferred to later phases)

  • MapLibre migration of UI components — Phase 4.
  • Zoom-aware aggregation, MVT/COG tile serving, map rendering performance — Phase 4.
  • Admin panel, dataset browser, external dataset polling, async ingest pipelines — Phase 6.
  • i18n, XLSX translation workflow, Playwright smoke suite, tooltip glossary — Phase 5.
  • First non-default adapter (CLIMADA compute engine, GEM ingest) — Phase 7.
  • Reports (PDF export, custom templates, export bundles) — later phases.

See phase-3-vision-draft.md priority ordering for the full Tier 2/3/4 backlog and which phase picks each one up.

Scope

Each work item below is issue-ready. The block at the end of each item is the input to /analyze: it names the files, gives a suggested tier, and lists acceptance criteria that /build can verify against.

3.1 — ADR-025: MapLibre commitment + library whitelist

Context. Round 3 resolution committed to MapLibre GL from the start (not a deferred migration), with the guardrail that only well-maintained, actively adopted libraries enter the dependency tree. This ADR freezes that decision and pins the whitelist so Phase 4 can execute without re-litigating the choice.

Acceptance criteria. - [ ] docs/DECISIONS.md gains ADR-025 following the existing template (Date, Status, Context, Decision, Alternatives, Rationale, Follow-up). - [ ] Decision documents: MapLibre GL core (BSD-3), native MVT support, and the plugin whitelist: @maplibre/maplibre-gl-inspect, @mapbox/mapbox-gl-draw (used against MapLibre), base-tile provider candidates (MapTiler, Stadia, self-host). - [ ] Alternatives section covers: staying on Leaflet, Leaflet + VectorGrid, Mapbox GL v2 (license issue), deck.gl. Each rejected with a one-line reason. - [ ] Follow-up section cross-references the Phase 4 map-stack work items. - [ ] **Phase**: 3 header line added to the ADR body.

Cross-stack: backbone docs only. No code change. Linked ADR: authors ADR-025. Suggested /analyze tier: 2 (Sonnet 4.6 · medium) — pure prose, existing template, no code.

3.2 — ADR-026: Dependency policy

Context. The engine stays lean (20KB vs CLIMADA's 500MB) as an explicit stance; the backbone and UI have no equivalent policy. Gap-13 in R3 flagged this; A7 resolved "yes, write the ADR." Without a policy every added dep is re-argued on the PR.

Acceptance criteria. - [ ] ADR-026 added to docs/DECISIONS.md. - [ ] Policy codifies: (a) every new runtime dep needs a one-paragraph rationale in the PR body; (b) activity / maintainer / adopter criteria (last commit within 6 months, test coverage, production users); (c) preference for stdlib or already-pulled-in deps before adding a new one; (d) license allowlist (Apache 2.0, MIT, BSD-2, BSD-3). - [ ] Scope clarified: dev dependencies (ruff, mypy, pytest-*) bypass the full rationale requirement but still hit the license allowlist. - [ ] Follow-up: add a PR-template checkbox "new runtime dep? link the rationale" in backbone, engine, and UI repos. - [ ] **Phase**: 3 header line.

Cross-stack: backbone docs, plus PR-template edits across three repos as a Follow-up (small cross-repo PRs). Linked ADR: authors ADR-026. Suggested /analyze tier: 2 (Sonnet · medium).

3.3 — ADR-027: Adapter-by-design architectural commitment

Context. Vision idea #32 (and the user's direction in R4 B6) commits the platform to adapter-extensible architecture across compute engine, ingest formats, report template engines, dataset sources, and auth providers — without overengineering. This ADR names the extension points, the in-process vs remote boundary, and the packaging rule (separate climate-lama-adapter-* repos, backbone declares optional extras).

Acceptance criteria. - [ ] ADR-027 added to docs/DECISIONS.md. - [ ] Enumerates the five extension points with their current status: compute engine (ModelInterface exists), ingest formats (needs ABC), report template engines (needs abstraction), dataset sources (needs ABC), auth providers (already abstracted via FastAPI dependency). - [ ] Packaging rule: first-party adapters live in their own repos (climate-lama-adapter-climada, climate-lama-adapter-gem, etc.); backbone declares optional extras (pip install climate-lama[climada]). - [ ] Security boundary: in-process Python adapters are trusted; remote webhook adapters (#24 Level 3b) stay Tier 4 with mandatory result validation; Python package reference at runtime (#24 Level 3a) is rejected. - [ ] Follow-up: each extension point gets its own sub-ADR as it's implemented. - [ ] **Phase**: 3 header line.

Cross-stack: backbone docs only. Linked ADR: authors ADR-027. References ADR-024 (CLIMADA isolation). Suggested /analyze tier: 3 (Sonnet · high) — five extension points, packaging + security analysis, more synthesis than the first two ADRs.

3.4 — ADR-028: Error taxonomy

Context. Gap-7, promoted to Tier 2 in R3. Backbone currently raises internal exceptions; the UI surfaces generic messages; SDKs (future) would have no stable error contract. A catalogued error-code system (E_HAZARD_INTENSITY_UNIT_MISMATCH, E_EXPOSURE_OUT_OF_BBOX, etc.) unblocks i18n of errors (#15), SDK quality (#I), and consistent UI presentation of error/warning/info.

Acceptance criteria. - [ ] ADR-028 added to docs/DECISIONS.md. - [ ] Defines the error envelope: {code: str, severity: error|warning|info, message_en: str, details: dict}. Codes are SCREAMING_SNAKE_CASE with an E_ prefix; W_ for warnings; I_ for info. - [ ] Message strings stay English in API responses; UI translates by code. Codes never change once published; deprecation via alias, not rename. - [ ] Initial catalogue of ~20 codes covering the most common existing failure modes (enumerate them in the ADR body). Full taxonomy grows over time. - [ ] Registry lives at src/climate_lama/core/errors.py — enum + message table. HTTP exception handler maps internal exceptions to codes. - [ ] Follow-up: retrofitting existing raises is a Phase 5 work item (tied to gap-7 UI presentation). This ADR only freezes the shape. - [ ] **Phase**: 3 header line.

Cross-stack: backbone docs. Phase 5 is where the retrofit + UI lands. Linked ADR: authors ADR-028. Suggested /analyze tier: 3 (Sonnet · high) — taxonomy design requires judgment; 20 initial codes means reading the existing exception surface.

3.5 — ADR-029: DO Spaces bucket layout + dataset versioning

Context. R4 B2 settled the bucket layout (raw/{source}/{data-type}/{version}/{dataset-id}/ plus processed/, tiles/, reports/ prefixes, mandatory manifest.json at every leaf). This ADR freezes the layout adjacent to the Phase 6 catalog work (#19) and the Phase 4 tile-serving work (#27b) so neither can drift.

Acceptance criteria. - [ ] ADR-029 added to docs/DECISIONS.md with the full layout tree from the vision draft's B2 resolution. - [ ] Invariants documented: {dataset-id} is the catalog UUID join key, manifest.json mandatory at every leaf, {version} pins upstream labels literally, tiles are regenerable, org data stays under processed/{org_id}/. - [ ] manifest.json schema specified (JSON Schema in an appendix or separate file): {source, source_url, fetched_at, checksum_sha256, license, crs, units, dataset_id, version}. - [ ] Follow-up points at Phase 4 tile-serving, Phase 6 catalog + admin UI, Phase 6 ingest pipeline. No infrastructure work in this ADR — just the layout + manifest schema. - [ ] **Phase**: 3 header line.

Cross-stack: backbone docs. Linked ADR: authors ADR-029. Suggested /analyze tier: 3 (Sonnet · high) — layout + manifest schema is bounded but needs precision.

3.6 — Docs site scaffold (MkDocs Material)

Context. Vision idea #17, Tier 1. Q6 resolved MkDocs Material. The docs site also hosts the ADR index (with the Phase column proposed in the structural discussion), the OpenAPI integration, and — per the landing-site resolution — doubles as the initial marketing landing page via MkDocs Material's landing-page feature.

Acceptance criteria. - [ ] docs/mkdocs.yml added at repo root of backbone with Material theme configured. - [ ] docs/ site structure: index.md (landing-styled hero), quickstart/, concepts/ (glossary, domain terms), api/ (auto-generated from OpenAPI via mkdocs-swagger-ui-tag or equivalent), architecture/ (ADR index + linked ADR pages), plan/ (mirror of docs/plan/ for public readers). - [ ] CI job builds the site on PR; deploy workflow publishes on tag to GitHub Pages (or Cloudflare Pages if the user prefers — decide during /build). - [ ] docs/DECISIONS.md stays the canonical ADR ledger; MkDocs includes it via include-markdown or symlink, not duplication. - [ ] Existing docs/CLIMATE_LAMA.md ported into the site structure under architecture/; no content rewrite, just layout. - [ ] First-page landing copy: hero + three feature bullets + CTA to quickstart. Marketing polish lives in Phase 7 (#F Astro split).

Cross-stack: backbone (adds docs/mkdocs.yml, CI workflow). No runtime code changes. UI repo unaffected. Linked ADR: none directly; implements the outcome of vision Q6. Suggested /analyze tier: 3 (Sonnet · high) — non-trivial because of MkDocs config + CI workflow, but no architectural judgment.

3.7 — Scenario model (data + API + minimal save/load UI)

Context. Vision #A, Tier 1. Currently the backbone has impact_results but no top-level user-facing Scenario entity. Every later phase (#14 reports, #20 cache, #B compare, #P versioning) depends on this. The minimum viable version is the data model + save/load endpoints + a thin UI surface to exercise it.

Acceptance criteria. - [ ] Alembic migration adds scenarios table: id UUID PK, org_id FK, created_by_user_id FK, name TEXT, description TEXT, inputs JSONB (exposure_id, hazard_id, impact_fn_id, year, scenario_label, measures, discount_rate, growth_rate), result_id FK nullable, created_at, updated_at, tags TEXT[]. - [ ] SQLAlchemy model + Pydantic schemas in src/climate_lama/models/. - [ ] Repository in src/climate_lama/db/repositories/scenario_repository.py with org-scoped access per Phase 2 RBAC pattern. - [ ] API endpoints in src/climate_lama/api/v1/scenarios.py: POST /v1/scenarios (create), GET /v1/scenarios (list, org-scoped), GET /v1/scenarios/{id} (read), PATCH /v1/scenarios/{id} (rename/retag), DELETE /v1/scenarios/{id} (soft delete). - [ ] Compute endpoint accepts a scenario reference — POST /v1/compute/impact gains optional scenario_id field; result is linked back via result_id. - [ ] UI (climate-lama-ui): "Save scenario" button on the result view (prompts for name + optional tags); "My scenarios" page lists saved scenarios and opens them back into the wizard with inputs pre-populated. No compare view yet (Phase 5+). - [ ] Unit tests: repository CRUD with org isolation; API tests for auth + RBAC; UI Vitest tests for save/load components. - [ ] scripts/demo.py updated to save a scenario end-to-end. - [ ] Postman collection updated.

Cross-stack: backbone (migration, model, repo, API, tests) + UI (save button, list page, router integration). Linked ADR: none required for data model; if API shape surfaces a generalizable pattern worth pinning, record it as a sub-ADR during /build. Suggested /analyze tier: 4 (Opus · high) — cross-module (schema + migration + API + UI), multiple repos, precondition for several later phases. Under-provisioning risks a flaky Scenario foundation that every subsequent phase pays for.

3.8 — Reproducibility metadata stamping

Context. Vision #C, Tier 1. Every saved scenario and every impact result must stamp engine_version, backbone_version, hazard_dataset_sha256, exposure_dataset_sha256, impact_function_id + version, random_seed. This is the single most load-bearing feature for the auditability claim that drives the whole platform positioning. The data model lands in this phase; Phase 4+ surfaces it in reports and compare views.

Acceptance criteria. - [ ] Alembic migration adds provenance JSONB column to impact_results (and scenarios if referencing post-compute results): {engine_version, backbone_version, hazard_dataset_sha256, exposure_dataset_sha256, impact_function_id, impact_function_version, random_seed, computed_at}. - [ ] impact_function model gains a version integer column; version bumps on any coefficient/curve change; old version rows stay immutable. - [ ] hazard_datasets and exposure_datasets tables carry a sha256 column populated at ingest (fall back to NULL for pre-existing rows; migration does not back-populate — that's a Phase 6 admin action). - [ ] Engine adapter (worker/models/engine_adapter.py) reads the engine package version via importlib.metadata.version("climate-lama-engine") and stamps it on every result. - [ ] GET /v1/results/{id} response includes the full provenance block. - [ ] Unit test: round-trip a compute job and assert every provenance field is populated and reproducible (same inputs → same SHAs). - [ ] UI: provenance not yet surfaced in the result panel (Phase 4 UI polish); endpoint correctness is what matters here.

Cross-stack: backbone (migration, model, adapter, API). UI unchanged. Linked ADR: none; implements a previously committed stance. Optionally add a short ADR "provenance stamping contract" if the /build agent finds non-obvious design choices worth recording. Suggested /analyze tier: 4 (Opus · high) — touches engine adapter (ADR-024 isolation boundary), cross-module (db + worker + api), and the reproducibility guarantee is security-adjacent.

3.9 — Monitoring instrumentation (Prometheus + Celery)

Context. Vision #18, Tier 1 instrumentation slice. Phase 2 shipped structlog; this work item adds the Prometheus metrics endpoint, Celery task instrumentation, and the bundled Grafana dashboards. Managed hosting (Tier 4) and self-hosters both benefit.

Acceptance criteria. - [ ] prometheus-fastapi-instrumentator wired into the FastAPI app; /metrics endpoint exposed (auth-gated or unauth, decide in ADR — likely unauth for self-hosters + bearer-token-required for managed). - [ ] Celery task instrumentation: per-task counter, duration histogram, failure counter. Labels include task_name, status. - [ ] OpenTelemetry tracing scaffold (opt-in via env var) — FastAPI → Celery span propagation. Skip if it slips; the Prometheus path is the must-have. - [ ] docker-compose.monitoring.yml profile adds Prometheus + Grafana + Loki containers, scraping the backbone. Opt-in: docker compose --profile monitoring up. - [ ] One Grafana dashboard committed (JSON in infra/monitoring/) covering: request rate + latency percentiles per endpoint, Celery queue depth + task durations, Postgres connection count. - [ ] Docs site page at architecture/observability.md documenting the stack and opt-in profile.

Cross-stack: backbone (instrumentation middleware, Celery hooks, compose profile, dashboards). No UI work. Linked ADR: consider a short ADR "observability stack choice" pinning Prometheus + Grafana + Loki vs alternatives — use judgment during /build. Suggested /analyze tier: 3 (Sonnet · high) — pattern is well-trodden, but multiple moving parts (instrumentation lib + Celery + compose profile + dashboard).

3.10 — Open-data attribution surface

Context. Vision #G, Tier 1. Every hazard, exposure, and impact-function dataset in the catalog needs source, license, citation, and source_url fields exposed in both the API and the UI. Trivial if the data model has the fields — likely partially there already from Phase 1/2; this item verifies + completes.

Acceptance criteria. - [ ] Audit existing hazard_datasets, exposure_datasets, and impact_functions tables. Any missing fields among {source, license, citation, source_url} → Alembic migration adds them (nullable for existing rows). - [ ] API response models include the attribution block on every relevant endpoint (GET /v1/hazards, GET /v1/exposures, GET /v1/impact-functions, and anywhere these are embedded). - [ ] Backfill seed data: existing JRC flood, LitPop, default impact functions all carry complete attribution. Seed script updates committed. - [ ] UI: attribution shown in every dataset selector (chip or small caption below the name) and in the scenario details panel. Accessible: the citation is a real link, not just text. - [ ] scripts/demo.py prints attribution alongside each loaded dataset.

Cross-stack: backbone (audit + migration + seeds + API) + UI (attribution chips). Linked ADR: none; implements Phase 3 baseline. Suggested /analyze tier: 3 (Sonnet · high) — touches three tables and two repos but the pattern is simple.

3.11 — Test data fixture generator

Context. Gap-2 (R3: "start with test data; seed with real-world data when accessible"). Every new hazard/exposure integration needs realistic fixtures; producing them ad-hoc is as expensive as the feature. A deterministic fixture generator scales.

Acceptance criteria. - [ ] tests/fixtures/generator.py provides make_hazard_fixture(hazard_type, n_events, bbox, seed), make_exposure_fixture(n_points, bbox, seed), make_impact_function_fixture(hazard_type, curve_shape, seed) returning deterministic numpy arrays + metadata. - [ ] Fixtures are reproducible (same seed → same bytes). SHA256 asserted in tests. - [ ] Existing tests/conftest.py fixtures migrated to use the generator where they duplicate logic. - [ ] Documented at docs/architecture/test-fixtures.md (site page) with a worked example per hazard type (RF, TC, WF, WS). - [ ] CI: generator module covered by its own unit tests; no large fixture files committed (everything generated in-memory).

Cross-stack: backbone tests + docs. No runtime code. Linked ADR: none. Suggested /analyze tier: 2 (Sonnet · medium) — bounded, test-only, pattern from existing conftest fixtures.

3.12 — Engine parity harness scaffold

Context. Gap-14 / A3: engine parity with CLIMADA is needed but "much later" per the user. This phase lays the scaffold only — a runnable harness that takes the same inputs through climate-lama-engine and a CLIMADA reference, compares outputs, and reports diffs. The harness must respect ADR-024 (backbone never imports CLIMADA); CLIMADA runs in a sidecar per the adapter-by-design ADR-027.

Acceptance criteria. - [ ] New repo-local directory tests/parity/ (not shipped with the backbone package) contains the harness. - [ ] tests/parity/harness.py orchestrates: load a fixture (from 3.11), run through the engine, run through a CLIMADA sidecar process (invoked via subprocess, not import), compare impact matrices, assert per-metric tolerance. - [ ] A minimal Dockerfile.climada-sidecar in tests/parity/ installs CLIMADA in a separate image. CI does not run this by default (gated by a dedicated workflow behind workflow_dispatch). - [ ] One working comparison test: river flood, small fixture, asserts EAD within 1%. This is the proof-of-concept; Tier 3 phases add coverage. - [ ] Docs site page architecture/engine-parity.md explains the harness and the ADR-024 boundary (why CLIMADA is subprocess-only).

Cross-stack: backbone tests + CI workflow. No runtime change. Linked ADR: references ADR-024 (isolation) and ADR-027 (adapter-by-design). Suggested /analyze tier: 4 (Opus · high) — CLIMADA-isolation-adjacent per the /analyze rubric ("Touches worker/models/engine_adapter.py or anything CLIMADA-isolation-adjacent → bump").

ADRs authored in this phase

ID Title Owner work item
ADR-025 MapLibre commitment + library whitelist 3.1
ADR-026 Dependency policy 3.2
ADR-027 Adapter-by-design architectural commitment 3.3
ADR-028 Error taxonomy + error envelope 3.4
ADR-029 DO Spaces bucket layout + dataset versioning 3.5

Each ADR carries a **Phase**: 3 header line so docs/CLIMATE_LAMA.md §11 can cross-reference by phase (see the plan.md structural section).

Exit criteria

  • All 12 work items closed (one issue per item; sub-issues allowed for 3.7 and 3.8 if they grow during implementation).
  • All 5 ADRs merged into docs/DECISIONS.md.
  • Docs site builds green in CI and publishes on tag.
  • scripts/demo.py exercises the Scenario save/load flow end-to-end.
  • Provenance fields populated on every impact_results row created by the demo.
  • No regressions in Phase 2 verification: auth, RBAC, multi-tenancy, structlog all still pass their existing tests.

Risks

  • 3.7 Scenario model is the largest item and has the widest downstream blast radius. If its data model drifts, Phases 4-7 pay for it. Treat the migration as irreversible once shipped; prefer a slightly over-general JSONB inputs column over a strict schema that needs a rewrite in Phase 5.
  • 3.8 provenance stamping touches engine_adapter.py — ADR-024 boundary. Any diff to that file needs human review during /build --review, not auto-merge.
  • 3.12 parity harness depends on a working CLIMADA install. If the sidecar Dockerfile turns out to be non-trivial, descope to "scaffold the harness, skip the working comparison test" and open a Phase 4 issue for the first real comparison.