Phase 9 — Ops execution plan: filling the Hetzner object store

Status: Authored 2026-08-07, merged (tracking issue #735 — CLOSED 2026-08-10 on OPS-5e's two-sided ≥6-haz-type pass). All waves OPS-0– OPS-5 have now run; the only unfinished piece is OPS-5a's review/approve leg, blocked by #828 (no platform admin exists in prod). Detail below. OPS-0, OPS-1 (1a-1d), OPS-2, OPS-3a and OPS-3c landed 2026-08-08 —

739/#740/#741 fixed in v0.9.1 (deployed to prod), plus #748 (api

container had no /data mount), #750 (sibling RLS-GUC gap in _wait_ingest_done), #757 (ESHM20 grid assembly required a complete rectangle; a real Greece query is ~23% missing cells, now filled as nan/nodata), #767 (the real Eurostat Census-GRID archive ships one combined multiband GeoTIFF, not a per-variable .tif — band now selected by description, verified against the archive's own read.me), and #774 (GHS-BUILT-S's citation, with the full Huizinga provenance note inlined, overflowed exposure_datasets.citation's VARCHAR(1024) — trimmed to point at properties.conversion_source instead) — all script-only fixes, re-delivered without a redeploy except where noted. OPS-3b (LitPop) LANDED 2026-08-08 via PR #781, Path B (--source climada-api, 150 arcsec / 2018, credential-free, checksum-verified per ADR-063): dataset_id=69e03607-8d99-4870-9560-6f998537f9d3, 7,896 points, verified in the prod DB. The owner-supplied Research Collection URL (30 arcsec / 2014, Path A) stays documented as the canonical-provenance alternative, unrun. OPS-4 exercised 2026-08-08 — fully blocked, nothing submitted to CDS/EWDS: 4a (wildfire FWI ×2) and 4c's C3S historical footprints both abort on the documented assert_netcdf_adapter_wired guard. Status resynced 2026-08-10 against trunk d4b2cc5: every CODE blocker the guard ever cited is now closed — #706 (PR #717), #761's two worker-side gaps (ADR-064), #782's aggregate-side event axis (PR #794), #791's dataset-row axis (PR #798, ADR-067), #793's single-slice event shape (PR #800, ADR-068) and #797's NaN-EAI compute gate (PR #804, ADR-069). SUPERSEDED 2026-08-10 (late): the evidence gate is RETIRED and #761 is CLOSED. The first end-to-end NetCDF ingest ran against prod that day (hazard_datasets c83be4f5-5bb1-4bef-82d8-a100061163cc, one year-labelled frequency-less hazard_events row), so layer 2 of assert_netcdf_adapter_wired and its --first-end-to-end-run escape were both deleted, and ADAPTER_GAP_ISSUE_URL went with them. The guard now performs the plan-time adapter-resolution check only. Everything below about the guard refusing every dispatch, about typing the escape, and about #761 having to stay open is historical — read scripts/packs/_wildfire_fwi_common.py on trunk rather than this doc. OPS-4 COMPLETE 2026-08-10 — this supersedes the "fully blocked, nothing submitted to CDS/EWDS" line above, which described 2026-08-08 and is no longer the state of anything. Real authenticated retrievals were submitted, delivered and ingested against prod. Evidence, by sub-wave: 4a (#614) — half done. EWDS historical INGESTED: wildfire-fwi-historical-greece-20230801, c83be4f5-5bb1-4bef-82d8-a100061163cc, haz_type=wildfire, active, COG present, 27,990 B raw (a plain .nc, not a ZIP), ~14 KB stored. This was the first end-to-end NetCDF ingest in the project's history and is what retired the evidence gate. C3S projections not ingested: the ZIP-per-experiment blocker is fixed (ADR-071, PR #822 / ffab274 — CDS archives are unpacked, one ingest per member), but the members are on a rotated-pole grid (grid_mapping_name=rotated_latitude_longitude, north pole 39.25 N / −162.0 E) that EPSG:4326-only NetCDF ingest cannot read, and PR

819 / 02f0ee3 added the guard that fails it closed. **Reprojection is the

single outstanding item for 4a. 4b (#616) — COMPLETE, and the "needs a human on each CDS dataset's live Download tab" blocker this banner used to carry is resolved: the _UNRESOLVED_CONFIRM_FROM_CDS_DOWNLOAD_FORM values were captured and both retrievals succeeded. Three datasets ingested, all active, all with COGs — heatwave-euro-cordex-europe-ssp5-8.5 (5857653d-4a63-40cc-95ce-3c2c19e54a01, 100 events 1986–2085), heatwave-euro-cordex-europe-ssp2-4.5 (591da928-cee9-4693-a39d-6a4afbcf0f9f, 100 events 1986–2085) and heatwave-ecde-europe (32fcd25e-59ec-4558-8b98-33365203cebd, 86 events 1940–2025). 238,249,300 B raw across two ZIPs, 203.33 MB stored. Caveat, not a blocker: #827 (open) — derive_dataset_axis drops the per-member scenario label at commit time, so both EURO-CORDEX rows carry the scenario in the name but scenario IS NULL in the column, and heatwave cannot yet answer a scenario-labelled matrix cell. 4c (#615) — COMPLETE for C3S; WISC dropped on evidence. C3S footprints INGESTED: windstorm-c3s-footprints-hist, 3f4f864d-01ab-485c-9f39-23c8856a0452, haz_type=storm_europe, active, COG present, one event (scenario NULL, year=2007, event_name=1463@2007, no frequency), grid 241×161 / 13,174 non-zero cells. The delivery was a 54,965 B ZIP holding one 54,713 B member — three orders of magnitude under this doc's ≤0.04 GB per-footprint planning bound, because the retrieval is scoped to a single storm rather than the whole domain. WISC synthetic was not retrieved, and the reason is now measured rather than suspected: the sis-european-wind-storm-synthetic-events process schema, read live from GET {cds_root}/retrieve/v1/processes/sis-european-wind-storm-synthetic-events, publishes only variable, version_id, year and month — no spatial or area input of any kind. Every scoping lever is temporal, so any WISC retrieval pulls whole-domain footprints and cannot be brought under this plan's no-multi-GB rule. Not pursued for Phase 9 on that basis. No ADR was written and none is implied — a later pass willing to fetch whole-domain footprints may revisit it. Mid-wave, a concurrent session's deploy-prod.yml dispatch recreated api/worker/beat to ship the already-tagged v0.10.0, wiping the docker-cp'd scripts — re-delivered via the host-side git clone recipe; see the OPS-4 comment on #735 for the full timeline. OPS-5 COMPLETE 2026-08-08 (5a excepted, still gated on 4b): 5c set the stale GHSL USD row's status to superseded (verified the lifecycle-gate mechanism in dataset_precedence.py actually applies to exposure_datasets before writing the UPDATE — first real use of that value on this table). 5b — staging/'s dir count (19) was far over this doc's baseline (3), but the byte total matched exactly; 16 of the extra dirs turned out to be harmless manifest.json leftovers from succeeded ingest_jobs rows (code gap filed as #784, fixed by ADR-066 — those 16 files are the only source provenance those datasets have, so leave them: post-ADR-066 ingests write it to hazards/{dataset_id}/source_manifest.json and purge staging/ wholesale), and the 3 real orphans plus 1 orphan hazards/ NPZ dir (matching baseline) were owner-authorized and deleted — re-verified after: 16 staging dirs / 13 KB remain (all legitimate), orphan NPZ gone, 24 hazards/ prefixes against 25 hazard_datasets rows (the 1 gap is a failed job, not an orphan). 5d: live bucket was 0.32 GB vs the ledger's ~24.6 GB raw-zone estimate — expected, since raw is clip-then-delete, not final storage. Re-measured 2026-08-10, after OPS-4 and after trash/ was purged: 158 objects / 359.87 MB against a ledger raw sum of ≈24.87 GB — same gap, same reason. trash/ is now 0 (28 objects / 237.40 MB deleted), and processed/ grew 89.77 → 158.75 MB when #611's national baseline finally got its COG, so the bucket is now almost entirely live data (hazards/ 200.72 + processed/ 158.75 = 359.47 of 359.87 MB). 5e: both sides agree exactly — coastal_flood 7, earthquake 2, river_flood 16 (3 distinct, 25 total) from both the DB query and GET /v1/hazards?limit=200 (the doc's own login recipe needed a fix: the auth cookies come back Secure-flagged, so a plain CookieJar silently drops them on the internal http:// follow-up request — extracting the raw Set-Cookie header and replaying it directly works). Still correctly short of the ≥6-type closing condition — the 3 missing types are exactly OPS-4's three. See the OPS-5 comments on #735 for full detail and exact commands. OPS-3b landed (see above). Status resynced 2026-08-10 — the "Next unblocked work: none" paragraph this banner used to carry is retired, and so is its premise. It read: "the only remaining gaps (OPS-4's three sub-waves) still need external input: owner CDS-form access for 4b, and for 4a/4c a decision to clear the assert_netcdf_adapter_wired evidence gate ... a deliberate refusal to submit a real authenticated retrieval until one has succeeded end to end." All three sub-waves have now run: the evidence gate was satisfied and deleted (PR #825 / 42f2a35), the 4b CDS-form values were resolved, and five new hazard datasets are live in prod — see the OPS-4 block above. OPS-5e was then RE-EXECUTED and PASSED, and #735 is CLOSED on that evidence — this supersedes both the "still correctly short of the ≥6-type closing condition" line above and an interim note here that called the check formally unverified. Measured 2026-08-10: DB gives coastal_flood 7 | earthquake 1 | heatwave 3 | river_flood 8 | storm_europe 1 | wildfire 1 = 6 distinct; the API (GET /v1/hazards, cookie session) agrees exactly — status=active returns 21 rows / 6 distinct with an identical breakdown, unfiltered returns 31 rows / 6 distinct, and a per-type ?haz_type= round-trip returns 200 with matching counts for all six. Trap worth carrying forward: the DB side must be read on a psql SUPERUSER connection. The app connection returns zero rows for that query under RLS, which reads as an empty catalog when the catalog is fine. Do not diff the 2026-08-10 counts against the 2026-08-08 ones above without noticing they are on different filters: 2026-08-08 counted rows of every status (25 total), 2026-08-10's headline is active-only (21), with 31 rows unfiltered. The gap between 21 and 31 is non-active rows; this pass did not enumerate their statuses, so do not read a specific composition into it. Remaining known gaps: rotated-pole reprojection for #614's C3S half, #827's dropped scenario label on #616's EURO-CORDEX rows, OPS-5a's review/approve leg (blocked by #828, see that wave), and the owner-run Full-Catalog walkthrough. Supervision contract loosened 2026-08-07: agents now run every wave autonomously, including under --ghost, with no owner-launch requirement and no [OWNER CONFIRM] gate on destructive steps — see below. Licence acceptances for every CDS/EWDS dataset here were completed by the owner on 2026-08-07** — this supersedes the "still need their licence acceptances" line in phase-9-full-catalog-walkthrough.md and the unticked acceptances checkbox in phase-9-build-plan.md; both lines are left as-is (see this doc's cross-link edits) rather than rewritten.

How to use this doc

What this is. Phase 9 shipped the code half of ten data packs (#611–#620, CLOSED) but never ran the ops half: download → Hetzner raw zone → prod ingest. When this doc was authored, every row in docs/data/volume-budget.md read Status: planned. That is no longer true and has not been since 2026-08-08: as of 2026-08-10 all ten rows are ingested or partially so — read the ledger's Status column, never this sentence. This is the runbook for that ops half — waves OPS-0 through OPS-5, each independently launchable and resumable.

Terminology. Every command below is written for a runtime agent — an interactive Claude Code session the owner launches to execute one wave (e.g. /forge "run OPS-2 of docs/plan/phase-9-ops-plan.md"). "Runtime agent" is used consistently instead of "you"/"the operator" so a command copied out of context still reads correctly.

Disambiguation. Waves here are always OPS-0…OPS-5, never bare "wave N" — that already means the Showcase walkthrough (phase-9-wave-0-walkthrough.md) and the build waves that landed the pack code (phase-9-build-plan.md). An OPS-N is never either of those, and this plan does not touch that walkthrough file.

Connection recipe. SSH only works through WSL (Windows keys are rejected). - One-shot read-only check: wsl -d Ubuntu -- ssh root@159.69.211.124 '<cmd>'. - Wave work (anything long, or that a dropped connection must not kill): attach once — wsl -d Ubuntu -- ssh -t root@159.69.211.124 'tmux new -A -s ops-<N>'. Every command in a wave's Commands block below is written bare, exactly as typed inside that tmux session — never re-wrapped in the ssh '<cmd>' form, which would defeat the outage protection. cwd: /opt/climate-lama unless stated otherwise.

Script delivery. docker/core.Dockerfile copies only scripts/seed_demo.py into the images (confirmed: docker exec climate-lama-api-1 ls scripts/ → seed_demo.py only), and there is no repo checkout on the host. Copying scripts from the workstation to run in-container is data-ops, not an infra change — done once, in OPS-0:

# from the workstation:
wsl -d Ubuntu -- scp -r scripts root@159.69.211.124:/opt/climate-lama/repo-scripts/
# inside the tmux session:
docker cp /opt/climate-lama/repo-scripts/. climate-lama-api-1:/app/scripts/
./dc exec -T api python scripts/packs/<pack>.py --help   # import smoke check

./dc exec does not propagate the container's exit code (#739, verified live on the prod host: ./dc exec -T api python -c 'import sys; sys.exit(1)' leaves $? at 0). Judge every pack script run through ./dc exec by its output, never by its exit status — including the --help smoke check above and every wave's Commands block below.

Copy the whole scripts/ tree, never single files — the wildfire packs import scripts/packs/_wildfire_fwi_common.py, and the derive_* scripts read sibling JSON artifacts (huizinga_2017_europe_damage_functions.json, esrm20_2021_pga_vulnerability.json) that must land beside them.

Object storage. Pack scripts write through the app's own storage layer using the MINIO_* env already in the containers (bucket climate-lama-storage, endpoint nbg1.your-objectstorage.com) — no host-side mc/aws CLI. For a bucket listing:

./dc exec -T api python -c "
from climate_lama.storage.client import get_minio_client
c = get_minio_client()
for o in c.list_objects('climate-lama-storage', prefix='raw/', recursive=True):
    print(o.object_name, o.size)
"

(Confirm the import path against src/climate_lama/storage/ before first use — not re-verified live here, per this plan's own out-of-scope fence.)

Owner-input table.

Placeholder	What it is	Resolved in
<admin-session>	Session with ≥ANALYST role — gates hazard ingest & curve seed (require_role, src/climate_lama/api/v1/hazards.py, .../impact_functions.py)	OPS-0
CDS/EWDS API credentials	Whether CDSAPI_KEY/EWDSAPI_KEY exist in the worker container — licence acceptance ≠ credentials	OPS-0 (check); absent ⇒ blocks OPS-4
#616 CDS request values	~~_UNRESOLVED_CONFIRM_FROM_CDS_DOWNLOAD_FORM sentinels in scripts/packs/heatwave_packs.py, read off each dataset's live "Download" tab~~ — RESOLVED 2026-08-10; both retrievals were submitted, delivered and ingested (see OPS-4b). No longer an owner input	OPS-4 (done)
LitPop --download-url	Now OPTIONAL (ADR-063). Only Path A needs it: the exact per-country CSV URL for Greece, from the ETH Research Collection handle page (doi:10.3929/ethz-b-000331316), which returns HTTP 403 to automated clients (anti-scraping, not JS-rendering) and must be opened in a real browser. Path B (--source climada-api) needs no owner input at all	OPS-3b

<admin-session> resolver. Pack scripts themselves don't need this — _demo_org_and_user resolves the demo org/user directly against the DB in-container. The cookie is needed only for this plan's own verification curls and manual API triggers. Check first:

./dc exec -T api env | grep -i CL_DEMO_USER_PASSWORD

If set, the account is demo@example.com at role ANALYST (scripts/seed_demo.py::_ensure_user — a set password grants ANALYST, an unset one leaves an unguessable-password VIEWER). Then:

docker exec climate-lama-api-1 curl -s -c /tmp/ops-session.txt -X POST \
  http://localhost:8000/v1/auth/login -H "Content-Type: application/json" \
  -d '{"email":"demo@example.com","password":"<the CL_DEMO_USER_PASSWORD value>"}'

If unset, that's a named owner input blocking API-side verification only (DB-side verification, this plan's source of truth, still works) — record and continue; don't set the password yourself.

Supervision contract (loosened 2026-08-07). Ops waves are agent-runnable autonomously — --ghost and fully unattended runs are allowed, no owner launch required, and no [OWNER CONFIRM] gate on destructive steps (prod-row delete/update, object delete) — a runtime agent runs a wave end-to-end on its own judgment. Two things still hold, because they're engineering discipline, not human-approval gates: (1) A failed precondition mid-wave = STOP, record state as a comment on #735, never improvise an infra change or invent a value this doc doesn't source (ADR-059 still applies — a guessed CDS request body or download URL is not an acceptable substitute for a resolved one). (2) Every prod mutation gets logged with its exact rollback command (the ghost protocol's standing mutation-ledger rule) — autonomy is not exemption from that record. This amends phase-9-build-plan.md ground rule 8 for the pack-ops waves only (rule 8 assumed ad hoc ops; this makes it a runnable, autonomous plan) — it does not extend to phase-9-wave-0-walkthrough.md, which stays owner-run and untouched.

Disk posture. Everything ends in object storage — nothing persists on server disk. One artifact at a time: download → upload/ingest → verify → delete; never two multi-GB artifacts co-resident. Every Disk check below is a formula against a live df -h / reading taken at run time — never the 2026-08-07 snapshot in this doc, which will be stale: - Zipped rasters (zip + extracted + processed co-resident): required_free = 2.5 × artifact size. - Single-file downloads (CSV, one NetCDF slice, one windowed clip): required_free = 1.5 × artifact size. If it doesn't fit: don't start. Comment the df -h / output on #735 and stop that wave — this is the standing abort action, not repeated per wave.

Resumability. Each wave is re-enterable by a fresh session from this doc + the ledger + live state alone. Check, in order: (1) the raw zone — a completed upload is never re-downloaded; (2) docs/data/volume-budget.md's Status column; (3) comments on #735 (OPS-4's submitted request IDs live there, separate from the ledger); (4) the wave's own Resume after interruption check. Pack-script idempotency is not established anywhere in this codebase — no wave claims "safe to blindly re-run" without a stated check first.

Status tracking. volume-budget.md stays the per-pack tracker (flips at execution time, not in this PR); #735 mirrors at wave granularity. Neither this doc nor its PR edits volume-budget.md.

Verification trap (#729, closed but never root-caused, intermittent). GET /v1/impact-functions?haz_type=river_flood has returned empty immediately after a logged-successful seed. Every check below treats the DB query as source of truth, never the API read alone. On disagreement, re-read after 60s before concluding anything — never re-ingest on one disagreeing read.

---

OPS-0 — Execution environment & credentials

Preconditions — wsl -d Ubuntu -- ssh -T root@159.69.211.124 succeeds.

Commands

wsl -d Ubuntu -- scp -r scripts root@159.69.211.124:/opt/climate-lama/repo-scripts/
wsl -d Ubuntu -- ssh -t root@159.69.211.124 'tmux new -A -s ops-0'
# inside tmux, cwd /opt/climate-lama:
docker cp /opt/climate-lama/repo-scripts/. climate-lama-api-1:/app/scripts/
./dc exec -T api python -c "import scripts.packs.river_flood_jrc"
./dc exec -T api python scripts/packs/earthquake_eshm20.py --help
./dc exec -T api python scripts/packs/ghsl_exposure.py --help
tmux -V
df -h /
./dc exec -T api env | grep -i CL_DEMO_USER_PASSWORD
./dc exec -T worker sh -c 'env | grep -iE "CDSAPI_|EWDSAPI_"'

The two --help calls plus the river_flood_jrc import are the import smoke check — success proves the sibling JSON/module imports resolved after docker cp. river_flood_jrc.py has no argparse/__main__ (it's a library module called from scripts/ingest_historical_catalog.py, see OPS-1b), so --help against it silently exits 0 with no output rather than failing — a plain import is the correct check there instead. A ModuleNotFoundError/FileNotFoundError on any of these means the copy is incomplete — re-check the whole tree, not just the failing file.

Disk check — n/a; the scripts/ tree is small.

Verification — all three --help calls exit 0; CDSAPI_KEY/ EWDSAPI_KEY/EWDSAPI_URL/CDSAPI_URL present in the worker env (.env.example names). Missing ⇒ record as an owner input blocking OPS-4 on #735 and continue with other waves.

Ledger rows flipped: none.

Resume after interruption — re-run the --help checks; if they pass, done. scp/docker cp are always safe to re-run (overwrite, no state).

Escalation — if wsl -d Ubuntu -- ssh itself fails, STOP — an access problem outside this plan's scope (see docs/RUNBOOK.md), not a key/host to swap.

---

OPS-1 — Curves + no-auth floods

1a — Curve seeding (Huizinga + ESRM20)

Preconditions — OPS-0 done.

Commands

./dc exec -T api python scripts/packs/derive_huizinga_flood_curves.py --check
./dc exec -T api python scripts/packs/derive_esrm20_earthquake_curves.py --check

Both are offline provenance checks against core/impact_function_seeder.py. If either fails, STOP — the seeder has drifted from its cited source (the exact defect #621/ADR-059 exists to prevent); this is a code question, not an ops one. If both pass:

docker exec climate-lama-api-1 curl -s -b /tmp/ops-session.txt -X POST \
  http://localhost:8000/v1/impact-functions/seed

(documented idempotent — "Safe to call repeatedly", src/climate_lama/api/v1/impact_functions.py).

Disk check — n/a (no download).

Verification (DB is source of truth, #729):

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT haz_type, count(*) FROM impact_functions GROUP BY haz_type;"

Expect an earthquake row alongside river_flood (today: river_flood only, 1 curve).

Ledger rows flipped: none — curves aren't a volume-budget row.

Resume after interruption — re-run the SELECT; a populated earthquake row means this already ran (re-POSTing is also safe here, unlike most steps below).

Escalation — --check failure ⇒ STOP, escalate as a code issue; never seed values known to be drifted from source.

1b — JRC river flood (E1)

Preconditions — 1a done.

Commands

./dc exec -T api python scripts/ingest_historical_catalog.py ingest-hazards --data-dir /data/historical_catalog

--data-dir must point under /data — the api container's only mount shared with the worker (#748). The default (_PROJECT_ROOT-relative, /app/data/...) writes to the container's own ephemeral layer, which the worker can never see.

Disk check — 9 windowed clips, ≤0.41 GB each (ledger upper bound ≤3.7 GB total): 1.5 × 0.41 ≈ 0.62 GB free per in-flight clip.

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, provider, status FROM hazard_datasets WHERE haz_type='river_flood' ORDER BY created_at DESC LIMIT 5;"

Expect a new row with provider populated (not blank).

Ledger rows flipped: #611 → downloaded → ingested.

Resume after interruption — re-run the SELECT first; a recent non-blank-provider JRC row means don't re-run (no idempotency established).

Escalation — chord failure ⇒ ./dc logs --tail=200 worker, capture request_id, record on #735, STOP.

1c — Aqueduct riverine re-ingest (E2, #612's ops half)

The 7 existing Aqueduct riverine rows predate the identity migration (blank provider/status). ADR-048's status column supports active/ superseded at the app level (no CHECK/enum), and ADR-056 ranks status == "superseded" last without dropping it. Mechanism: re-ingest fresh rows, then mark the old 7 superseded — never DELETE. Located directly in docs/DECISIONS.md; not a STOP-AND-ASK case.

What #740 changed (read before running this wave). The first live attempt at this wave was a total no-op: catalog identity matched purely on (org_id, name), and the 7 legacy rows already carry the exact names this manifest generates, so ingest logged already in catalog, skipping for every dataset and created nothing for the UPDATE below to supersede to. Identity is now scoped by provider as well, and the ingest payload stamps provider + status='active' on the rows it writes. Two consequences for this wave:

• The org transiently holds two rows per dataset name — the blank provider legacy one and the fresh one — between the ingest and the UPDATE. That is expected and inserts cleanly (hazard_datasets has no unique constraint on name), and ADR-056 already ranks the attributed row ahead of the blank one, so resolution is correct even in that window.
• Re-running ingest after a successful pass is a no-op again, because the fresh rows now carry the provider the gate matches on. Idempotency is preserved; it is the legacy rows that no longer suppress the re-ingest.

Preconditions — none beyond OPS-0 (independent of 1b).

Commands

./dc exec -T api python scripts/ingest_scenario_hazards.py verify
./dc exec -T api python scripts/ingest_scenario_hazards.py ingest --data-dir /data/scenario_hazards

--data-dir must point under /data (see the OPS-1b note above, #748). Preview before superseding:

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, source, created_at FROM hazard_datasets
     WHERE haz_type='river_flood' AND provider IS NULL
       AND source <> 'Climate-Lama reference fixture';"

If the row set matches this doc's expectations (7 rows — see the Escalation note below if it doesn't):

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "UPDATE hazard_datasets SET status='superseded'
     WHERE haz_type='river_flood' AND provider IS NULL
       AND source <> 'Climate-Lama reference fixture';"

Both statements exclude the seeded demo fixture on purpose — do not drop that clause. Verified live on 2026-08-07: prod holds 8 rows matching provider IS NULL, not 7 — the 7 Aqueduct rows plus JRC River Flood Greece (demo), whose source is Climate-Lama reference fixture. A bare provider IS NULL preview therefore returns 8 and trips the ≠ 7 ⇒ STOP rule below on a perfectly healthy system, and the matching bare UPDATE would supersede the demo fixture along with the legacy rows. The fixture is not part of #612's identity migration and is already demoted by eligible_candidates' is_seeded_fixture check (#631) once real catalog data exists, so it must be left alone here.

Disk check — 21 clips, ledger upper bound ≤0.09 GB total: 1.5 × 0.09 ≈ 0.14 GB.

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT provider, status, count(*) FROM hazard_datasets WHERE haz_type='river_flood' GROUP BY provider, status;"

Expect the 7 old rows status='superseded'; new rows with provider populated.

Ledger rows flipped: #612 → downloaded → ingested.

Resume after interruption — re-run the verification query; if the 7 already show superseded, don't re-run the UPDATE — re-check the row set matches this doc's expectations each resume, don't assume a prior partial run left it in the right state.

Escalation — preview count ≠ 7 ⇒ STOP, record on #735 — prod drifted from this doc's stated ground truth; don't guess a fix, record the drift for a later pass. Count the preview with the fixture clause above: 7 is the expected Aqueduct-only count, and a bare provider IS NULL query returns 8 on a healthy prod (the extra row is the demo fixture — see the note beside the preview). If you get 8 from the scoped query, that is real drift and the STOP applies.

1d — Aqueduct coastal (E3's ingestable half, ADR-062)

No supersede step — no pre-existing coastal rows. LISCOAST stays permanently out of hazard-ingest scope (ADR-062); do not touch its ledger note.

Commands

./dc exec -T api python scripts/packs/coastal_flood.py verify
./dc exec -T api python scripts/packs/coastal_flood.py ingest --data-dir /data/coastal_flood

--data-dir must point under /data (see the OPS-1b note above, #748).

Disk check — 21 clips, ≤0.09 GB: 1.5 × 0.09 ≈ 0.14 GB.

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT haz_type, count(*) FROM hazard_datasets WHERE haz_type='coastal_flood';"

Expect ≥1 row (today: 0).

Ledger rows flipped: #613 → downloaded → ingested (Aqueduct half only).

Resume after interruption — non-zero count ⇒ already ran.

Escalation — same as 1b.

---

OPS-2 — Earthquake (E9, #617)

A timing, not volume, wave: one measured point request took ~7.8s; the full Greece bbox (5 return periods × 7,296 points) is estimated at 30–40 minutes cumulative (scripts/packs/earthquake_eshm20.py docstring, note b).

Preconditions — OPS-0 done; 1a's ESRM20 curves seeded.

Commands

./dc exec -T api python scripts/packs/earthquake_eshm20.py --help  # confirm exact subcommand/flags first
./dc exec -T api python scripts/packs/earthquake_eshm20.py ingest --data-dir /data/earthquake_eshm20

--data-dir must point under /data (see the OPS-1b note above, #748).

Disk check — ledger upper bound ≤0.002 GB raw; negligible.

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT haz_type, count(*) FROM hazard_datasets WHERE haz_type='earthquake';"

Expect ≥1 row. This trips the cross-repo UI gate — climate-lama-ui#109 (renders eai_grade/eai_grade_note) is CLOSED and deployed in ui v0.5.0 (verified 2026-08-07), so ingesting here is safe. If a future resume finds the deployed UI tag older than v0.5.0, STOP — don't ingest against a UI that can't label screening-grade EAI.

Ledger rows flipped: #617 → downloaded → ingested.

Resume after interruption — non-zero count ⇒ already ran. If interrupted mid-sequence, check which return periods already produced rows before re-requesting the rest.

Escalation — repeated 5xx/timeouts from EFEHR = upstream availability, not a code defect; stop-loss at 3 attempts, then escalate rather than loop.

---

OPS-3 — Exposure

3a — Eurostat Census 2021 grid (E12, #620)

Commands

./dc exec -T api python scripts/packs/eurostat_census_grid.py ingest --data-dir /data/eurostat_census_grid

--data-dir must point under /data (see the OPS-1b note above, #748).

Disk check — measured zip 0.566 GB (real HTTP HEAD): zipped-raster bucket, 2.5 × 0.566 ≈ 1.42 GB (≈1.32 GiB).

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, value_unit FROM exposure_datasets WHERE name ILIKE '%census%' OR name ILIKE '%eurostat%' ORDER BY created_at DESC LIMIT 5;"

Expect a new row, value_unit='persons'.

Ledger rows flipped: #620 → downloaded → ingested.

Resume after interruption — matching recent row ⇒ already ran; confirm the local zip was deleted after ingest.

Escalation — reprojection or zip-member-selection failure (own module raises rather than guesses) ⇒ record exact error, STOP; never pass a guessed --tif-member.

3b — LitPop (E10, #618)

LANDED 2026-08-08 via PR #781, Path B below: dataset_id=69e03607-8d99-4870-9560-6f998537f9d3, 7,896 points, verified in the prod DB. The commands below stay documented for a re-run or for a fresh operator who wants Path A's canonical provenance instead.

Two sources — pick one (ADR-063). They are different products and are catalogued under different dataset names, so running both is legal but produces two rows:

Path	Source	Grid	Value epoch	Owner input
A (default, canonical)	ETH Research Collection, doi:10.3929/ethz-b-000331316	30 arcsec (~1 km)	2014	Yes — a human resolves the CSV URL in a browser
B (no owner input)	CLIMADA Data API, climada.ethz.ch/data-api/v1/dataset/	150 arcsec (~4.6 km, screening-grade)	2018	No — resolved live, no credentials

Path B is not a CLIMADA dependency: it is a plain HTTPS file server, nothing imports or installs climada, and ADR-024 fences code and packaging rather than data. See ADR-063 for the full decision.

---

Path A — ETH Research Collection (canonical provenance, needs the owner)

Preconditions — --download-url resolved: a human visits the ETH Research Collection handle page (doi:10.3929/ethz-b-000331316 → research-collection.ethz.ch), locates the real Greece CSV, supplies its URL. Never guess this URL (ADR-059). The site returns HTTP 403 "Access Restricted — due to a high volume of automated traffic (scraping)" to automated clients. That is an IP-level anti-scraping control, not a JS-rendering problem (this doc previously said otherwise), and we do not evade it — a real browser session is the supported path, or use Path B.

Commands

./dc exec -T api python scripts/packs/litpop_exposure.py ingest --help
./dc exec -T api python scripts/packs/litpop_exposure.py ingest \
  --download-url '<resolved URL>' --data-dir /data/litpop

--data-dir must point under /data (see the OPS-1b note above, #748).

Disk check — calculated upper bound ≤0.02 GB (Greece land area at ~1 km² cells): 1.5 × 0.02 = 0.03 GB. Treat this as unverified: DataCite reports the whole release as 16.63 GB across text/csv + application/x-tar, and nothing establishes Greece's share of it. Re-check against the file's actual Content-Length before downloading.

---

Path B — CLIMADA Data API (no owner input; runs unattended)

The API release ships as HDF5, and the api image installs no extras, so it has no h5py. The conversion therefore runs off-box, on any machine with h5py (pip install 'climate-lama[packs]'), and the CSV is handed in with docker cp:

# 1. on a workstation (NOT in the api container)
python scripts/packs/litpop_exposure.py convert \
  --source climada-api --out LitPop_150arcsec_GRC.csv
# resolves the dataset live, verifies the declared md5 + byte size, converts

# 2. hand the CSV to the host
docker cp LitPop_150arcsec_GRC.csv climate-lama-api-1:/data/litpop/

# 3. ingest in-container — no h5py needed on this path
./dc exec -T api python scripts/packs/litpop_exposure.py ingest \
  --source climada-api --source-file /data/litpop/LitPop_150arcsec_GRC.csv \
  --data-dir /data/litpop

If the api image is ever rebuilt with the packs extra, steps 1–3 collapse to a single in-container ingest --source climada-api --data-dir /data/litpop (it downloads, verifies and converts itself). Do not add the extra to the image just for this — the off-box path costs nothing and needs no redeploy.

Disk check — measured, not an estimate: the HDF5 is 1,678,480 bytes (0.0017 GB) and its CSV is of the same order. Negligible against any bucket.

Resolution — the pack defaults --resolution-arcsec to the selected source's own spacing (30 for eth-rc, 150 for climada-api). Do not override it; forcing the 150-arcsec source onto a 30-arcsec grid scatters ~7,900 points across a ~1M-cell mostly-empty raster.

---

Verification (either path)

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, value_unit, upstream_version FROM exposure_datasets WHERE name ILIKE '%litpop%' ORDER BY created_at DESC LIMIT 5;"

Path A lands LitPop -- Greece default asset-value exposure (upstream_version='2014'); Path B lands LitPop -- Greece default asset-value exposure (150 arcsec, 2018) (upstream_version='2018'). The names differ on purpose — the name is the idempotency key, so the two releases never overwrite each other.

Ledger rows flipped: #618 → downloaded → ingested (done — Path B, 2026-08-08, see above).

Resume after interruption — re-run the verification query and check which name came back. On Path A, comment the resolved URL on #735 the first time so a fresh session doesn't re-derive it.

Escalation — Path A: the 403 persists ⇒ a human resolves the URL via a real browser session; never push through by retrying headlessly or by working around the block. Path B: resolve_climada_api_dataset raising on zero/multiple matches means the API's catalog shifted — record the listed candidates and STOP; never hand-pick one. A checksum mismatch means the downloaded artefact is not what the API declared — STOP, do not ingest.

3c — GHSL (E11, #619): BUILT-S → BUILT-V → GHS-POP

One product at a time — land, ingest, verify, delete, then the next. Never hold two GHSL zips on disk together.

Preconditions — OPS-0 done; a fresh live df -h / taken before touching BUILT-S.

Commands

./dc exec -T api python scripts/packs/ghsl_exposure.py ingest built-s --data-dir /data/ghsl_exposure
# verify, confirm cleanup, then:
./dc exec -T api python scripts/packs/ghsl_exposure.py ingest built-v --data-dir /data/ghsl_exposure
# verify, confirm cleanup, then (gated by the disk check below):
./dc exec -T api python scripts/packs/ghsl_exposure.py ingest pop --data-dir /data/ghsl_exposure

--data-dir must point under /data (see the OPS-1b note above, #748).

Disk check — per product, against a fresh live reading each time (measured HTTP HEAD sizes, zipped-raster bucket, 2.5×):

Product	Zip size	required_free = 2.5×
BUILT-S	3.21 GB	≈8.02 GB ≈ 7.47 GiB
BUILT-V	4.36 GB	≈10.91 GB ≈ 10.16 GiB
GHS-POP	12.55 GB ≈ 11.7 GiB	≈31.4 GB ≈ 29.2 GiB

The 2026-08-07 snapshot (~12 GiB free, 68% used) is stale by the time this runs — take a new reading. BUILT-S/BUILT-V don't share fate with GHS-POP: ingest them regardless of what GHS-POP's later check shows.

GHS-POP remediation branch — if the live check can't fit ≈29.2 GiB: don't start the GHS-POP download. BUILT-S/BUILT-V still count as complete. (1) Comment the live df -h / output + rationale on #735; (2) leave its sub-checkbox unchecked; (3) do not propose growing the server or attaching a volume — an owner infra decision out of this plan's scope.

Verification (after each product):

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, value_unit FROM exposure_datasets WHERE name ILIKE '%ghs%' ORDER BY created_at DESC LIMIT 10;"

Expect built-s at value_unit='EUR' (Huizinga €/m² conversion), built-v at value_unit='m3' (deliberately unconverted — see OPS-5c), pop (if run) at value_unit='persons'.

Ledger rows flipped: #619 → downloaded → ingested per landed product; leave GHS-POP at planned if deferred.

Resume after interruption — check which products already have a row before re-running any ingest. A zip on disk with no row is a partial state — re-run only the ingest half if --help shows a local-file flag, else re-download.

Escalation — _require_supported_crs failure ⇒ wrong variant fetched; STOP, don't force a reprojection (ADR-051's fail-fast stance is deliberate).

---

OPS-4 — CDS/EWDS retrievals

Submit everything first — queue latency, not bytes, is the long pole. Record each submitted request as a comment on #735 at submission time (the in-flight resume record, deliberately separate from the ledger's per-pack ingest status); check once that the request-ID format embeds no account/billing data before posting, redact if so.

Preconditions — OPS-0 found CDSAPI_KEY/EWDSAPI_KEY present. If either is absent, STOP this whole wave, record the gap on #735.

4a — Wildfire FWI ×2 (E4/E5, #614)

./dc exec -T api python scripts/packs/wildfire_fwi_ewds.py ingest --year 2023 --month 8 --day 1 --data-dir /data/wildfire_fwi_ewds
./dc exec -T api python scripts/packs/wildfire_fwi_c3s.py ingest --data-dir /data/wildfire_fwi_c3s

--data-dir must point under /data (see the OPS-1b note above, #748).

SUPERSEDED 2026-08-10 (late) — the shared evidence gate is retired and #761 is CLOSED. The first end-to-end NetCDF ingest ran against prod (hazard_datasets c83be4f5-5bb1-4bef-82d8-a100061163cc; one year-labelled hazard_events row, scenario NULL, no frequency columns), which is exactly the evidence the gate demanded. Layer 2 of assert_netcdf_adapter_wired, the --first-end-to-end-run escape and the ADAPTER_GAP_ISSUE_URL constant were all deleted; the guard is now the plan-time adapter-resolution check only, and there is no flag to type. The C3S refusal below is unaffected and still stands — it is that product's own UNINGESTABLE_REASON, was never part of this gate, and has no escape. Everything from here to the end of OPS-4a about the guard refusing every dispatch, about #761 staying open and about the one-shot escape is historical record, kept because it explains why the gate was justified while it stood.

RESULT 2026-08-10 — the EWDS line RAN AND LANDED; the C3S line still refuses. This supersedes the sentence below, which is kept only as the record of what was true until that run: "Both packs' ingest still aborts on the assert_netcdf_adapter_wired guard, by design — expect it, and do NOT route around it."

EWDS — DONE. wildfire-fwi-historical-greece-20230801, id c83be4f5-5bb1-4bef-82d8-a100061163cc, haz_type=wildfire, status=active, COG present. Raw download 27,990 bytes, delivered as a plain NetCDF at data/wildfire_fwi_ewds/wildfire-fwi-historical-greece-20230801.nc — not a ZIP, unlike the C3S deliveries. Stored footprint ≈14,125 bytes (hazards/ 0.01 MB + processed/ 0.00 MB). Note this is a different retrieval from the 2026-08-09 #802 probe that measured 28,402 bytes: different submission, different day, both real — neither corrects the other.

Update 2026-08-10 (#614) — the C3S line above still aborts, but only one of its two reasons survives. Its request vocabulary is resolved and proven (job 520e1990-…, 201 → successful, 29.4 MB). Opening the delivered file found two blockers; the first is now FIXED: the asset is a ZIP of one NetCDF per experiment rather than the single .nc the dispatch path expected, and PR #822 (ffab274, ADR-071) makes the source layer unpack whatever archive a CDS retrieval delivers and dispatch one ingest per member. The second still stands and is the only thing left: the members are on a rotated-pole grid (grid_mapping_name=rotated_latitude_longitude, north pole 39.25 N / −162.0 E) and NetCDF ingest is EPSG:4326-only with no reprojecting read — PR #819 (02f0ee3) added the explicit CF grid_mapping_name guard that fails it closed instead of silently ingesting the mid-Atlantic misplacement. run_ingest_hazards refuses with UNINGESTABLE_REASON before any DB or store call. Reprojection is the single outstanding item for this sub-wave — do not go looking for a flag, an escape or a further code blocker; there is none. The C3S --dimension KEY=VALUE flag is gone; use --horizon YEAR if you are driving projections_request for a download.

Read the guard on trunk, not this paragraph. The authoritative statement of what the guard is waiting for lives in scripts/packs/_wildfire_fwi_common.py — the NETCDF_INGEST_BLOCKERS tuple, the ADAPTER_GAP_ISSUE_URL constant, and the module/assert_netcdf_adapter_wired docstrings. That pointer has moved five times (#706 → #761 → #782 → #791 → #793), so any issue number written into this doc is stale by default. git show origin/main:scripts/packs/_wildfire_fwi_common.py before acting on the summary below.

Summary as of trunk d4b2cc5 (2026-08-10): no known CODE blocker remains. All six are closed:

Was	Fix	Verified
#706 — NetCDF ingest unreachable from the dispatch path	PR #717 (per-job source_format override)	closed
#761 — non-adapter-aware validate_source + empty worker-side layout registry	ADR-064 (layouts declared in core/ingest/netcdf_layouts.py, registered at import)	code half fixed; the "issue deliberately still OPEN" note that stood here is superseded — #761 is CLOSED as of 2026-08-10, see the RESULT block at the top of OPS-4a
#782 — aggregation on a return-period axis it does not have	PR #794 / afc3aa4 (derive_event_axis from the planned chunks)	closed 2026-08-08
#791 — committed hazard_datasets row cannot describe a scenario × year source	PR #798 / 7f1a70e (ADR-067)	closed 2026-08-08
#793 — a single-slice source has no representable hazard_events shape	PR #800 / a6d5fd5 (ADR-068)	closed 2026-08-08
#797 — a frequency-less dataset computes a NaN EAI instead of being refused	PR #804 / afc042a (ADR-069)	closed 2026-08-09

And the guard still refuses every dispatch anyway. That is the current gate, and it is deliberate. The reason is not a missing fix: it is that no NetCDF ingest has ever been staged, chunked and committed through the chord in any environment. One EWDS retrieval has now been downloaded and opened — a hand-built cems-fire-historical-v1 request run against prod credentials on 2026-08-09 (#802, PR #806 / 13529a4) — and on first contact it falsified three of this pack family's declared assumptions at once: the in-file variable is fwinx not fwi, it carries an undeclared valid_time axis, and the request omitted data_format plus three other required inputs. Nothing about that file was ever ingested. The C3S projections half (sis-tourism-fire-danger-indicators) has never had a single file opened. Five successive "last" blockers were each found by reading the next step of the chord, so removing the guard is gated on a first successful end-to-end retrieval, not on code review.

#761 is OPEN and must stay open. It was repurposed after its two original code blockers were fixed: it is now the lead end-to-end evidence-gate issue that ADAPTER_GAP_ISSUE_URL points at. Do not close it on the strength of ADR-064, and do not read "its blockers closed" as "the issue closed" — an earlier revision of this doc asserted the latter in several places and was wrong. Reference it with Refs, never Resolves.

The guard raises before any retrieval is submitted, which is the point: without it the failure would land only after a real CDS/EWDS queue wait and a staged upload. Record the abort against #735 and move on to 4b.

The one-shot escape — --first-end-to-end-run. Gating the guard's removal on a successful run while refusing every run is a deadlock: the evidence demanded can only come from the dispatch that is forbidden. The single supported way out is an explicit operator escape — append --first-end-to-end-run to the EWDS command above, once, typed deliberately, never baked into a script, alias or compose command. It is a CLI flag rather than an environment variable precisely so it stays visible in shell history and cannot survive in a profile or compose file as a silent permanent bypass. It waives the evidence gate only; the plan-time adapter-resolution check still runs, so a spec that would hand a staged .nc to the GeoTIFF reader is still refused. The run logs a NETCDF_FIRST_END_TO_END_RUN WARNING, so ./dc logs api | grep NETCDF_FIRST_END_TO_END_RUN proves afterwards that it happened and against what. Read scripts/packs/_wildfire_fwi_common.py::NETCDF_INGEST_BLOCKERS on trunk for the live statement of the gate rather than trusting this paragraph. Then:

1. Verify the committed hazard_datasets / hazard_events rows actually describe the retrieved file — axis columns, slice count, no fabricated frequency — not merely that the chord reported success.
2. On success, delete the guard's layer-2 block (and only that block) in its own PR citing the run, then close #761.
3. On failure, leave the guard in place and file the blocker it found — that is the sixth one, and finding it this way is exactly what the guard exists to stop us paying for blindly.

Prefer EWDS over C3S for the first run: the C3S projections pack has a second, independent gate (the --dimension inputs this repo has no captured enum for, and no sis-tourism-fire-danger-indicators file has ever been opened).

Disk check — both packs' upper bounds sum to <1 MB; negligible.

Verification

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT haz_type, count(*) FROM hazard_datasets WHERE haz_type='wildfire';"

Ledger rows flipped: #614 → downloaded → partially ingested (EWDS half, 2026-08-10 — the C3S half stays short of ingested until rotated-pole reprojection exists).

Resume after interruption — non-zero count ⇒ at least one landed; check dataset name before re-running either. As of 2026-08-10 the count is 1 (wildfire-fwi-historical-greece-20230801); re-running the EWDS line would duplicate it.

4b — Heatwave (E7/E8, #616)

COMPLETE 2026-08-10 — both datasets retrieved and ingested, all scenarios. Three hazard_datasets rows, all status=active, all with COGs: heatwave-euro-cordex-europe-ssp5-8.5 (5857653d-4a63-40cc-95ce-3c2c19e54a01, 100 events 1986–2085, grid 599×425, 100 bands, 101.25 MB stored), heatwave-euro-cordex-europe-ssp2-4.5 (591da928-cee9-4693-a39d-6a4afbcf0f9f, 100 events 1986–2085, grid 599×425, 100 bands, 100.82 MB stored) and heatwave-ecde-europe (32fcd25e-59ec-4558-8b98-33365203cebd, 86 events 1940–2025, grid 271×185, 86 bands, 1.26 MB stored). Raw: EURO-CORDEX delivered a 203,731,118 B ZIP holding two 101,849,863 B members (HWD_EU_health_rcp45_mean_v1.0.nc, HWD_EU_health_rcp85_mean_v1.0.nc — the upstream filenames say rcp45/rcp85 while the catalog names them ssp2-4.5/ssp5-8.5); ECDE delivered a 34,518,182 B ZIP holding one 34,512,676 B member. Raw total 238,249,300 B ≈ 0.238 GB; processed total 203.33 MB. The two-members-per-ZIP delivery is handled by ADR-071/PR #822 — one ingest per member — which is why one retrieval produced two catalogued datasets. The sis-ecde-climate-indicators pick for the issue's "ERA5-HEAT" half is confirmed by execution: it yields heatwave days in days over 1940–2025, the same unit as the EURO-CORDEX primary, exactly as the ledger's [ASSUMPTION] predicted.

Open caveat, not a blocker: #827 (filed 2026-08-10, open) — derive_dataset_axis discards the per-member scenario label at commit time, so both EURO-CORDEX rows carry their scenario in the name but have scenario IS NULL in the column. Heatwave therefore cannot yet answer a scenario-labelled matrix cell. Do not report 4b as fully closed without saying so.

The commands below stay documented for a re-run:

./dc exec -T api python scripts/packs/heatwave_packs.py print-registry
./dc exec -T api python scripts/packs/heatwave_packs.py validate

validate reports unresolved request keys — as of 2026-08-10 there are none left to fill: the _UNRESOLVED_CONFIRM_FROM_CDS_DOWNLOAD_FORM sentinels were resolved and both retrievals succeeded, so the "a human fills them in from each dataset's live CDS Download tab" step this wave was gated on is done, not pending. The entries live in DATASET_SOURCE_COPERNICUS_RETRIEVALS (config, not code — restart the worker to reload Settings). These registry entries are what OPS-5a's live polling→review→approve→ingest cycle can exercise; with 4b landed, OPS-5a's stated gate is cleared, though OPS-5a itself has not been run.

Disk check — ~~raw volume is genuinely TBD (no anonymous CDS byte count)~~ measured 2026-08-10: 0.238 GB across the two ZIPs. Zipped-raster bucket: 2.5 × 0.238 ≈ 0.6 GB free. Still take a fresh live df -h / before a re-run.

Verification — same query pattern, haz_type='heatwave'. Expect 3 rows.

Ledger rows flipped: #616 → downloaded → ingested (done 2026-08-10, with the #827 caveat above).

Resume after interruption — check #735's comments for an already-submitted request ID before resubmitting; and check the three dataset ids above before re-ingesting anything, since a re-run would duplicate them.

4c — Windstorm, scoped (E6, #615)

./dc exec -T api python scripts/packs/windstorm_c3s_wisc.py print-registry
# register windstorm-c3s-footprints-hist + windstorm-wisc-synthetic via
# DATASET_SOURCE_COPERNICUS_RETRIEVALS, restart the worker, then:
./dc exec -T api python scripts/packs/windstorm_c3s_wisc.py ingest windstorm-c3s-footprints-hist --data-dir /data/windstorm_c3s_wisc

--data-dir must point under /data (see the OPS-1b note above, #748).

C3S historical footprints — DONE 2026-08-10. windstorm-c3s-footprints-hist, id 3f4f864d-01ab-485c-9f39-23c8856a0452, haz_type=storm_europe, status=active, COG present. One event committed: scenario NULL, year=2007, event_name=1463@2007, no frequency — a year-labelled, frequency-less single footprint. Grid 241×161, 13,174 non-zero cells, 1 band. The CDS delivery was a 54,965 B ZIP holding one 54,713 B member, and the stored footprint is 0.12 MB (124,719 B) — hazards/ 0.08 + processed/ 0.04. That download is three orders of magnitude below this doc's ≤0.04 GB (37.5 MB) per-footprint disk-check figure, and the figure is not wrong: it sized an unscoped whole-domain (25W–35E, 30N–70N) float32 band, while the retrieval actually submitted is scoped to a single storm. Keep the larger number as the ceiling for an unscoped request.

WISC synthetic — NOT RETRIEVED, and the question this paragraph posed is now answered. The paragraph read: "WISC scoping is a runtime-agent judgment call, not an owner input — inspect windstorm_c3s_wisc.py's WISC_SYNTHETIC request dict and the live CDS 'Show API request' panel for a subsetting field before submitting. Full set = 22,980 footprints ≈142.6 GB unscoped; this plan sets a hard ≤5 GB ceiling on what's actually retrieved [ASSUMPTION — planning bound, not verified against WISC metadata; the live disk-check abort below is the real safety backstop]. If CDS offers no subsetting (unconfirmed), don't retrieve WISC in this pass." CDS offers no subsetting — this is now measured, not unconfirmed. The sis-european-wind-storm-synthetic-events process schema, read live from GET {cds_root}/retrieve/v1/processes/sis-european-wind-storm-synthetic-events, publishes exactly four inputs: variable, version_id, year, month. There is no area and no spatial input of any kind. Every scoping lever is temporal, so any WISC retrieval fetches whole-domain footprints and the ≤5 GB ceiling cannot be met at retrieval time. Per this wave's own fallback, WISC was therefore not pursued for Phase 9. Record this as a measured negative: no ADR was written and none is implied — a later pass willing to accept whole-domain footprints, or that finds a subsetting path this schema read did not show, is free to revisit. The ledger's #615 note (a) ops action to re-confirm WISC's click-through licence is moot for the same reason: nothing is retrieved, so nothing is accepted.

If a future pass does resolve a scoping path, the retrieval uses the same subcommand:

./dc exec -T api python scripts/packs/windstorm_c3s_wisc.py ingest windstorm-wisc-synthetic --data-dir /data/windstorm_c3s_wisc

ingest fetches the configured item (cache-hit aware — a .nc already cached under --data-dir is not re-downloaded) and dispatches it through the unified pipeline in one call (#737, PR #760), mirroring the wildfire packs' ingest subcommand — register_netcdf_layouts() and scripts.ingest_scenario_hazards._dispatch_ingest now run automatically inside it, so the hand-written in-container Python snippet this doc used to call for is obsolete. SUPERSEDED 2026-08-10 — the paragraph that stood here ("It still aborts on the same assert_netcdf_adapter_wired guard as OPS-4a, by design ... what remains is the end-to-end evidence gate tracked on the still-open #761") is FALSE on both counts. #761 is CLOSED and PR #825 (42f2a35) deleted the evidence-gate layer, the --first-end-to-end-run escape and ADAPTER_GAP_ISSUE_URL; the surviving guard is the plan-time adapter-resolution check only, and STORM_EUROPE dispatches through it. The C3S ingest above is the proof. Read NETCDF_INGEST_BLOCKERS on trunk, per the OPS-4a note, rather than any issue number written into this doc.

Disk check — C3S historical footprint: the ≤0.04 GB/footprint bound gives 1.5 × 0.04 ≈ 0.06 GB, and the real scoped retrieval measured 54,965 B, so this is negligible in practice — keep the bound for an unscoped request. WISC scoped to ≤5 GB: 1.5 × 5 = 7.5 GB required free — moot while WISC is not retrievable under a scoping constraint (see above), but retained for any future pass.

Verification — hazard_datasets query, haz_type='storm_europe' (the enum value confirmed live 2026-08-10 by the ingested row, superseding this doc's earlier 'windstorm' guess and its "confirm the exact enum value in models/enums.py first" hedge).

Ledger rows flipped: #615 → downloaded → ingested (done 2026-08-10 — C3S historical half; WISC not retrieved).

Resume after interruption — re-run the same ingest command; it skips dispatch without re-fetching or re-submitting a CDS job if the dataset is already in the catalog (matched by name), and skips the CDS submission alone if --data-dir already holds a cached .nc from a prior attempt. Check #735 for an already-submitted WISC request ID first regardless.

Escalation — CdsRequestRejectedError ⇒ the reconstructed request body (built without live form access) doesn't match CDS's real schema; a human opens the live "Show API request" panel and corrects it — don't guess twice.

---

OPS-5 — Close-out

Five independent pieces; run in any order once the relevant prior waves have landed.

5a — Live polling→review→approve→ingest cycle (#597's exit demo)

Machinery: worker/dataset_polling.py (weekly beat) → core/dataset_polling.py::diff_and_queue_source → a PendingIngest row (awaiting_review) → GET /v1/admin/pending-ingests → POST /v1/admin/pending-ingests/{id}/approve (or /reject). Demo against whichever DatasetSource has a real, resolved retrieval — OPS-4b's heatwave entries if ready, else any already-configured one.

Status 2026-08-10 — the POLL half is PROVEN IN PRODUCTION; the review/approve half is BLOCKED, and not for a data reason. This supersedes both "gated on OPS-4b" and an interim note here that said the wave simply had not been run.

• Poll — DONE, unattended. The weekly beat fired on its own at 05:00 UTC on 2026-08-10 and queued 22 awaiting_review rows — jrc_flood ×1, worldpop ×21. No manual poll_dataset_sources.delay() was needed; the machinery works on its own schedule, which is a stronger result than the hand-triggered demo this wave was written for.
• Review/approve — BLOCKED by #828. Production has zero users with is_platform_admin = true, so every /v1/admin/* route 403s for everyone — including the GET /v1/admin/pending-ingests and POST /v1/admin/pending-ingests/{id}/approve calls below. This is a privilege/bootstrap gap, not a data gap and not an ops gap: there is nothing to re-ingest, nothing to re-poll and nothing to scope; someone has to be granted platform admin before the cycle can be closed. Do not "fix" it by re-running the poll or by re-ingesting a dataset.

OPS-4b landing did clear this wave's original gate — its CDS request values are resolved and its retrievals were delivered — so #828 is the only thing standing between here and a completed #597 exit demo.

Commands

# confirm the exact task name/signature against worker/dataset_polling.py first:
./dc exec -T worker python -c "
from climate_lama.worker.dataset_polling import poll_dataset_sources
poll_dataset_sources.delay()
"
docker exec climate-lama-api-1 curl -s -b /tmp/ops-session.txt \
  http://localhost:8000/v1/admin/pending-ingests

The pending-ingests GET response above is the preview; then:

docker exec climate-lama-api-1 curl -s -b /tmp/ops-session.txt -X POST \
  http://localhost:8000/v1/admin/pending-ingests/<id>/approve

Disk check — depends on which dataset is demoed; apply that pack's own check above.

Verification — the approved item's dataset appears in hazard_datasets/exposure_datasets via the standard DB query pattern. Record this cycle as done on #735 — a named phase-9 exit criterion.

Ledger rows flipped: whichever pack was demoed, if not already flipped.

Resume after interruption — check GET /v1/admin/pending-ingests for an already-awaiting_review item before polling again.

Escalation — no DatasetSource has a resolved retrieval yet ⇒ run OPS-4b first; this wave cannot demo a cycle against nothing. As of 2026-08-10 that escalation no longer applies — retrievals are resolved and 22 items are queued. The live escalation is #828: a 403 from any /v1/admin/* call here is expected, not a session/cookie problem. Do not burn a pass re-deriving the admin cookie; check SELECT count(*) FROM users WHERE is_platform_admin; first.

5b — Staging-orphan cleanup + orphan hazards/ NPZ audit

Commands — list first, always:

./dc exec -T api python -c "
from climate_lama.storage.client import get_minio_client
c = get_minio_client()
for o in c.list_objects('climate-lama-storage', prefix='staging/', recursive=True):
    print(o.object_name, o.size)
"

Cross-reference hazards/{dataset_id}/ prefixes against SELECT id FROM hazard_datasets; for the orphan.

The listing + cross-reference above is the preview. Delete the specific identified keys — never a prefix-wide delete.

Verification — re-run the listing; confirmed keys gone, nothing else changed.

Ledger rows flipped: none.

Resume after interruption — re-run the listing; orphans already gone ⇒ done.

Baseline updated 2026-08-10 — the original one is obsolete and would mis-fire. This wave's authored baseline (3 staging/ dirs ≈237 MB, 1 orphan hazards/ NPZ) describes the pre-cleanup state and no longer exists. After the 2026-08-08 run: 16 staging/ dirs, ≈13 KB total, all legitimate, no orphan NPZ, 24 hazards/ prefixes against 25 hazard_datasets rows (the one gap is a failed job). An agent applying the old numbers would STOP on a healthy system. The mechanism also changed under ADR-066 (#784, PR #795 / 9568595): source provenance now lands at hazards/{dataset_id}/source_manifest.json and a successful ingest purges staging/ wholesale, so post-ADR-066 ingests contribute no lasting staging/ dirs at all. The 16 that remain are pre-ADR-066 manifest.json leftovers from succeeded jobs and are the only source provenance those datasets have — do not delete them.

trash/ PURGED 2026-08-10 — the baseline above is otherwise unchanged. The trash/2026-08-08-ops5b/ prefix (28 objects / 237.40 MB, the quarantined output of this wave's 2026-08-08 run) is now empty. Deleted key by key, never prefix-wide, after three checks all came back clean: all four referenced UUIDs were absent from hazard_datasets, none was a live ingest_jobs row, and no dataset had a cog_path pointing under trash/. staging/ is untouched at 16 dirs / ≈13 KB — the purge did not go near it, and those 16 must still be left alone per the note above. Post-purge bucket: 158 objects / 359.87 MB.

Escalation — a staging/ count materially above the current baseline (16 dirs / ≈13 KB), or any new dir written by a post-ADR-066 ingest, or a hazards/ NPZ with no hazard_datasets row ⇒ STOP, record — a growing count outside this wave's own activity is a code question, not a cleanup one.

5c — GHSL USD→m3 row correction (if OPS-3c didn't already supersede it)

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT id, name, value_unit, created_at FROM exposure_datasets WHERE name ILIKE '%ghs%' ORDER BY created_at;"

If a value_unit='USD' row still exists alongside (not superseded by) a fresh m3 row: confirm exposure_datasets actually has a status-like column before writing any UPDATE (\d exposure_datasets via the same psql recipe) — ADR-048 doesn't guarantee the hazard-side mechanism transfers. If it doesn't exist, STOP and record on #735 rather than inventing a correction mechanism — this needs a real schema/design decision, not a guessed one.

Same preview-then-act pattern as OPS-1c — no owner confirmation required.

Verification — the SELECT above shows no live USD-labelled GHSL row.

Ledger rows flipped: none (data-quality fix, not a new pack).

Resume after interruption — re-run the SELECT; already clean ⇒ skip.

5d — Storage bill vs volume-budget ledger check

./dc exec -T api python -c "
from climate_lama.storage.client import get_minio_client
c = get_minio_client()
total = sum(o.size for o in c.list_objects('climate-lama-storage', recursive=True))
print(f'{total / 1e9:.2f} GB total in bucket')
"

Compare against volume-budget.md's Totals row (≈24.87 GB raw across the ten packs as recomputed 2026-08-10 — this now includes #616's measured 0.238 GB and still excludes the WISC full-set hypothetical; it supersedes the ≈24.64 GB figure this line used to cite). Record the comparison as a #735 comment — a named exit criterion, not a pass/fail gate on its own.

Last measurement, 2026-08-10 (post-OPS-4, post-trash/-purge): 158 objects, 359.87 MB (0.360 GB) — superseding both the 0.325 GB read on 2026-08-08 and the 528.30 MB / 185-object intermediate reading taken earlier the same day, before the purge. Per zone:

Zone	Objects	Size
hazards/	55	200.72 MB
processed/	47	158.75 MB
raw/	2	0.04 MB
reports/	7	0.24 MB
results/	30	0.12 MB
staging/	16	0.01 MB
tiles/	1	0.00 MB
trash/	0	— (purged 2026-08-10)

The zone column adds to 359.88 MB against a reported bucket total of 359.87 — a 0.01 MB artifact of rounding each zone independently, not a missing object. Two movements explain the delta from the pre-purge reading: trash/ 28 obj / 237.40 MB → 0 (see OPS-5b's purge note), and processed/ 89.77 → 158.75 MB from the COG rebuild below. The bucket is now almost entirely live data — hazards/ + processed/ = 359.47 of 359.87 MB, of which #616's three heatwave datasets are 203.33 MB. The 0.360 GB-vs-24.87 GB gap is the same expected one ADR-070 ratified: raw is clip-then-delete, never archived.

Storage integrity audit, 2026-08-10 — run in BOTH directions, clean:

Check	Result
ACTIVE datasets with no cog_path	0
ACTIVE whose cog_path is absent from the bucket	0
storage UUIDs with no catalog row	0
any-status rows with no intensity.npz	2 — both superseded, both known failed-stage rows

Both directions matter: catalog→storage catches a row promising an object that is not there, storage→catalog catches an object no row owns. Only the fourth line is non-zero and it is accounted for.

COG rebuild — #754 confirmed fixed in production, on the dataset that broke it. The first check above returned 0 only after a repair: JRC river flood Greece (national, historical baseline) (67854ef5-8052-435d-9828-1a1015146654) had been active since 2026-08-07 with its intensity.npz intact but cog_path NULL. That is the same dataset that originally triggered #754 — a classic-TIFF write overflowing 4 GB, at 11,400×9,000 × 9 bands ≈ 3.7 GB uncompressed. Re-dispatching build_dataset_cog from the retained NPZ succeeded in 141.6 s, 9 bands, zero GDAL errors. #754's fix is therefore proven against the real failing dataset rather than a fixture, and this is the entire reason processed/ grew by ≈68.98 MB (a figure derived by difference, not a directly measured object size).

Ledger rows flipped: none — reads the ledger, doesn't edit it. ~~(a future PR updates Status/Totals from real ops results)~~ that PR is #829, which carried every measurement on this page into volume-budget.md.

Resume after interruption — read-only; re-run any time.

5e — Two-sided ≥6-haz-type check (closing condition for #735)

EXECUTED AND PASSED 2026-08-10 — #735 is CLOSED on this evidence.

haz_type	count (active)
coastal_flood	7
earthquake	1
heatwave	3
river_flood	8
storm_europe	1
wildfire	1

6 distinct types, 21 active datasets, and both sides agree exactly. The DB read above and the API read match cell for cell: GET /v1/hazards filtered to status=active returns 21 rows across the same 6 types with an identical breakdown, unfiltered returns 31 rows across the same 6 types, and a per-type ?haz_type= round-trip returns 200 with matching counts for all six. The #729 re-read-after-60s protocol below was not needed — there was no disagreement to resolve.

Read the DB side on a psql SUPERUSER connection. The app connection returns zero rows for this query under RLS. An agent that uses the app connection here will conclude the catalog is empty and STOP on a healthy system — the exact failure mode the #729 trap below warns about, arriving by a different route.

curl -fsS -H "Cookie: <session>" \
  "https://api.climate-lama.online/v1/hazards?limit=200" \
  | jq -r '.data[] | "\(.haz_type)\t\(.source)\t\(.name)"' | sort | uniq -c

docker exec climate-lama-postgres-1 psql -U climate_lama -d climate_lama -c \
  "SELECT haz_type, count(*) FROM hazard_datasets GROUP BY haz_type;"

Expected: both show ≥6 distinct haz_type values, in agreement. On disagreement: re-read after 60s (#729 trap) — never re-ingest on one disagreeing read. Still disagreeing after a second read ⇒ STOP, record both outputs on #735 — this is the closing condition for the whole tracking issue, so don't close it on an unconfirmed single read.

Escalation — fewer than 6 haz_type values with all of OPS-1–OPS-4 complete ⇒ recount against this doc's own wave list before assuming a data gap; a wave may have silently no-op'd.