Skip to content

Troubleshooting Runbook — scripts/smoke_e2e.py and the local dev stack

This runbook is for triaging a failing run of scripts/smoke_e2e.py — the end-to-end walk that exercises the full vertical slice (bootstrap → load an admin boundary → ingest hazard → upload exposure → assign centroids → compute impact → read back results) against a live docker compose up stack (docker-compose.yml). It replaced the old demo/smoke script (scripts/demo.py), deleted whole alongside the rest of the demo scaffolding — issue #892 — because it drove a bespoke seeded path rather than the real user journey; see ADR-077.

Every command and claim below is grounded in a committed file — cited inline — so this stays trustworthy even though it can't be exercised against a running stack in this environment (no Docker/PostGIS here). Where something is a genuinely external convention (e.g. how Celery keys its Redis results) rather than something this repo defines, it's called out as such.

For production operations (release flow, backups, the Hetzner host), see DEPLOYMENT.md instead — this doc is about the local dev stack (docker-compose.yml), which is what scripts/smoke_e2e.py targets by default (--base-url http://localhost:8000).

Before you start — a checklist that catches most first-run failures

  1. .env exists, with CL_BOOTSTRAP_ADMIN_EMAIL / CL_BOOTSTRAP_ADMIN_PASSWORD set. cp .env.example .env (README.md "First-time setup"), then fill in both — there is no default credential, so leaving them blank leaves the environment with no admin (see DEPLOYMENT.md). The migrations, bootstrap, api, worker, and beat services all declare env_file: .env (docker-compose.yml) — docker compose up refuses to start them without the file existing.
  2. The stack is up: docker compose up (or docker compose --profile fullstack up for the full UI+API slice, per CLAUDE.md's "Demoable MVP Slice"). The bootstrap one-shot runs after migrations on every up and provisions the organization + admin from step 1 — docker compose logs bootstrap confirms it ran (Bootstrap complete: … or Already bootstrapped: …, both success).
  3. scripts/smoke_e2e.py needs requests importable — it's a module-level import (scripts/smoke_e2e.py) with no other project dependency, so any Python 3 environment with requests installed can run it (e.g. uv pip install -e ".[dev,worker]", the install path .github/workflows/sdk-smoke.yml uses in CI).
  4. The hazard-ingest payload needs a raster staged on disk that a clean clone doesn't have. By default the walk ingests the tiny tracked reference raster, tests/fixtures/reference_datasets/river_flood.tif, from the worker-visible path /data/river_flood.tif (DEFAULT_HAZARD_TIF, scripts/smoke_e2e.py). That path is served by the ./data:/data:ro bind mount on the worker service (docker-compose.yml), and data/ is gitignored (.gitignore), so a fresh clone has nothing there until you stage it — the same step .github/workflows/sdk-smoke.yml's "Stage the reference hazard raster for the worker" step runs in CI: 
    mkdir -p data
cp tests/fixtures/reference_datasets/river_flood.tif data/river_flood.tif
    Point at a different raster with --hazard-tif (or $SDK_SMOKE_HAZARD_TIF) if you need one.
  5. Every stage past bootstrap needs Role.ANALYST, and the walk fails immediately without it — unlike the retired demo script, there is no best-effort skip-on-401 (see scripts/smoke_e2e.py's module docstring, "Authentication"). If the compose bootstrap one-shot already provisioned an account (step 2 above), point the walk at those same credentials and skip re-provisioning: 
    SDK_SMOKE_EMAIL="$CL_BOOTSTRAP_ADMIN_EMAIL" \
SDK_SMOKE_PASSWORD="$CL_BOOTSTRAP_ADMIN_PASSWORD" \
SDK_SMOKE_ORG_SLUG="${CL_BOOTSTRAP_ORG_SLUG:-default}" \
  python scripts/smoke_e2e.py --skip-bootstrap
    Drop --skip-bootstrap (and the SDK_SMOKE_* vars, using --role instead) to have the walk provision its own account via scripts/bootstrap_env.py — needs a privileged DB role, see DEPLOYMENT.md.

Quick triage — where to look first

  1. GET /health (liveness — process is up) then GET /health/ready (readiness — probes Postgres, Redis, and MinIO individually and reports per-check detail) — src/climate_lama/main.py health() / health_ready(). scripts/smoke_e2e.py's wait_for_health() (step 3, unless --skip-bootstrap / --skip-boundaries move it earlier) polls GET /health until it answers or times out, so a dependency outage that keeps the API from ever coming up surfaces there; a readiness-level failure (Postgres/Redis/MinIO individually) instead shows up as the first stage that touches the broken dependency failing its resp.raise_for_status().
  2. docker compose ps — which containers aren't running/healthy.
  3. docker compose logs <service> --tail=100 for whichever check health/ready flagged.

Postgres

Symptoms: GET /health/ready reports "postgres": {"status": "error", "detail": ...} (main.py); scripts/smoke_e2e.py fails at step 1 (bootstrap_env.py needs a reachable DB to even start) unless --skip-bootstrap is passed, in which case it instead fails at step 3's health wait.

First three checks:

  1. docker compose ps postgres — healthcheck is pg_isready -U climate_lama (docker-compose.yml).
  2. docker compose logs postgres --tail=100 (container name climate_lama_postgres, docker-compose.yml).
  3. docker compose logs migrations — the one-shot migrations service runs alembic upgrade head and only starts once Postgres is healthy (depends_on: postgres: condition: service_healthy, docker-compose.yml); api, worker, and beat all wait on migrations: condition: service_completed_successfully, so a wedged or failed migration blocks every other service, not just Postgres.

Gotcha — port mismatch between alembic.ini and docker-compose.yml: the compose Postgres is published on host port 5433, not 5432 ("5433:5432", docker-compose.yml), but migrations/alembic.ini's own default sqlalchemy.url still points at localhost:5432. Running alembic upgrade head / alembic current directly from the host (outside the migrations container — e.g. the Quickstart's uv run alembic upgrade head, docs/quickstart/index.md) needs DATABASE_SYNC_URL exported first, or it fails closed with a connection error: 

export DATABASE_SYNC_URL="postgresql+psycopg2://climate_lama:password@localhost:5433/climate_lama"
uv run alembic upgrade head
(migrations/env.py reads DATABASE_SYNC_URL from the environment and overrides alembic.ini's default when it's set.)

Redis

Symptoms: GET /health/ready reports "redis": {"status": "error", ...} (main.py); scripts/smoke_e2e.py jobs never leave pending (poll_job() keeps printing ... {label}: pending — the worker can't receive tasks without the broker).

First three checks:

  1. docker compose ps redis — healthcheck is redis-cli ping (docker-compose.yml).
  2. docker compose exec redis redis-cli ping.
  3. Broker and result backend are two different logical Redis databases on the same instance — broker on db 0, results on db 1 (CELERY_BROKER_URL=redis://redis:6379/0, CELERY_RESULT_BACKEND=redis://redis:6379/1, docker-compose.yml; wired the same way in src/climate_lama/worker/celery_app.py via Celery(..., broker=settings.celery_broker_url, backend=settings.celery_result_backend)). If you're inspecting keys by hand, pick the right db: docker compose exec redis redis-cli -n 0 keys '*' for queued tasks, -n 1 for stored results.

MinIO

Symptoms: GET /health/ready reports "minio": {"status": "error", "detail": "bucket 'climate-lama' does not exist"} — MinIO itself can be reachable while the bucket is still missing, because health_ready() explicitly checks bucket_exists, not just connectivity (main.py). Hazard-ingest or exposure-upload jobs fail while staging to object storage.

First three checks:

  1. docker compose ps minio minio-init — minio's healthcheck is mc ready local; minio-init is a one-shot job that creates the bucket and only prints Bucket ready. on success (docker-compose.yml, minio-init entrypoint: mc alias set local http://minio:9000 minioadmin minioadmin && mc mb --ignore-existing local/climate-lama).
  2. docker compose logs minio-init — confirms whether the bucket-creation step actually ran and succeeded.
  3. Inspect the bucket directly, mirroring minio-init's own alias pattern: 
    docker compose exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
docker compose exec minio mc ls local/climate-lama
    (credentials are .env.example's dev defaults: MINIO_ACCESS_KEY=minioadmin / MINIO_SECRET_KEY=minioadmin.) Or use the browser console at http://localhost:9001 (docker-compose.yml port mapping 9001:9001; also listed in README.md's service table).

API

Symptoms: any scripts/smoke_e2e.py stage raising via resp.raise_for_status(); an unauthenticated run fails immediately at step 3 rather than 401ing partway through — every stage past that point needs Role.ANALYST and there is no best-effort skip (see the module docstring's "Authentication" section).

First three checks:

  1. GET /health then GET /health/ready (main.py) — establishes whether the API process itself is up before treating a failure as request-specific.
  2. GET /v1/info — reports version, environment, git_sha and features (main.py info()), so you know which build actually answered and which feature lines it has switched on (features.live_mode, issue #1280; features.geocoding, issue #1336 — the latter is geocoder_enabled AND geocoder_advertised, so a false there does not by itself mean the geocoder is disabled). Useful when comparing a locally docker compose --build-ed API against a pulled ghcr.io/cortomaltese3/climate-lama:${CLIMATE_LAMA_TAG} image (docker-compose.yml).
  3. docker compose logs api --tail=100 (container climate_lama_api). The dev api service mounts ./src and runs with --reload (command: uvicorn climate_lama.main:app --host 0.0.0.0 --port 8000 --reload, docker-compose.yml), so a stack trace from a local edit shows up here immediately.

Reading the error envelope: every API error follows one shape — {"error": {code, severity, message_en, details}, "meta": {request_id, timestamp}} (ADR-028; exercised directly by scripts/smoke_e2e.py's ingest and compute stages). Match error.code against the Code enum in src/climate_lama/core/errors.py, e.g.:

  • E_ENGINE_UNKNOWN (422) / E_ENGINE_UNAVAILABLE (400) — the optional X-Engine header on POST /v1/compute/impact picked an unregistered engine name, or a registered-but-not-deployed one (src/climate_lama/api/v1/compute.py).
  • E_INGEST_CHORD_TIMEOUT / E_INGEST_WEDGED — hazard ingest stalled past its time budget; see the Worker section below.
  • E_GEOCODE_UNAVAILABLE (503) — GET /v1/geocode when the opt-in geocoding profile (self-hosted Nominatim) isn't running (docker-compose.yml, nominatim service). Not exercised by scripts/smoke_e2e.py, which never calls that endpoint.
  • E_SERVICE_BUSY (503, Retry-After: 5) — a request timed out waiting for a database connection (a SQLAlchemy pool checkout or the asyncpg connect), or on another bounded wait (main.py timeout_error_handler). The API is overloaded, not broken: retry. A burst of these alongside unanswered /health/live probes means something is blocking the single uvicorn event loop — check docker compose logs api --timestamps for a gap between lines (issue #1457). A vector tile route (/v1/tiles/results|exposures|portfolios/…) also answers it on purpose, with the cause in details.reason (issue #1804): tile_query_timeout — PostgreSQL cancelled the tile's statement at VECTOR_TILE_STATEMENT_TIMEOUT_MS (default 5000); the tile is slow, so look for the statement in pg_stat_activity / the PostgreSQL log (canceling statement due to statement timeout) rather than raising the timeout first. tile_concurrency_cap — VECTOR_TILE_MAX_CONCURRENCY (default 8) tile requests were in flight in that API process and none finished within VECTOR_TILE_SLOT_WAIT_MS (default 2000) of this one arriving. The map does not ask again for a tile answered 503: that tile stays blank until the next pan or zoom, when the map asks for it afresh. A steady stream of these means tiles are slow enough to pile up. Neither reason on any other route's 503 means the pool itself ran dry (DB_POOL_SIZE + DB_MAX_OVERFLOW, default 5 + 10). The API log names which: … -> 503 E_SERVICE_BUSY: TileQueryTimeoutError, TileConcurrencyCapError, or the driver's timeout type.

Worker

Symptoms: ingest/compute jobs stuck at pending/running past scripts/smoke_e2e.py's --timeout (default 600s) — poll_job() raises TimeoutError: Job ... did not complete within 600s. GET /v1/hazards/ingest-jobs/{id} (scripts/smoke_e2e.py step 4) shows status stuck at writing/aggregating with progress_pct not moving.

First three checks:

  1. docker compose ps worker — note: the dev docker-compose.yml's worker service defines no healthcheck (unlike docker-compose.prod.yml's worker, which probes celery -A climate_lama.worker.celery_app inspect ping -d celery@$$HOSTNAME), so docker compose ps can show worker as merely "running" even if Celery itself failed to initialize on boot — don't trust the state column alone, check logs.
  2. docker compose logs worker --tail=100 — on boot, Celery prints its resolved task list; compare against the include=[...] list in celery_app.py (e.g. climate_lama.worker.tasks, climate_lama.worker.ingest.stage_source, ...). A task missing from that boot-time list means its module failed to import.
  3. docker compose exec worker celery -A climate_lama.worker.celery_app inspect active (also try inspect registered, inspect ping) — the same -A climate_lama.worker.celery_app invocation the worker/beat services themselves use (command: celery -A climate_lama.worker.celery_app worker --loglevel=info --queues=compute,impact, docker-compose.yml); confirms the worker process is actually consuming the compute and impact queues. On prod, impact runs are on impact-worker (--queues=impact), so an impact job stuck pending there is triaged with docker compose exec impact-worker ….

If a job looks permanently wedged rather than merely slow: the chord-timeout and chunk-wedge watchdogs (sweep-wedged-ingest-chords every 10 minutes, sweep-wedged-writing-ingest-jobs every ingest_wedged_writing_sweep_minutes minutes, default 10 — both in celery_app.py's beat_schedule) only run if the separate beat service is up: "Without this service those tasks never run anywhere" (docker-compose.yml, comment on the beat service). docker compose ps beat is worth checking any time an ingest job looks stuck for good — with beat down, a wedged job never gets flipped to failed and just sits there past ingest_chord_timeout_seconds (default 6h) or ingest_wedged_writing_ttl_seconds (default 45m) (config.py).

Live worker — the live queue

What it is: a second Celery consumer (live-worker, container climate_lama_live_worker) on the same image as worker, bound to the live queue only (celery … worker --loglevel=info --queues=live --concurrency=1). It exists so a live-feed tick — which fires every few minutes — never queues behind the multi-hour ingest and impact work on compute. Un-profiled in all three compose files, so a plain docker compose up (dev) and deploy-prod.yml's up -d --remove-orphans (prod) both start it; there is no profile to enable and no separate image to pull.

Symptoms: live feeds never advance — last_polled_at frozen — while ordinary ingest/compute jobs run fine (that split is the tell: compute has a consumer, live does not). Nothing appears in the live-worker logs at all.

First three checks:

Check What it tells you
docker compose ps live-worker Is the container up? Like dev's worker it has no healthcheck in docker-compose.yml (prod's does — the same celery … inspect ping -d celery@$$HOSTNAME probe), so "running" alone does not prove Celery initialised — read the logs too.
docker compose logs live-worker --tail=100 The boot banner lists the queues this process consumes. It must show live — if it shows compute, the command: override was lost and the container is running the image's default CMD (--queues=compute,impact), leaving live with no consumer.
docker compose exec live-worker celery -A climate_lama.worker.celery_app inspect active Confirms the process is reachable on the broker. inspect registered should list the live-feed tasks; a task missing there means its module failed to import.

Tick fires but nothing happens (no error): check LIVE_MODE_ENABLED in the container, not in the file — docker compose exec live-worker printenv LIVE_MODE_ENABLED. All four services that care (api, worker, live-worker, beat) declare it the same way, as ${LIVE_MODE_ENABLED:-false} in their own environment: block, so they resolve identically from one source. Unset or false makes the tick a logged no-op by design. Check beat too — it is the scheduler that fires the tick, and a beat/live-worker disagreement looks exactly like "nothing happens": docker compose exec beat printenv LIVE_MODE_ENABLED. (Before #1281 beat read the flag only through its env_file, so a value exported in the shell rather than written to .env/.env.prod reached the workers and not the scheduler.)

live-worker up but the logs show a missing live_feeds relation: this is the pre-migration window, not a broken deploy. deploy-prod.yml rolls the containers (up -d --remove-orphans) before running alembic upgrade head — the migration runs inside the new api container after a health wait — so on a release that adds or changes the live-feed tables, live-worker and beat come up against the old schema and a scheduled tick can fire in that gap. The tick tolerates it (no-op, logged) rather than crash-looping. Resolve it by restarting the two once migrations are at head:

# on the prod host, from /opt/climate-lama
P="-f docker-compose.prod.yml -f docker-compose.hetzner.yml --env-file .env.prod"
docker compose $P exec -T api alembic current          # must print '(head)'
docker compose $P restart live-worker beat
docker compose $P logs live-worker --tail=20          # fresh boot banner, queue `live`

If the relation is still missing after that, the migration itself did not land — triage that, not the worker.

Queue depth: the dev monitoring profile's redis-exporter exports every queue as key-size metrics (REDIS_EXPORTER_CHECK_SINGLE_KEYS: "0=compute,0=impact,0=live", docker-compose.yml). Every entry is db 0 because a Celery queue is a Redis list named after the queue in the broker database (CELERY_BROKER_URL = redis://redis:6379/0); db 1 is the result backend and holds no queue lists. Query it as redis_key_size{key="live"}, or read it straight from Redis: docker compose exec redis redis-cli -n 0 llen live.

Reading a job's state without the API

  • Coarse: GET /v1/jobs/{job_id} (src/climate_lama/api/v1/jobs.py) — status (pending/running/completed/failed), celery_task_id, result_id, error_message.
  • Fine-grained (hazard ingest only): GET /v1/hazards/ingest-jobs/{ingest_job_id} (src/climate_lama/api/v1/hazards.py) — status walks pending → running → chunking → writing → aggregating → committed → succeeded (terminal states failed/cancelled), per IngestJobStatus (src/climate_lama/models/enums.py), plus progress_pct, chunks_total/chunks_done, error_code, and paginated logs (?logs_offset=N).
  • Directly in Redis (last resort — only when even the API is unreachable): the result backend lives on db 1 (CELERY_RESULT_BACKEND=redis://redis:6379/1, docker-compose.yml). Celery's Redis backend keys results as celery-task-meta-<task_id> — this is Celery's own convention, not something this repo defines, so treat the exact key format as unverified against a specific Celery version rather than a repo guarantee: 
    docker compose exec redis redis-cli -n 1 get "celery-task-meta-<celery_task_id>"
    Get <celery_task_id> from GET /v1/jobs/{job_id}'s celery_task_id field above. The DB-backed Job / IngestJob rows via the API are the supported way to check status.

The bell said a job failed — triaging a JOB_FAILED notification

JOB_FAILED (job_failed) is the sixth NotificationType (src/climate_lama/models/enums.py), added in Phase 14 (issue #1182). It means one thing precisely: a jobs (or ingest_jobs) row was moved to failed and the submitter was told. Before it existed, failure paths wrote status / error_message onto the row and nobody was ever notified, so "no notification" used to be indistinguishable from "no failure" — it is not any more.

It is emitted from exactly one place per lane, and those are the only writers of JobStatus.FAILED: JobRepository.mark_failed (async), mark_job_failed_sync / reap_jobs_sync (worker and beat sweeps) and IngestJobRepository.mark_failed (job_repository.py, ingest_job_repository.py). A static (AST) guard, tests/test_failed_writers_guard.py, fails the build if anything outside db/repositories/ assigns JobStatus.FAILED — so a failed job with no notification is a bug in one of those four functions, not a missing call site somewhere in the tree.

The payload (built once by core/notifications.py::job_failed_payload, so the async and worker lanes cannot drift):

Key Meaning
job_id The jobs row — the deep-link target. null when the failing work has no jobs row (several ingest dispatch paths never create one)
job_type e.g. impact_calc, data_ingest
error_code The machine-readable half: a Code from the ADR-028 registry (core/errors.py)
error_message Free text, capped at the 1024 characters jobs.error_message stores
title Ready to render, from core/job_titles.py::derive_failure_title — the same resolver that names the GET /v1/jobs row
ingest_job_id Ingest lane only; absent on the compute lane

Where to read the error_code. Three surfaces, same value, no parsing of English:

curl -s "$BASE/v1/jobs/$JOB_ID"          | jq '.data.error_code, .data.error_message'
curl -s "$BASE/v1/jobs?status=failed"    | jq '.data.items[] | {id, title, error_code}'
# ingest lane, when the payload's job_id is null — use ingest_job_id instead
curl -s "$BASE/v1/hazards/ingest-jobs/$INGEST_JOB_ID" | jq '.data.error_code, .data.logs'

or straight from the database when the API is unreachable:

select id, job_type, status, error_code, left(error_message, 200), completed_at
from jobs where status = 'failed' order by completed_at desc limit 20;

The first two things to check:

  1. Read error_code, not error_message. A specific code is a verdict about the input, not the stack — E_HAZARD_INVALID_SOURCE means the file was refused, and no amount of restarting the worker changes that. Look the code up in core/errors.py's registry (it carries a severity and a one-line meaning) and stop there if it names the fault.
  2. If the code is the generic E_JOB_FAILED, ask whether the job was reaped rather than failed. That code is the fallback used when nothing more specific fits — including by the beat sweeps in worker/job_watchdog.py, which close rows whose worker died. A completed_at far later than started_at (hours, at one of the sweep TTLs) is a reaped row, not an execution failure: the work never reported back, so read the next section rather than the error text.

A job stuck in running that never resolves

select id, status, job_type, created_at, started_at, error_message
from jobs where status in ('pending', 'running') order by created_at;

A row here that is hours (or days) old with a NULL error_message is stranded, not slow: its worker was killed mid-task, its broker message was lost, or a failure path closed the ingest_jobs row without closing this one. Three sweeps exist, in increasing order of how long they wait (src/climate_lama/worker/celery_app.py registers all three in beat_schedule):

Sweep Watches Fires after Closes the jobs row?
sweep_wedged_writing_ingest_jobs chunk liveness of a writing ingest INGEST_WEDGED_WRITING_TTL_SECONDS (45m) yes
sweep_wedged_ingest_chords any non-terminal ingest_jobs row INGEST_CHORD_TIMEOUT_SECONDS (6h) yes
sweep_stale_jobs the jobs row itself, any job type JOB_STALE_TTL_SECONDS (24h, floored at the chord timeout) yes

The last one (src/climate_lama/worker/job_watchdog.py) is the backstop: it is the only one that covers a job with no ingest_jobs row at all (POST /v1/hazards/{id}/assign-centroids, impact_calc, report_render). Check Celery beat is actually running before concluding a row is un-reapable — with no beat process, none of the three ever fire.

To inspect or close stranded rows now, without waiting for a tick:

python scripts/reap_stale_jobs.py            # dry-run (default) — lists candidates, writes nothing
python scripts/reap_stale_jobs.py --apply    # flips them to 'failed' with the watchdog's message

Both are idempotent and never touch a job inside the TTL. Reaping only closes the book-keeping row — it does not roll back partial work; re-submit the request to retry.

Retrying a step without rerunning the whole script

scripts/smoke_e2e.py has no per-step retry/resume flag. It does have two skip-a-stage flags — --skip-bootstrap and --skip-boundaries (the latter applies to every leg) — but those exist to point the walk at an already-provisioned stack (e.g. one whose compose bootstrap one-shot already ran), not to resume from an arbitrary failed step. There is no documented resume-from-step-N mechanism beyond that — inventing one here would be worse than not documenting it.

Since issue #1045 the walk runs once per country leg (Greece, then Cyprus, on the same org). A pass prints one E2E_READY_GRC hazard_id=... exposure_id=... result_id=... line and one E2E_READY_CYP hazard_id=... exposure_id=... result_id=... line (format_ready_line(), one call per leg) naming every id that leg produced; any other outcome exits 1 with the failing leg and stage named, e.g. [CYP] ....

A gated leg prints its own line only when armed, and sdk-smoke.yml requires it anyway, so an unarmed leg fails the job rather than vanishing:

Leg Line and what it carries Armed by Run it locally
Series (#1296) E2E_READY_SERIES_GRC live_event_id=… feed_id=… snapshots=N results=N elapsed=…s — the first watch and its replay feed; N is the staged series' own length (len(manifest.observations), 1 for the vendored Evia 2021 series), and results counts results the feed auto-computed (origin=feed), not the leg's one manual compute SDK_SMOKE_SERIES_DIR or --series-dir: the worker-visible path of the staged series (sdk-smoke.yml exports /data/live_series/evia_2021) Stage it where live-worker's ./data:/data:ro mount sees it: mkdir -p data/live_series/evia_2021 && cp tests/fixtures/reference_datasets/live_series/evia_2021/manifest.json tests/fixtures/reference_datasets/live_series/evia_2021/*.tif data/live_series/evia_2021/. Bring the stack up with LIVE_MODE_ENABLED=true (the leg needs live-worker and beat), then SDK_SMOKE_SERIES_DIR=/data/live_series/evia_2021 python scripts/smoke_e2e.py --skip-bootstrap

A red series leg names its stage. was paused quotes the feed's paused_reason — an unstaged or empty directory reads replay series manifest is unreadable. A TimeoutError after 300 s carries the snapshot/scored counts and the feed's last_error; read the live-worker logs next (see "Live worker" above).

Two grounded workarounds instead:

  1. Capture the ids from a passing run's E2E_READY_GRC / E2E_READY_CYP line, then call the specific downstream endpoint yourself — e.g. re-POST /v1/compute/impact with the captured hazard_dataset_id / impact_function_id (src/climate_lama/api/v1/compute.py) — instead of rerunning the whole script.
  2. Be aware reruns are not idempotent for hazard ingest: POST /v1/hazards/ingest (src/climate_lama/api/v1/hazards.py) creates a new IngestJob and hazard-dataset row on every call, with no dedup-by-name check. Rerunning scripts/smoke_e2e.py against an already-provisioned stack accumulates additional "Smoke river flood (reference tile)" (Greek leg) and "Smoke river flood (reference tile, CYP)" (Cyprus leg) datasets rather than reusing the earlier ones — and once a leg's name is no longer unique, that leg's own walk fails loudly at the "resolve the ingested dataset" stage (_unique_match()) instead of silently picking one. Running the walk through scripts/smoke_walk_wsl.sh --clean-slate (below) sidesteps this entirely — it resets the database and /uploads to empty before each run, so there is nothing left to accumulate.

Reference — dev stack ports and container names (docker-compose.yml)

Service Container name Host port(s) Notes
postgres climate_lama_postgres 5433 → 5432 PostGIS 16; not 5432 on the host
redis climate_lama_redis 6379 broker on db 0, result backend on db 1
minio climate_lama_minio 9000 (S3 API), 9001 (console)
migrations climate_lama_migrations — one-shot alembic upgrade head
api climate_lama_api 8000 --reload, mounts ./src read-write
worker climate_lama_worker — --queues=compute,impact; no healthcheck in dev
live-worker climate_lama_live_worker — --queues=live --concurrency=1; same image as worker, no healthcheck in dev
beat climate_lama_beat — drives the watchdog / nightly-maintenance schedule
titiler climate_lama_titiler 7800 → 80 COG raster tiles

Real-stack walk on WSL — scripts/smoke_walk_wsl.sh

scripts/smoke_walk_wsl.sh is the whole walk as one command: it rebuilds the COPY-based images from the checkout, brings the fullstack profile up, asserts alembic_version against the repo's highest migration, runs scripts/smoke_e2e.py, probes POST /v1/hazards/upload for a staged-file leak, and prints a provenance block to paste into an issue as evidence. It restores everything it touched from an EXIT trap, so an interrupted run leaves no dirty file and no stray stack.

scripts/smoke_walk_wsl.sh                                    # walk the current trunk
scripts/smoke_walk_wsl.sh --require-merged '1234:^fix\(' \
    --require-symbol src/climate_lama/api/v1/hazards.py:some_symbol \
    --waiting-label "waiting on the fix"                     # exit 3 if unmet, nothing run
scripts/smoke_walk_wsl.sh --clean-slate                      # reset DB + /uploads before walking

On this box it publishes compose postgres on 5434, not the stock 5433: the native PG cluster 16 owns 5433 and stopping it needs a sudo password. Only ADMIN_LOADER_DATABASE_URL consumes that host port — containers reach the database at postgres:5432 — so the remap is behaviourally inert. It edits docker-compose.yml in place after the pull (a dirty compose file makes git pull --ff-only refuse) and reverts it in the trap; the provenance block records the file as dirty rather than claiming a clean tree. Free 6379/9000 first if the OasisEvaluation stack is up: docker compose -p oasisevaluation stop.

The walk is not idempotent on its own — see point 2 above — so a second consecutive run against a stack the first run left up fails at _unique_match() rather than passing cleanly. --clean-slate is the fix: it runs docker compose down -v (the ONE explicitly-guarded exception to the script's "never down -v" rule — see the header comment in the script itself) before up. -v drops every named volume in docker-compose.yml, not just postgres and /uploads: postgres_data, redis_data, minio_data (all previously ingested hazard/exposure objects, not only the two datasets this walk itself ingests), hazard_uploads (/uploads), nominatim_data (the geocoding profile's OSM import, if it was ever run — reimporting has a 45-minute healthcheck start_period), prometheus_data, grafana_data and loki_data. Nothing the walk itself needs must be backed up first — bootstrap recreates the org/admin, the walk loads its own GADM boundaries per leg, and migrations replay 0001 → head from scratch (slower, but a strictly stronger check than the incremental path) — but anything else this box was used to ingest or run (MinIO objects, a completed Nominatim import, Grafana dashboards) is gone with it. The flag refuses with exit 3, before touching Docker, unless every one of its project guards holds: .env has no APP_ENV/ENVIRONMENT equal to prod or the legacy production (case-insensitive — matches src/climate_lama/config.py's normalize_app_env), DATABASE_URL/POSTGRES_HOST in .env (when set) resolve to localhost/127.0.0.1, and the checkout lives under $HOME or /mnt (a WSL DrvFs-mounted Windows path) — never both true of the actual prod host, a separate Linux box at /opt/climate-lama as root. The flag is meant to be impossible to fire against production by accident, and the guard fails closed (a missing or unreadable .env refuses rather than proceeding). The two other down calls in the script — the EXIT-trap teardown and the pre-bind-test teardown — are unaffected and stay -v-free, so a green non---clean-slate walk still leaves its database up for inspection.

Two traps this script exists to make unmissable:

  • A cached migrations image migrates to its own head and exits 0 — the service COPYs the repo in, and compose will not rebuild an image that already exists, so the schema silently pins at whatever revision the image was built with. The script rebuilds worker migrations bootstrap and then asserts select version_num from alembic_version.
  • Two checkouts of one repo on one machine (the Windows one and the WSL one) — a grep in the wrong tree produces phantom findings about what did or did not land. The script always works from git rev-parse --show-toplevel and resolves precondition symbols against origin/main, never the working tree.

Note that --require-merged <issue> on its own is a weak gate: a docs PR that merely mentions the issue matches it. Append a title regex ('1234:^fix\(') or pair it with --require-symbol, which is the authoritative check.

Operator scripts — the one-shot repair/migration tools

These do not run on a schedule. Each is idempotent, defaults to a dry run, and prints the number of rows it matched. As of the release that added COPY scripts/ ./scripts/ to docker/core.Dockerfile, all of them exist inside the api image — before that they had to be docker cp'd in by hand, and the copy did not survive a container recreate.

Script What it repairs
scripts/backfill_dataset_identity.py Fills provider / upstream_version / native_resolution on hazard_datasets. Nothing emitted these before, so every row read blank — which also left core/dataset_precedence.py's heaviest term (resolution, 60 pts) scoring 0 for everything, making the finest-wins rule inert.
scripts/backfill_dataset_footprint.py Stamps hazard_datasets.footprint (ADR-045 topic 2 / ADR-057, issue #596) for datasets ingested before that column existed. Derives the envelope from the dataset's hazard_centroids extent, falling back to the stored intensity NPZ's grid geometry only when a dataset has no centroid rows at all. Until this runs for a given dataset, the "which datasets cover this point/bbox" coverage rule silently skips it.
scripts/backfill_source_manifests.py Writes hazards/{dataset_id}/source_manifest.json for pre-ADR-066 datasets. Relocates the real captured record from staging/{ingest_job_id}/manifest.json where it survives (keeping the genuine checksums) and only reconstructs — explicitly marked capture_mode: "reconstructed" — where it does not.
scripts/migrate_org_data.py Moves one org's data to another. Data moves; identity, credentials, history and org policy stay. 
# always dry-run first and read the matched count
./dc exec -T api python scripts/backfill_dataset_identity.py
./dc exec -T api python scripts/backfill_dataset_identity.py --apply

The failure mode to watch for is a false clean. Every org-scoped table is under RLS, and a connection without app.current_org_id bound sees zero rows — so a script can cheerfully report "nothing to do" against a full database. Each of these exits non-zero and names RLS when it matches zero rows, rather than reporting success; if you see MATCHED 0, treat it as a connection/permissions problem, not as "already done".

backfill_dataset_footprint.py specifically

Run it once against a database whose hazard_datasets table predates migration 0075_hazard_dataset_footprint.py (i.e. any environment that had data before that migration shipped) — check first with:

SELECT count(*) FROM hazard_datasets WHERE footprint IS NULL AND status = 'active';

A non-zero count means there is work to do; 0 means either it already ran or the environment only ever had post-#596 ingests. Then:

./dc exec -T api python scripts/backfill_dataset_footprint.py            # dry-run
./dc exec -T api python scripts/backfill_dataset_footprint.py --apply

Expected dry-run output shape — the two RLS-guard counts first, then one line per candidate, then a summary:

INFO scripts.backfill_dataset_footprint: SCANNED 42 hazard_datasets row(s) across 3 org(s); 5 carry a NULL footprint
INFO scripts.backfill_dataset_footprint: WOULD SET [centroids] JRC river flood — Greece (id=3fa9..., haz_type=river_flood, org_id=8b1c...) -> footprint=(19.30000, 34.70000, 28.30000, 41.80000)
INFO scripts.backfill_dataset_footprint: WOULD SET [npz_grid] legacy-windstorm-footprint (id=7c21..., haz_type=storm_europe, org_id=8b1c...) -> footprint=(-10.00000, 35.00000, 30.00000, 60.00000)
INFO scripts.backfill_dataset_footprint: 0 row(s) would be stamped, 0 unresolved, 5 seen with a NULL footprint.
INFO scripts.backfill_dataset_footprint: dry-run: nothing was written. Re-run with --apply to write.

If the first line ever reads SCANNED 0 hazard_datasets row(s), the run exits non-zero and names the RLS false-clean explicitly — that is a connection/permissions problem (app.current_org_id never bound), never a legitimately empty catalog, unless you pass --allow-empty.

Home's coverage grid depends on the footprint (issue #1318)

GET /v1/catalog/coverage — the grid behind Home's Start-here card and the per-country peril tiles — derives each hazard dataset's country from COALESCE(footprint, bbox). hazard_datasets.bbox is the upstream extent and no ingest path writes it, so in practice the footprint is the only extent an ingested hazard dataset has. Such a row with footprint IS NULL therefore derives no country and is reported under undetermined: correct behaviour (NULL means "not derivable", never "in no country"), but it reads on Home as "No hazard datasets held here yet." under a country that in fact holds dozens.

So on any environment whose Home grid shows gaps, run the backfill and then re-read the grid:

./dc exec -T api python scripts/backfill_dataset_footprint.py            # dry-run — read the matched count
./dc exec -T api python scripts/backfill_dataset_footprint.py --apply

Verify — as a superuser role (climate_lama); the app role is under FORCE ROW LEVEL SECURITY and an unbound app.current_org_id returns zero rows, which reads as a clean database rather than as no access:

-- The backfill's candidate set: non-superseded datasets with no stamped
-- footprint. Do NOT read this as "the undetermined count" — under
-- COALESCE(footprint, bbox) a row with a usable bbox and no footprint is
-- still placeable. The undetermined number is the `(undetermined)` row of
-- the second query, and that is the one that must drop after --apply.
SELECT count(*) FROM hazard_datasets
 WHERE footprint IS NULL
   AND (status IS NULL OR lower(btrim(status)) <> 'superseded');

-- The country split the grid will report, mirroring the shipped derivation:
-- COALESCE(footprint, bbox_envelope(bbox)), largest admin-0 overlap, ties on
-- iso3, superseded excluded. Per org, because the grid is org-scoped — one
-- organization's numbers never include another's.
WITH placed AS (
    SELECT hd.org_id,
           (SELECT ab.iso3
              FROM admin_boundaries ab
             WHERE ab.admin_level = 0
               AND ST_Intersects(ab.geom, hd.extent)
             ORDER BY ST_Area(ST_Intersection(ST_Envelope(ab.geom), hd.extent)) DESC,
                      ab.iso3 ASC
             LIMIT 1) AS iso3
      FROM (
        SELECT org_id,
               COALESCE(
                 footprint,
                 CASE WHEN jsonb_typeof(bbox -> 0) = 'number'
                       AND jsonb_typeof(bbox -> 1) = 'number'
                       AND jsonb_typeof(bbox -> 2) = 'number'
                       AND jsonb_typeof(bbox -> 3) = 'number'
                      THEN ST_MakeEnvelope(
                             LEAST((bbox ->> 0)::float8, (bbox ->> 2)::float8),
                             LEAST((bbox ->> 1)::float8, (bbox ->> 3)::float8),
                             GREATEST((bbox ->> 0)::float8, (bbox ->> 2)::float8),
                             GREATEST((bbox ->> 1)::float8, (bbox ->> 3)::float8),
                             4326)
                 END
               ) AS extent
          FROM hazard_datasets
         WHERE status IS NULL OR lower(btrim(status)) <> 'superseded'
      ) hd
)
SELECT org_id, COALESCE(iso3, '(undetermined)') AS country, count(*) AS datasets
  FROM placed
 GROUP BY 1, 2
 ORDER BY 1, 3 DESC;

Then read the endpoint itself, remembering the grid is cached per organization for 60 s (COVERAGE_CACHE_TTL_SECONDS) — a request inside that window still serves the pre-backfill answer, so wait it out rather than concluding the backfill did nothing:

curl -s -H "Authorization: Bearer $TOKEN" https://<host>/v1/catalog/coverage | jq '{
  countries: [.data.countries[] | {iso3, held: ([.hazards[].dataset_count] | add)}],
  undetermined: ([.data.undetermined.hazards[].dataset_count] | add)
}'

undetermined should drop by exactly the number of rows the backfill stamped, and the countries those datasets cover should gain them — the same movement the second SQL query's (undetermined) row shows. Those two answers are held in lockstep by tests/test_db/test_hazard_repository_coverage.py, which executes the SQL above straight out of this file and asserts it equals what HazardRepository.coverage_groups returns; if you edit the queries, that test is what tells you whether the edit still describes the shipped rule. The prod run itself is owner-driven, not part of this fix — #1323 tracks it.

migrate_org_data.py specifically

Read --print-storage-remediation before you consider the move finished. Rendered reports live at reports/{org_id}/{subject_id}/{template}/ and that key is recomputed from the row's live org_id — no column stores it. After the move every existing PDF is unreachable, and because the readiness check is the manifest's presence, it fails as a silent 404 that reads as "not rendered yet", not as an error. COGs are safe by contrast: cog_path is read from the stored column, so it keeps resolving even though the acme org UUID stays embedded in the string.

Stop api and worker before --apply: it takes row locks, and briefly an ACCESS EXCLUSIVE lock while it drops and recreates the three composite FKs that include org_id (no update ordering satisfies them, so those five tables move in one transaction). It needs a superuser DSN — the RLS policies are FORCE ROW LEVEL SECURITY with no WITH CHECK, so binding the GUC is not enough and the script refuses to run without rolsuper/rolbypassrls.

Management tasks — the hand-invoked builders

None of them runs on a schedule, and all are deliberately absent from celery_app.beat_schedule. They are Celery tasks rather than scripts/ tools because they need the worker's session, MinIO client and org GUC.

Task What it does
build_reference_surfaces Builds the org-less reference_risk_* plane from a designated scenario matrix — dispatch half queues the cells, promote half writes them. The only path into the shared plane. Verified as part of the RLS cut-over in DEPLOYMENT.md ("one matrix run").
resave_run_surfaces Re-derives an existing org_risk_surfaces row's cells from the run that already produced them. Writes no new answer — it exists so a stored surface gains a cell metric it was written too early to carry.
backfill_result_cells Builds the H3 and GADM aggregates (result_h3_cells, result_admin_cells) of an impact result written before #1470, from its stored eai_exp array. Recomputes nothing — it exists so an older result's map is served as vector tiles instead of the GeoJSON fallback.

resave_run_surfaces — when a stored surface has no rollup metric

Symptom: an admin-unit or portfolio rollup returns metric_absent_from_cells for a hazard whose surfaces are plainly there, and the address-level card for a point inside the same ground bands fine.

Why: a cell's metrics blob carries the active score scheme's metric key only if the run that wrote it knew to put it there. ADR-075 (issue #932) added that at run-save time; every surface written before it carries only {eai, eai_density_km2, points}. Both rungs read cell.metrics[scheme.metric], so they see nothing.

This is recurring, not a one-off backfill. The rungs resolve the active scheme at read time (core/rollup_service.py), so re-versioning a scheme or renaming its metric darkens every stored surface until it is re-saved under the new key. Run this after any migration that seeds a new scheme version, and after any change to a scheme's metric.

Two other ways to reach the same state, for completeness: re-running the scenario also fixes it (a run-save replaces a surface's cells wholesale), and the reference plane is refreshed by re-promoting through build_reference_surfaces, which already takes a result_id.

Always dry-run first — it is the default, and it reads everything the apply would (including each dataset's intensity matrix), so the plan it prints is a real preview rather than a guess:

# dry run: one org, or every live org when org_id is omitted
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import resave_run_surfaces; \
             print(json.dumps(resave_run_surfaces.apply(kwargs={'org_id': '<org-uuid>'}).get(), indent=2))"

# apply
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import resave_run_surfaces; \
             print(json.dumps(resave_run_surfaces.apply(kwargs={'org_id': '<org-uuid>', 'dry_run': False}).get(), indent=2))"

.apply() runs the task inline in that container rather than queueing it, which is what you want for an operator action you are watching.

The report carries dry_run, surfaces, counts and one entries row per surface. Every count key is always present, so two runs are directly comparable. What each outcome means:

Outcome Meaning Action
resaved Re-derived and written (or, under dry_run, would be — the row's applied flag tells them apart). none
skipped_legacy dataset_id IS NULL: written before #589, so it names no hazard dataset and there is no identity to join a stored result to. Re-run the scenario to mint a dataset-keyed surface.
skipped_no_result Candidates were searched and none of them would have written this surface's key. Re-run the scenario.
skipped_orphaned_dataset The hazard dataset the surface names is gone, so no run could be searched for at all. Kept separate from skipped_no_result precisely because re-running is not available as a remedy. Nothing to re-run. Either the surface is stale and can be deleted, or the dataset was removed in error.
skipped_drift The exposure set changed since the run, so the stored array can no longer be attributed to named exposures (#370). Refused rather than filed under wrong coordinates. Re-run the scenario against the current exposures.
no_design_rp The dataset publishes no event at the peril's design return period (or has no stored intensity matrix). Reported as unenrichable; the surface is left untouched. Nothing to do here — the peril's design RP or its dataset's event table is the thing to fix.
no_metric_key No seeded score scheme claims this peril at this intensity unit, so there is no key to file a metric under. Seed the scheme.
failed That surface raised; the rest of the pass continued. Read the reason and the worker log.

Two failure modes worth knowing:

  • A false clean. org_risk_surfaces is under RLS, so a connection with no app.current_org_id bound sees zero rows. The task enumerates organizations from organizations (which is not org-scoped) and binds the GUC per org for exactly this reason, and logs a warning when a whole pass matches no surfaces at all. If you see that warning against a database you know is not empty, treat it as a permissions problem, not as "already done".
  • Applying with run-saves switched off. resave_run_surfaces refuses to run with dry_run: False while SURFACE_RUN_SAVE_ENABLED is false, because the writer would decline every write and the pass would report a clean it did not achieve.

backfill_result_cells — when an older result's map has no tiles

Symptom: a result's GET /v1/results/{id} carries no metadata.cells block (or one whose status is "failed"), so its map is drawn from the GeoJSON fallback instead of vector tiles.

Why: a result's H3 and GADM aggregates are written by compute_impact when the result is stored (issue #1470, ADR-083). Every result written before that has none, and nothing else creates them. This task builds them from the stored per-asset eai_exp array, paired with the exposure rows load_aligned_exposures re-derives in compute order (the #370 contract), and writes through the same write_result_cells the compute hook uses — the impact is never recomputed, and impact_results is touched only by the metadata.cells stamp. A result already stamped "built" is skipped, so a second pass is a no-op; one stamped "failed" is retried.

Always dry-run first — it is the default, and it runs every read the apply would (the array download, the compute-order join, the H3 build and the GADM sums), so the counts it prints are the ones the apply will store:

# dry run: one org, or every live org when org_id is omitted
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import backfill_result_cells; \
             print(json.dumps(backfill_result_cells.apply(kwargs={'org_id': '<org-uuid>'}).get(), indent=2))"

# apply
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import backfill_result_cells; \
             print(json.dumps(backfill_result_cells.apply(kwargs={'org_id': '<org-uuid>', 'dry_run': False}).get(), indent=2))"

The report carries dry_run, results, counts and one entries row per result (with points, h3_cells, admin_cells and applied). Every count key is always present. Each result runs in its own transaction, so stopping a long pass keeps the results already built. What each outcome means:

Outcome Meaning Action
built Aggregates built and stamped "built" (or, under dry_run, would be — the row's applied flag tells them apart). none
skipped_built metadata.cells.status is already "built". Nothing was read or written. none
skipped_no_array The result has no eai_exp_path, so there is no per-asset array to aggregate. Re-run the scenario if the map matters.
skipped_drift The exposure set changed since the run, so the stored array can no longer be attributed to named exposures (#370). Refused rather than filed under wrong coordinates. Re-run the scenario against the current exposures.
failed That result raised; its transaction was rolled back and the rest of the pass continued. Read the reason and the worker log.

The task walks results under RLS with the org GUC bound per org, so read any SELECT count(*) FROM result_h3_cells check as the owner role (psql -U "$POSTGRES_USER") — through the app role the table reads as empty.

Recovering a mis-rejected dataset review (re-arm)

Symptom: a dataset that should be in the review queue is not, and GET /v1/admin/pending-ingests?status=rejected shows it as rejected (or ingest_failed). A single mis-click does this, and so does a bulk sweep — 22 rows were rejected on prod in one scripted run on 2026-08-11, 21 of them inside 0.22 s.

Why it does not fix itself: the weekly poll records every item it sees in dataset_source_catalog, whether or not anyone approves it. A decided row is therefore skipped by every later poll until the upstream content changes. That is deliberate — it is what stops a declined item nagging admins every week — but it means the decision is absorbing. Re-arm is the supported way back: it deletes that item's last-seen marker so a poll sees the item as new again, and it dispatches a scoped poll (that one source, those items, no admin notification fan-out) so the item is back in the queue in seconds rather than at 05:00 next Monday.

Re-arm never touches the decided row — it stays as the audit record — and it never triggers an ingest. The item comes back through review, not around it. Only rejected and ingest_failed rows are re-armable; anything else is a 422.

Every call below needs a platform admin token (is_platform_admin = true), not an org admin. Set API=https://<host> and TOKEN=<jwt> first.

1. Find the rows

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=rejected&limit=500" \
  | jq -r '.data.items[] | [.id, .source_id, .item_id, .reviewed_at] | @tsv'

Narrow to the sweep you are undoing before you re-arm anything — reviewed_at is the field that separates one bad batch from years of deliberate decisions:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=rejected&limit=500" \
  | jq '[.data.items[] | select(.reviewed_at >= "2026-08-11T00:00:00")
         | select(.source_id == "worldpop") | .id]' > /tmp/rearm-ids.json
jq 'length' /tmp/rearm-ids.json   # sanity-check the count before sending it

2. Re-arm

One row:

curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests/<id>/re-arm" | jq .data

The whole batch in one call (max 500 ids):

jq -c '{ingest_ids: .}' /tmp/rearm-ids.json > /tmp/rearm-body.json
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" --data @/tmp/rearm-body.json \
  "$API/v1/admin/pending-ingests/re-arm" | jq '{counts, polls_dispatched, failed}'

The bulk call is 200 even when every id fails — read counts, not the status code. Ids that do not exist, or rows that are not in a dead-end status, come back in failed with a code of not_found / not_rearmable while every other id is still re-armed.

Read three fields on each re-armed row:

Field Meaning if it is not what you expect
last_seen_cleared false = there was no marker, so the item was already pollable. Harmless; not a failure.
source_registered false = the item's source is no longer registered in this build. The marker was cleared but no poll can act on it — the item stays invisible until the source is registered again and a poll runs.
polls_dispatched [] = nothing was polled (this is always the case when source_registered is false). Otherwise one entry per source, with the item count.

3. Confirm it came back

The scoped poll runs on the compute queue and takes as long as the source's manifest fetch — seconds for JRC/WorldPop, longer for a queued Copernicus retrieval. Then:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=awaiting_review&limit=500" \
  | jq -r '.data.items[] | [.source_id, .item_id] | @tsv' | sort

If the item is still missing after the poll has had time to run:

# Did the scoped poll actually see the item?
docker logs climate_lama_worker 2>&1 | grep poll_dataset_source | tail -20
  • seen=0 — the source no longer publishes that item_id. Nothing is broken; the upstream dataset is gone or was renamed, and re-arm cannot conjure it back.
  • queued=0 with seen=1 — an awaiting_review row already exists for that item at the same content key (the poll's re-queue guard). Check the queue again; it is already there.
  • error non-null — the source's manifest fetch failed. Triage it like any other source failure, then re-run the poll: ./dc exec -T worker python -c "from climate_lama.worker.dataset_polling import poll_dataset_source; print(poll_dataset_source.delay('<source_id>', ['<item_id>']).id)"

Every re-arm writes a pending_ingest.rearmed audit entry recording who did it, last_seen_cleared, source_registered, and whether it was part of a batch — so a recovery is as auditable as the rejection it undoes.

Multi-engine agreement verification

scripts/verify_engine_agreement.py runs one impact scenario through every available compute engine and prints a divergence table with a gating exit code — see the script's module docstring for the full contract. Worked invocation:

uv run python scripts/verify_engine_agreement.py \
    --base-url http://localhost:8000 --hazard <hazard_dataset_id> \
    [--exposure <exposure_dataset_id>] [--impf <impact_function_id>] \
    [--max-ead-divergence-pct 1.0] [--allow-single-engine] \
    [--require-engine climada --require-engine oasis]

The three-engine proof (issue #1481) brings up both sidecars (docker compose --profile climada --profile oasis up -d), ingests a return-period river-flood hazard and an exposure through the real async APIs, and runs the command with --require-engine climada --require-engine oasis. It passes only with three engine rows, each with its curve_basis column (OEP on all three). A required engine that is excluded, failed or absent exits 1, so a stack without the Oasis sidecar cannot pass as a two-engine run.

Exit codes: 0 — every --require-engine engine and every compared engine completed and stayed under --max-ead-divergence-pct (default 1.0, the house parity tolerance); 2 — only one engine was ever available (the rest registered but not deployed here, per excluded_engines) and it completed cleanly — pass --allow-single-engine to accept this as 0 on a stack that has not brought the CLIMADA sidecar up; 1 — anything else that is not a clean pass: a failed engine, a divergence at/over threshold, an unreachable API, or a timed-out poll. --allow-single-engine only ever downgrades the single-engine-availability case (2 → 0); it never masks a real 1.

CLIMADA engine unavailable / comparison shows failed_engines: ["climada"]

Two different situations both read as "CLIMADA isn't in the answer," and they need different triage — src/climate_lama/worker/models/climada_adapter.py's module docstring is the canonical source for the split:

  • Config-off (not a failure) — CLIMADA_SIDECAR_URL is unset (or present but empty) on this deployment. The registry's availability probe (_climada_sidecar_configured in worker/models/registry.py, backed by Settings.climada_sidecar_url) reports the engine unavailable before any job is queued: a comparison that omits engines lists it in the response's excluded_engines with reason: "unavailable" and proceeds with whatever else is available, while naming climada explicitly fails fast with 400 E_ENGINE_UNAVAILABLE. No job ever runs and none ever fails — this is the expected shape on any stack that hasn't opted into the climada compose profile. See DEPLOYMENT.md → "Enabling the CLIMADA engine" to turn it on.
  • Sidecar down (a real failure) — the URL is configured but the container is unreachable, times out, or errors. The engine was queued like any other, so its job fails and climada lands in divergence.failed_engines — not excluded_engines — while the rest of the comparison still reports normally.

Triage the second case:

  1. In-network health probe. The sidecar is never exposed over HTTP — only the worker reaches it, in-network — so probe from inside the worker container: 
    docker compose $P exec -T worker curl -fsS http://climada-sidecar:8100/health
    ($P is the Release → Production shorthand with --profile climada folded in — see DEPLOYMENT.md.) A healthy sidecar answers {"status": "ok", ...}; scripts/verify-deploy.sh runs the same probe (plus a --max-time bound and a status field check) as its own step 6 (the CLIMADA sidecar check moved from step 5 to step 6 when the UI bundle identity check, issue #1315, was inserted ahead of it).
  2. Sidecar logs: 
    docker compose $P logs climada-sidecar
  3. Restart without rolling api/worker. The profile makes it an ordinary compose service, so a scoped restart is just that: 
    docker compose $P restart climada-sidecar
    docker compose $P ps before and after confirms only climada-sidecar recreated — api/worker/beat are untouched.
  4. Re-run the worked invocation above against the same hazard/exposure pair to confirm the comparison completes clean.

Country data run on prod — the pack invocation that works

The Cyprus data run (#1048, executed 2026-08-31/09-02) found every checklist command for a country pack run wrong as written. None of this was in this runbook or in DEPLOYMENT.md before. Eight things a second-country run needs to know before it starts (item 8 was added by the Phase 13 replay run, #1146, 2026-09-03):

  1. Every pack needs the ingest subcommand. add_country_argument is attached to the ingest subparser under a required=True group, so e.g. coastal_flood.py --country CYP (no subcommand) exits 2.
  2. --org-slug <slug> is mandatory (scripts/org_target.py — "no target organization"; there is deliberately no default).
  3. --data-dir and --worker-data-dir must both be passed explicitly, as /data/<pack>. The default is /app/data/<pack> inside the container, which is not the ./data:/data mount — leaving either at its default reproduces the #748 class, where clipped files never reach the worker.
  4. heatwave_packs.py ingest takes a required positional item_id (e.g. heatwave-ecde-europe) — a checklist that omits it silently no-ops at argparse. The CDS heatwave collections declare no server-side area, so a heatwave pack has no per-country clip: a second country's --country CYP run against the same item_id correctly reports "already in catalog, skipping" — that message is the pass condition for a second country, not a failure.
  5. LitPop's convert runs off-box, on the operator workstation, because the api image has no h5py: 
    uv run --with h5py python scripts/packs/litpop_exposure.py convert \
    --source climada-api --country-iso3 CYP \
    --data-dir /c/tmp/litpop-cyp --out /c/tmp/litpop-cyp/LitPop_150arcsec_CYP.csv
    --country-iso3 is independent of --country and defaults to GRC — omitting it on a non-Greek run silently fetches the Greek release. Verify the right release downloaded by checking region_id on every row (ISO 3166-1 numeric: Cyprus 196, Greece 300), not just the row count. The ingest half runs in-container as usual (no h5py needed on that path — it only imports lazily inside convert).
  6. Working form of the standalone-pack invocation, combining points 1-3: 
    ./dc exec -T api python scripts/packs/<pack>.py ingest \
    --org-slug <slug> --country CYP --data-dir /data/<pack> --worker-data-dir /data/<pack>
    Append `