Troubleshooting Runbook — `scripts/demo.py` and the local dev stack¶

This runbook is for triaging a failing run of scripts/demo.py — the end-to-end demo/smoke script that exercises the full vertical slice (ingest hazard → upload exposure → assign centroids → compute impact → read back results) against a live docker compose up stack (docker-compose.yml).

Every command and claim below is grounded in a committed file — cited inline — so this stays trustworthy even though it can't be exercised against a running stack in this environment (no Docker/PostGIS here). Where something is a genuinely external convention (e.g. how Celery keys its Redis results) rather than something this repo defines, it's called out as such.

For production operations (release flow, backups, the Hetzner host), see DEPLOYMENT.md instead — this doc is about the local dev stack (docker-compose.yml), which is what scripts/demo.py targets by default (--base-url http://localhost:8000).

Before you start — a checklist that catches most first-run failures¶

.env exists. cp .env.example .env (README.md "First-time setup"). The migrations, api, worker, and beat services all declare env_file: .env (docker-compose.yml) — docker compose up refuses to start them without it.
The stack is up: docker compose up (or docker compose --profile fullstack up for the full UI+API slice, per CLAUDE.md's "Demoable MVP Slice").
scripts/demo.py needs requests importable — it's a module-level import (scripts/demo.py) with no other project dependency, so any Python 3 environment with requests installed can run it (e.g. uv pip install -e ".[dev,worker]", the install path .github/workflows/sdk-smoke.yml uses in CI).
The default hazard-ingest payload needs real data on disk that a clean clone doesn't have. INGEST_PAYLOAD in scripts/demo.py reads six GeoTIFFs from /data/river_flood_hazard_maps_for_europe/... inside the container. That path is served by the ./data:/data:ro bind mount on both the api and worker services (docker-compose.yml), and data/ is gitignored (.gitignore), so a fresh clone has nothing there — hazard ingest fails at the file-read step. Two grounded ways to fix it:
Stage the real JRC pack yourself under ./data/river_flood_hazard_maps_for_europe/ — the source is named in INGEST_PAYLOAD["source_url"]: https://data.jrc.ec.europa.eu/collection/id-0054 (scripts/demo.py).
Or use the tiny tracked fixture instead — this is exactly what .github/workflows/sdk-smoke.yml's "Stage demo hazard raster for the worker" / "Seed demo data" steps do:
```
mkdir -p data
cp demo_data/river_flood.tif data/river_flood.tif
SDK_SMOKE_HAZARD_TIF=/data/river_flood.tif python scripts/demo.py
```
Setting SDK_SMOKE_HAZARD_TIF makes _resolve_ingest_payload() in scripts/demo.py switch to a single-file ingest against that path instead of the six-file JRC pack.
Auth-gated steps best-effort-skip on 401 unless you configure credentials. Hazard ingest, exposure upload, scenario save, and compute all require Role.ANALYST (scripts/demo.py module docstring, _build_session()). With .env.example's defaults (CL_SEED_DEMO=true, CL_DEMO_USER_PASSWORD=demo1234), the seeded demo@example.com user is an analyst in org acme-demo (scripts/seed_demo.py: DEMO_ORG_SLUG = "acme-demo", DEMO_USER_EMAIL = "demo@example.com"; config.py's comment on cl_demo_user_password documents that setting it grants the analyst role), so:
```
SDK_SMOKE_EMAIL=demo@example.com SDK_SMOKE_PASSWORD=demo1234 SDK_SMOKE_ORG_SLUG=acme-demo \
  python scripts/demo.py
```
exercises the fully-authenticated path end to end.

Quick triage — where to look first¶

GET /health (liveness — process is up) then GET /health/ready (readiness — probes Postgres, Redis, and MinIO individually and reports per-check detail) — src/climate_lama/main.py health() / health_ready(). scripts/demo.py's very first call is GET /health/ready (step 1), so a dependency outage fails the demo immediately with the specific check's detail message rather than a generic connection error later on.
docker compose ps — which containers aren't running/healthy.
docker compose logs <service> --tail=100 for whichever check health/ready flagged.

Postgres¶

Symptoms: GET /health/ready reports "postgres": {"status": "error", "detail": ...} (main.py); scripts/demo.py fails at step 1 before anything else runs.

First three checks:

docker compose ps postgres — healthcheck is pg_isready -U climate_lama (docker-compose.yml).
docker compose logs postgres --tail=100 (container name climate_lama_postgres, docker-compose.yml).
docker compose logs migrations — the one-shot migrations service runs alembic upgrade head and only starts once Postgres is healthy (depends_on: postgres: condition: service_healthy, docker-compose.yml); api, worker, and beat all wait on migrations: condition: service_completed_successfully, so a wedged or failed migration blocks every other service, not just Postgres.

Gotcha — port mismatch between alembic.ini and docker-compose.yml: the compose Postgres is published on host port 5433, not 5432 ("5433:5432", docker-compose.yml), but migrations/alembic.ini's own default sqlalchemy.url still points at localhost:5432. Running alembic upgrade head / alembic current directly from the host (outside the migrations container — e.g. the Quickstart's uv run alembic upgrade head, docs/quickstart/index.md) needs DATABASE_SYNC_URL exported first, or it fails closed with a connection error:

export DATABASE_SYNC_URL="postgresql+psycopg2://climate_lama:password@localhost:5433/climate_lama"
uv run alembic upgrade head

(migrations/env.py reads DATABASE_SYNC_URL from the environment and overrides alembic.ini's default when it's set.)

Redis¶

Symptoms: GET /health/ready reports "redis": {"status": "error", ...} (main.py); scripts/demo.py jobs never leave pending (poll_job() keeps printing ... {label}: pending — the worker can't receive tasks without the broker).

First three checks:

docker compose ps redis — healthcheck is redis-cli ping (docker-compose.yml).
docker compose exec redis redis-cli ping.
Broker and result backend are two different logical Redis databases on the same instance — broker on db 0, results on db 1 (CELERY_BROKER_URL=redis://redis:6379/0, CELERY_RESULT_BACKEND=redis://redis:6379/1, docker-compose.yml; wired the same way in src/climate_lama/worker/celery_app.py via Celery(..., broker=settings.celery_broker_url, backend=settings.celery_result_backend)). If you're inspecting keys by hand, pick the right db: docker compose exec redis redis-cli -n 0 keys '*' for queued tasks, -n 1 for stored results.

MinIO¶

Symptoms: GET /health/ready reports "minio": {"status": "error", "detail": "bucket 'climate-lama' does not exist"} — MinIO itself can be reachable while the bucket is still missing, because health_ready() explicitly checks bucket_exists, not just connectivity (main.py). Hazard-ingest or exposure-upload jobs fail while staging to object storage.

First three checks:

docker compose ps minio minio-init — minio's healthcheck is mc ready local; minio-init is a one-shot job that creates the bucket and only prints Bucket ready. on success (docker-compose.yml, minio-init entrypoint: mc alias set local http://minio:9000 minioadmin minioadmin && mc mb --ignore-existing local/climate-lama).
docker compose logs minio-init — confirms whether the bucket-creation step actually ran and succeeded.
Inspect the bucket directly, mirroring minio-init's own alias pattern:
```
docker compose exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
docker compose exec minio mc ls local/climate-lama
```
(credentials are .env.example's dev defaults: MINIO_ACCESS_KEY=minioadmin / MINIO_SECRET_KEY=minioadmin.) Or use the browser console at http://localhost:9001 (docker-compose.yml port mapping 9001:9001; also listed in README.md's service table).

API¶

Symptoms: any scripts/demo.py step raising via resp.raise_for_status(); 401s on auth-gated steps when no token/login env vars are set (steps 2, 4, 6, 7b, 8 — see scripts/demo.py's module docstring "Authentication" section).

First three checks:

GET /health then GET /health/ready (main.py) — establishes whether the API process itself is up before treating a failure as request-specific.
GET /v1/info — reports version, environment, and git_sha (main.py info()), so you know which build actually answered. Useful when comparing a locally docker compose --build-ed API against a pulled ghcr.io/cortomaltese3/climate-lama:${CLIMATE_LAMA_TAG} image (docker-compose.yml).
docker compose logs api --tail=100 (container climate_lama_api). The dev api service mounts ./src and runs with --reload (command: uvicorn climate_lama.main:app --host 0.0.0.0 --port 8000 --reload, docker-compose.yml), so a stack trace from a local edit shows up here immediately.

Reading the error envelope: every API error follows one shape — {"error": {code, severity, message_en, details}, "meta": {request_id, timestamp}} (ADR-028; exercised directly by scripts/demo.py step 12). Match error.code against the Code enum in src/climate_lama/core/errors.py, e.g.:

E_ENGINE_UNKNOWN (422) / E_ENGINE_UNAVAILABLE (400) — the optional X-Engine header on POST /v1/compute/impact picked an unregistered engine name, or a registered-but-not-deployed one (src/climate_lama/api/v1/compute.py).
E_INGEST_CHORD_TIMEOUT / E_INGEST_WEDGED — hazard ingest stalled past its time budget; see the Worker section below.
E_GEOCODE_UNAVAILABLE (503) — step 16's geocoding probe when the opt-in geocoding profile (self-hosted Nominatim) isn't running (docker-compose.yml, nominatim service).

Worker¶

Symptoms: ingest/compute jobs stuck at pending/running past scripts/demo.py's --timeout (default 300s) — poll_job() raises TimeoutError: Job ... did not complete within 300s. GET /v1/hazards/ingest-jobs/{id} (scripts/demo.py step 2a) shows status stuck at writing/aggregating with progress_pct not moving.

First three checks:

docker compose ps worker — note: the dev docker-compose.yml's worker service defines no healthcheck (unlike docker-compose.prod.yml's worker, which probes celery -A climate_lama.worker.celery_app inspect ping -d celery@$$HOSTNAME), so docker compose ps can show worker as merely "running" even if Celery itself failed to initialize on boot — don't trust the state column alone, check logs.
docker compose logs worker --tail=100 — on boot, Celery prints its resolved task list; compare against the include=[...] list in celery_app.py (e.g. climate_lama.worker.tasks, climate_lama.worker.ingest.stage_source, ...). A task missing from that boot-time list means its module failed to import.
docker compose exec worker celery -A climate_lama.worker.celery_app inspect active (also try inspect registered, inspect ping) — the same -A climate_lama.worker.celery_app invocation the worker/beat services themselves use (command: celery -A climate_lama.worker.celery_app worker --loglevel=info --queues=compute, docker-compose.yml); confirms the worker process is actually consuming the compute queue.

If a job looks permanently wedged rather than merely slow: the chord-timeout and chunk-wedge watchdogs (sweep-wedged-ingest-chords every 10 minutes, sweep-wedged-writing-ingest-jobs every ingest_wedged_writing_sweep_minutes minutes, default 10 — both in celery_app.py's beat_schedule) only run if the separate beat service is up: "Without this service those tasks never run anywhere" (docker-compose.yml, comment on the beat service). docker compose ps beat is worth checking any time an ingest job looks stuck for good — with beat down, a wedged job never gets flipped to failed and just sits there past ingest_chord_timeout_seconds (default 6h) or ingest_wedged_writing_ttl_seconds (default 45m) (config.py).

Reading a job's state without the API¶

Coarse: GET /v1/jobs/{job_id} (src/climate_lama/api/v1/jobs.py) — status (pending/running/completed/failed), celery_task_id, result_id, error_message.
Fine-grained (hazard ingest only): GET /v1/hazards/ingest-jobs/{ingest_job_id} (src/climate_lama/api/v1/hazards.py) — status walks pending → running → chunking → writing → aggregating → committed → succeeded (terminal states failed/cancelled), per IngestJobStatus (src/climate_lama/models/enums.py), plus progress_pct, chunks_total/chunks_done, error_code, and paginated logs (?logs_offset=N).
Directly in Redis (last resort — only when even the API is unreachable): the result backend lives on db 1 (CELERY_RESULT_BACKEND=redis://redis:6379/1, docker-compose.yml). Celery's Redis backend keys results as celery-task-meta-<task_id> — this is Celery's own convention, not something this repo defines, so treat the exact key format as unverified against a specific Celery version rather than a repo guarantee:
```
docker compose exec redis redis-cli -n 1 get "celery-task-meta-<celery_task_id>"
```
Get <celery_task_id> from GET /v1/jobs/{job_id}'s celery_task_id field above. The DB-backed Job / IngestJob rows via the API are the supported way to check status.

A job stuck in `running` that never resolves¶

select id, status, job_type, created_at, started_at, error_message
from jobs where status in ('pending', 'running') order by created_at;

A row here that is hours (or days) old with a NULL error_message is stranded, not slow: its worker was killed mid-task, its broker message was lost, or a failure path closed the ingest_jobs row without closing this one. Three sweeps exist, in increasing order of how long they wait (src/climate_lama/worker/celery_app.py registers all three in beat_schedule):

Sweep	Watches	Fires after	Closes the `jobs` row?
`sweep_wedged_writing_ingest_jobs`	chunk liveness of a `writing` ingest	`INGEST_WEDGED_WRITING_TTL_SECONDS` (45m)	yes
`sweep_wedged_ingest_chords`	any non-terminal `ingest_jobs` row	`INGEST_CHORD_TIMEOUT_SECONDS` (6h)	yes
`sweep_stale_jobs`	the `jobs` row itself, any job type	`JOB_STALE_TTL_SECONDS` (24h, floored at the chord timeout)	yes

The last one (src/climate_lama/worker/job_watchdog.py) is the backstop: it is the only one that covers a job with no ingest_jobs row at all (POST /v1/hazards/{id}/assign-centroids, impact_calc, report_render). Check Celery beat is actually running before concluding a row is un-reapable — with no beat process, none of the three ever fire.

To inspect or close stranded rows now, without waiting for a tick:

python scripts/reap_stale_jobs.py            # dry-run (default) — lists candidates, writes nothing
python scripts/reap_stale_jobs.py --apply    # flips them to 'failed' with the watchdog's message

Both are idempotent and never touch a job inside the TTL. Reaping only closes the book-keeping row — it does not roll back partial work; re-submit the request to retry.

Retrying a step without rerunning the whole script¶

scripts/demo.py has no per-step retry/resume flag — the only alternate entry point is --seed-only, which runs steps 1–7 (health check, hazard ingest, exposure upload, centroid assignment, impact-function seed) and prints one SEED_READY hazard_id=... exposure_id=... impact_function_id=... line (run()'s seed_only branch, scripts/demo.py), matching what .github/workflows/sdk-smoke.yml's "Seed demo data" step parses. There is no documented resume-from-step-N mechanism beyond that — inventing one here would be worse than not documenting it.

Two grounded workarounds instead:

Capture the IDs from a --seed-only run, then call the specific downstream endpoint yourself — e.g. re-POST /v1/compute/impact with the captured hazard_dataset_id / impact_function_id (src/climate_lama/api/v1/compute.py) — instead of rerunning the whole script.
Be aware reruns are not idempotent for hazard ingest: POST /v1/hazards/ingest (src/climate_lama/api/v1/hazards.py) creates a new IngestJob and hazard-dataset row on every call, with no dedup-by-name check. Rerunning scripts/demo.py against an already-seeded stack accumulates additional "JRC River Flood Greece 1km" datasets rather than reusing the earlier one.

Reference — dev stack ports and container names (`docker-compose.yml`)¶

Service	Container name	Host port(s)	Notes
postgres	`climate_lama_postgres`	5433 → 5432	PostGIS 16; not 5432 on the host
redis	`climate_lama_redis`	6379	broker on db 0, result backend on db 1
minio	`climate_lama_minio`	9000 (S3 API), 9001 (console)
migrations	`climate_lama_migrations`	—	one-shot `alembic upgrade head`
api	`climate_lama_api`	8000	`--reload`, mounts `./src` read-write
worker	`climate_lama_worker`	—	`--queues=compute`; no healthcheck in dev
beat	`climate_lama_beat`	—	drives the watchdog / nightly-maintenance schedule
titiler	`climate_lama_titiler`	7800 → 80	COG raster tiles

Operator scripts — the one-shot repair/migration tools¶

These do not run on a schedule. Each is idempotent, defaults to a dry run, and prints the number of rows it matched. As of the release that added COPY scripts/ ./scripts/ to docker/core.Dockerfile, all of them exist inside the api image — before that they had to be docker cp'd in by hand, and the copy did not survive a container recreate.

Script	What it repairs
`scripts/backfill_dataset_identity.py`	Fills `provider` / `upstream_version` / `native_resolution` on `hazard_datasets`. Nothing emitted these before, so every row read blank — which also left `core/dataset_precedence.py`'s heaviest term (resolution, 60 pts) scoring 0 for everything, making the finest-wins rule inert.
`scripts/backfill_source_manifests.py`	Writes `hazards/{dataset_id}/source_manifest.json` for pre-ADR-066 datasets. Relocates the real captured record from `staging/{ingest_job_id}/manifest.json` where it survives (keeping the genuine checksums) and only reconstructs — explicitly marked `capture_mode: "reconstructed"` — where it does not.
`scripts/migrate_org_data.py`	Moves one org's data to another. Data moves; identity, credentials, history and org policy stay.

# always dry-run first and read the matched count
./dc exec -T api python scripts/backfill_dataset_identity.py
./dc exec -T api python scripts/backfill_dataset_identity.py --apply

The failure mode to watch for is a false clean. Every org-scoped table is under RLS, and a connection without app.current_org_id bound sees zero rows — so a script can cheerfully report "nothing to do" against a full database. Each of these exits non-zero and names RLS when it matches zero rows, rather than reporting success; if you see MATCHED 0, treat it as a connection/permissions problem, not as "already done".

`migrate_org_data.py` specifically¶

Read --print-storage-remediation before you consider the move finished. Rendered reports live at reports/{org_id}/{subject_id}/{template}/ and that key is recomputed from the row's live org_id — no column stores it. After the move every existing PDF is unreachable, and because the readiness check is the manifest's presence, it fails as a silent 404 that reads as "not rendered yet", not as an error. COGs are safe by contrast: cog_path is read from the stored column, so it keeps resolving even though the acme org UUID stays embedded in the string.

Stop api and worker before --apply: it takes row locks, and briefly an ACCESS EXCLUSIVE lock while it drops and recreates the three composite FKs that include org_id (no update ordering satisfies them, so those five tables move in one transaction). It needs a superuser DSN — the RLS policies are FORCE ROW LEVEL SECURITY with no WITH CHECK, so binding the GUC is not enough and the script refuses to run without rolsuper/rolbypassrls.

Management tasks — the answer plane's two hand-invoked builders¶

Neither runs on a schedule, and both are deliberately absent from celery_app.beat_schedule. They are Celery tasks rather than scripts/ tools because they need the worker's session, MinIO client and org GUC.

Task	What it does
`build_reference_surfaces`	Builds the org-less `reference_risk_*` plane from a designated scenario matrix — dispatch half queues the cells, promote half writes them. The only path into the shared plane. Verified as part of the RLS cut-over in DEPLOYMENT.md ("one matrix run").
`resave_run_surfaces`	Re-derives an existing `org_risk_surfaces` row's cells from the run that already produced them. Writes no new answer — it exists so a stored surface gains a cell metric it was written too early to carry.

`resave_run_surfaces` — when a stored surface has no rollup metric¶

Symptom: an admin-unit or portfolio rollup returns metric_absent_from_cells for a hazard whose surfaces are plainly there, and the address-level card for a point inside the same ground bands fine.

Why: a cell's metrics blob carries the active score scheme's metric key only if the run that wrote it knew to put it there. ADR-075 (issue #932) added that at run-save time; every surface written before it carries only {eai, eai_density_km2, points}. Both rungs read cell.metrics[scheme.metric], so they see nothing.

This is recurring, not a one-off backfill. The rungs resolve the active scheme at read time (core/rollup_service.py), so re-versioning a scheme or renaming its metric darkens every stored surface until it is re-saved under the new key. Run this after any migration that seeds a new scheme version, and after any change to a scheme's metric.

Two other ways to reach the same state, for completeness: re-running the scenario also fixes it (a run-save replaces a surface's cells wholesale), and the reference plane is refreshed by re-promoting through build_reference_surfaces, which already takes a result_id.

Always dry-run first — it is the default, and it reads everything the apply would (including each dataset's intensity matrix), so the plan it prints is a real preview rather than a guess:

# dry run: one org, or every live org when org_id is omitted
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import resave_run_surfaces; \
             print(json.dumps(resave_run_surfaces.apply(kwargs={'org_id': '<org-uuid>'}).get(), indent=2))"

# apply
docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "import json; from climate_lama.worker.tasks import resave_run_surfaces; \
             print(json.dumps(resave_run_surfaces.apply(kwargs={'org_id': '<org-uuid>', 'dry_run': False}).get(), indent=2))"

.apply() runs the task inline in that container rather than queueing it, which is what you want for an operator action you are watching.

The report carries dry_run, surfaces, counts and one entries row per surface. Every count key is always present, so two runs are directly comparable. What each outcome means:

Outcome	Meaning	Action
`resaved`	Re-derived and written (or, under `dry_run`, would be — the row's `applied` flag tells them apart).	none
`skipped_legacy`	`dataset_id IS NULL`: written before #589, so it names no hazard dataset and there is no identity to join a stored result to.	Re-run the scenario to mint a dataset-keyed surface.
`skipped_no_result`	Candidates were searched and none of them would have written this surface's key.	Re-run the scenario.
`skipped_orphaned_dataset`	The hazard dataset the surface names is gone, so no run could be searched for at all. Kept separate from `skipped_no_result` precisely because re-running is not available as a remedy.	Nothing to re-run. Either the surface is stale and can be deleted, or the dataset was removed in error.
`skipped_drift`	The exposure set changed since the run, so the stored array can no longer be attributed to named exposures (#370). Refused rather than filed under wrong coordinates.	Re-run the scenario against the current exposures.
`no_design_rp`	The dataset publishes no event at the peril's design return period (or has no stored intensity matrix). Reported as unenrichable; the surface is left untouched.	Nothing to do here — the peril's design RP or its dataset's event table is the thing to fix.
`no_metric_key`	No seeded score scheme claims this peril at this intensity unit, so there is no key to file a metric under.	Seed the scheme.
`failed`	That surface raised; the rest of the pass continued.	Read the reason and the worker log.

Two failure modes worth knowing:

A false clean. org_risk_surfaces is under RLS, so a connection with no app.current_org_id bound sees zero rows. The task enumerates organizations from organizations (which is not org-scoped) and binds the GUC per org for exactly this reason, and logs a warning when a whole pass matches no surfaces at all. If you see that warning against a database you know is not empty, treat it as a permissions problem, not as "already done".
Applying with run-saves switched off. resave_run_surfaces refuses to run with dry_run: False while SURFACE_RUN_SAVE_ENABLED is false, because the writer would decline every write and the pass would report a clean it did not achieve.

Recovering a mis-rejected dataset review (re-arm)¶

Symptom: a dataset that should be in the review queue is not, and GET /v1/admin/pending-ingests?status=rejected shows it as rejected (or ingest_failed). A single mis-click does this, and so does a bulk sweep — 22 rows were rejected on prod in one scripted run on 2026-08-11, 21 of them inside 0.22 s.

Why it does not fix itself: the weekly poll records every item it sees in dataset_source_catalog, whether or not anyone approves it. A decided row is therefore skipped by every later poll until the upstream content changes. That is deliberate — it is what stops a declined item nagging admins every week — but it means the decision is absorbing. Re-arm is the supported way back: it deletes that item's last-seen marker so a poll sees the item as new again, and it dispatches a scoped poll (that one source, those items, no admin notification fan-out) so the item is back in the queue in seconds rather than at 05:00 next Monday.

Re-arm never touches the decided row — it stays as the audit record — and it never triggers an ingest. The item comes back through review, not around it. Only rejected and ingest_failed rows are re-armable; anything else is a 422.

Every call below needs a platform admin token (is_platform_admin = true), not an org admin. Set API=https://<host> and TOKEN=<jwt> first.

1. Find the rows¶

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=rejected&limit=500" \
  | jq -r '.data.items[] | [.id, .source_id, .item_id, .reviewed_at] | @tsv'

Narrow to the sweep you are undoing before you re-arm anything — reviewed_at is the field that separates one bad batch from years of deliberate decisions:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=rejected&limit=500" \
  | jq '[.data.items[] | select(.reviewed_at >= "2026-08-11T00:00:00")
         | select(.source_id == "worldpop") | .id]' > /tmp/rearm-ids.json
jq 'length' /tmp/rearm-ids.json   # sanity-check the count before sending it

2. Re-arm¶

One row:

curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests/<id>/re-arm" | jq .data

The whole batch in one call (max 500 ids):

jq -c '{ingest_ids: .}' /tmp/rearm-ids.json > /tmp/rearm-body.json
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" --data @/tmp/rearm-body.json \
  "$API/v1/admin/pending-ingests/re-arm" | jq '{counts, polls_dispatched, failed}'

The bulk call is 200 even when every id fails — read counts, not the status code. Ids that do not exist, or rows that are not in a dead-end status, come back in failed with a code of not_found / not_rearmable while every other id is still re-armed.

Read three fields on each re-armed row:

Field	Meaning if it is not what you expect
`last_seen_cleared`	`false` = there was no marker, so the item was already pollable. Harmless; not a failure.
`source_registered`	`false` = the item's source is no longer registered in this build. The marker was cleared but no poll can act on it — the item stays invisible until the source is registered again and a poll runs.
`polls_dispatched`	`[]` = nothing was polled (this is always the case when `source_registered` is `false`). Otherwise one entry per source, with the item count.

3. Confirm it came back¶

The scoped poll runs on the compute queue and takes as long as the source's manifest fetch — seconds for JRC/WorldPop, longer for a queued Copernicus retrieval. Then:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$API/v1/admin/pending-ingests?status=awaiting_review&limit=500" \
  | jq -r '.data.items[] | [.source_id, .item_id] | @tsv' | sort

If the item is still missing after the poll has had time to run:

# Did the scoped poll actually see the item?
docker logs climate_lama_worker 2>&1 | grep poll_dataset_source | tail -20

seen=0 — the source no longer publishes that item_id. Nothing is broken; the upstream dataset is gone or was renamed, and re-arm cannot conjure it back.
queued=0 with seen=1 — an awaiting_review row already exists for that item at the same content key (the poll's re-queue guard). Check the queue again; it is already there.
error non-null — the source's manifest fetch failed. Triage it like any other source failure, then re-run the poll: ./dc exec -T worker python -c "from climate_lama.worker.dataset_polling import poll_dataset_source; print(poll_dataset_source.delay('<source_id>', ['<item_id>']).id)"

Every re-arm writes a pending_ingest.rearmed audit entry recording who did it, last_seen_cleared, source_registered, and whether it was part of a batch — so a recovery is as auditable as the rejection it undoes.

DEPLOYMENT.md — production operations (Hetzner host, release flow, backups).
quickstart/index.md — first-run setup; step 5 is exactly the scripts/demo.py run this runbook is for.
scripts/demo.py — the script itself; read its module docstring for the full authentication contract.

Troubleshooting Runbook — scripts/demo.py and the local dev stack¶

Before you start — a checklist that catches most first-run failures¶

Quick triage — where to look first¶

Postgres¶

Redis¶

MinIO¶

API¶

Worker¶

Reading a job's state without the API¶

A job stuck in running that never resolves¶

Retrying a step without rerunning the whole script¶

Reference — dev stack ports and container names (docker-compose.yml)¶

Operator scripts — the one-shot repair/migration tools¶

migrate_org_data.py specifically¶

Management tasks — the answer plane's two hand-invoked builders¶

resave_run_surfaces — when a stored surface has no rollup metric¶

Recovering a mis-rejected dataset review (re-arm)¶

1. Find the rows¶

2. Re-arm¶

3. Confirm it came back¶

Related docs¶

Troubleshooting Runbook — `scripts/demo.py` and the local dev stack¶

A job stuck in `running` that never resolves¶

Reference — dev stack ports and container names (`docker-compose.yml`)¶

`migrate_org_data.py` specifically¶

`resave_run_surfaces` — when a stored surface has no rollup metric¶