Skip to content

forge-climada-api-archive — CLIMADA Data API download-and-archive script

forge mini-spec, 2026-08-10. Trivial-case lane: one issue → /build. This doc is the issue body for the single work item below.

Resume state (paused 2026-08-10)

Run paused mid-forge by the owner. Where things stand and how to continue:

  • Origin: /forge --worktree --ghost run replacing the empty ../climada-datasets side project (conda env only, zero scripts — stays untouched) with an in-repo download-and-archive script. Validation verdict: reasonable, with the design correction that the CLIMADA Data API is plain REST (litpop_exposure.py proves it), so no climada package / conda env is needed and ADR-024 stays intact.
  • Done: ground-truth checks (no duplicate issues; milestone:forge-climada-api-archive unused; adjacent issues #624/#607 are related-not-duplicate catalog packs); planner agent produced this mini-spec, grounded and file-verified.
  • Issue filed (2026-08-13): #888 — the durable, self-contained handoff (spec + assessor watch-items folded in as pre-implementation checks + build guidance), labelled deferred + milestone:forge-climada-api-archive. The assessor agent never ran; its three watch-items are now #888's pre-implementation checklist.
  • To resume: /build 888 --worktree. Do NOT re-create the issue — check gh issue list --label "milestone:forge-climada-api-archive" --state all first; #888 already exists.
  • Assessor watch-items (queued for it, unresolved): (a) standalone script vs. the existing admin downloads API + download_to_spaces Celery task — script chosen for local/agent-driven ops use, assessor should sanity-check; (b) whether non-catalog UUIDs under raw/climada-api/… create phantom-orphan findings in the bucket audit tooling (audit regexes currently target hazards/ and processed/ only — likely fine, verify); (c) product axis: this is ops tooling, not MVP-slice work — explicitly owner-requested.
  • Tropical-cyclone-for-Greece question, resolved 2026-08-12 (does not change this script's design): a session asked whether this archiver could pull Greek TC data in place of #624/#607's "non-Greek" scoping. Checked the live API directly — it does return 17 active country_iso3alpha=GRC tropical_cyclone records — but downloading and opening 3 representative HDF5 files (both track-generation methodologies) found every one has an empty intensity/data array: real events/centroids over Greece, zero non-zero wind intensity. The API's country tag is a metadata-layer false positive; #624/#607's "non-Greek" framing is a verified data constraint, not an oversight — see the comment threads on those issues and docs/plan/phase-9-full-catalog-walkthrough.md's tropical_cyclone row. This archiver script is still generically useful for any CLIMADA Data API source (format-agnostic download+manifest+verify) — only the Greek-TC use case is ruled out.

1. Goal

One command (python scripts/climada_data.py fetch ...) downloads a chosen dataset/country/hazard/version from the public CLIMADA Data API, archives it verbatim into the production object-storage bucket under an ADR-029 raw/ leaf with a manifest.json, verifies the upload, and deletes the local copy — so the owner never needs a climada conda env or the empty ../climada-datasets side project again.

2. Scope + acceptance criteria

  • [ ] New scripts/climada_data.py with argparse subcommands list and fetch, following the repo's script conventions (_PROJECT_ROOT sys.path fixup, logging.basicConfig, def main(argv: list[str] | None = None) -> int, sys.exit(main())) as in scripts/backfill_dataset_cogs.py:39-214.
  • [ ] Filters on both subcommands: --data-type (required for list), --country (ISO3 → country_iso3alpha query param), --name, --version, --status (default active). fetch additionally accepts --uuid to address one record exactly, which bypasses the filter query.
  • [ ] list prints a stable one-line-per-dataset table (uuid, name, version, status, file count, total bytes) and exits 0 on zero matches (empty result is not an error for list; it is an error for fetch).
  • [ ] HTTP via httpx only (already a first-party dependency, pyproject.toml:47); follow_redirects=True (the non-slash path 301s) against https://climada.ethz.ch/data-api/v1/dataset/. No climada import anywhere (ADR-024).
  • [ ] fetch pipeline, in this order, per file of the resolved record:
  • Streamed/chunked download (client.stream("GET", url), iter_bytes(chunk_size=8*1024*1024)) to a local temp dir (--work-dir, default tempfile.mkdtemp), writing .tmp then replace() — never a whole-file in-memory read.
  • md5 computed incrementally during the stream and compared to the record's declared check_sum ("md5:<hex>"); mismatch aborts before any upload.
  • Upload to raw/climada-api/{data_type}/{version}/{climada-dataset-uuid}/{filename} via build_minio_client() + get_settings().minio_bucket.
  • Verify with stat_object — uploaded size must equal the local file size.
  • Write the leaf manifest.json (same prefix) after all files verify.
  • Only then delete the local temp files; --keep-local skips step 6.
  • [ ] manifest.json provenance payload: API query URL + resolved record URL, dataset uuid / name / version / data_type / status / license, upstream record properties verbatim, per-file {filename, object_key, size_bytes, md5, source_url}, retrieved_at (UTC ISO-8601), and archived_by: "scripts/climada_data.py".
  • [ ] Idempotent: before uploading, stat_object the target key; if it exists with matching size, log and skip (still counted as success, manifest still rewritten). --force re-uploads unconditionally.
  • [ ] --dry-run resolves and prints the planned object keys without downloading or uploading.
  • [ ] Module docstring documents the ADR-029 deviation explicitly: the leaf {dataset-id} component is the CLIMADA Data API's own dataset UUID, not a hazard_datasets/exposure_datasets row id — deliberate, because archived raw sources have no catalog row. Cross-reference _raw_prefix() in src/climate_lama/core/catalog.py:148-162, which builds the same shape from a catalog row UUID.
  • [ ] {source} component is the literal climada-api; {version} pins the upstream release label literally (e.g. v3), falling back to v1 only when the record carries none — matching _raw_prefix's own fallback.
  • [ ] Failure semantics: any upload/verify failure leaves all local files in place (no cleanup) and returns a nonzero exit code with a log line naming the object key. Catch minio.error.S3Error and httpx.HTTPError specifically — never bare Exception.
  • [ ] Tests at tests/scripts/test_climada_data.py (mocked httpx transport + mocked Minio client, no Docker, no PostgreSQL, runs in the plain uv env):
  • list filtering — params sent include status=active by default; non-dict records are dropped; both bare-list and {"datasets": [...]} response shapes are accepted (the API returns either — see _extract_records, scripts/packs/litpop_exposure.py:550-559).
  • fetch happy path — correct object key, stat_object called, manifest content asserted, local temp dir empty afterwards.
  • Idempotent skip — pre-existing key with matching size ⇒ no put_object, exit 0; --force ⇒ put_object called.
  • Upload failure (S3Error) ⇒ local files still present, exit code nonzero, no manifest.json written.
  • --keep-local ⇒ files retained on success.
  • [ ] ruff check + ruff format --check clean at line length 100; mypy src-config clean for the script's annotations; type hints + Google-style docstrings on every public function.
  • [ ] Landed per NO-PR-CI: never add the ci label; the debounced trunk gate verifies the merge.

3. Out of scope

  • Any catalog or DB registration — this script creates no hazard_datasets/exposure_datasets rows, writes nothing to PostgreSQL, and does not update catalog.v1.json.
  • Ingest packs: #624 (tropical-cyclone pack consuming CLIMADA Data API footprints) and #607 (TC track/HDF5 ingest path) are related, not duplicated — they are catalog-ingest packs; this is a generic raw-source archiver. Neither is implemented or modified here.
  • Touching, migrating, or deleting ../climada-datasets (owner does that manually once this lands).
  • Any UI or /v1/ API surface; any Copernicus/CDS/EWDS source; any format conversion (HDF5→CSV etc. — that already lives in scripts/packs/litpop_exposure.py).
  • Deploying the script into a container image (scripts/ is not copied into the images except seed_demo.py) — it runs from a clone with the repo's own env.

4. Proposed decomposition

ONE work item, size M. No sensible split: the manifest, verify, and cleanup steps are one transaction and would be untestable apart.

Files touched: - scripts/climada_data.py (new, ~350–450 lines incl. docstring) - tests/scripts/test_climada_data.py (new) - No changes to src/, migrations/, pyproject.toml, or Docker/compose.

5. Milestone id

forge-climada-api-archive

6. Open design points resolved

  • Response shape / pagination. The API returns either a bare JSON list or a mapping with datasets/data; there is no pagination cursor in the observed contract. Reuse the normalisation logic proven in _extract_records (litpop_exposure.py:550) rather than assuming a list. Records carry uuid, name, version, status, license, properties (dict), and files (list of {url, file_size, check_sum, file_name}) — confirmed by the same file's climada_api_identity() usage at lines 349-396.
  • Country filter mapping. ISO3 maps to the query param country_iso3alpha (not country), as used at litpop_exposure.py:516-520; data_type and status are passed under their own names.
  • HTTP client choice. httpx — a top-level dependency (pyproject.toml:47); requests>=2.32 appears only in an optional extra group (line 126), so httpx is the correct choice for a script runnable from the base env.
  • Checksum field. The API's check_sum is prefixed md5:; strip the prefix before comparing. Verified byte-exact for at least one dataset per the litpop pack docstring (line 106), so treating it as authoritative is safe — but a missing check_sum must warn and continue rather than abort (not all records are guaranteed to carry one).
  • stat_object failure modes. Three distinct cases, handled separately: (a) S3Error with code == "NoSuchKey" on the pre-check ⇒ normal "not yet archived", proceed; (b) S3Error with any other code (e.g. AccessDenied, NoSuchBucket) ⇒ fatal, exit nonzero without touching local files; (c) post-upload stat_object succeeds but size mismatches ⇒ fatal, keep local files, log both sizes. Do not use stat_object truthiness — it raises, it does not return None.
  • Credentials. Sourced entirely from get_settings() via build_minio_client() (src/climate_lama/worker/ingest/_minio.py:24-32) — no endpoint or key literals in the script. Pointing at production means running with the production .env; the script logs the resolved bucket name (never the credentials) at INFO before its first write.