Skip to content

Phase 6a — Backbone admin/ops + ingest

Status: Closed 2026-06-09 (split from the original Phase 6 per mid-review-phase-6.md Option 2). The 6 backbone/ops items below shipped and merged; the 8 UI-bearing / cross-stack items moved to Phase 6b — Admin panel UX. Shipped (6a): 6.6 async ingest (#293), 6.7 SSRF + admin download (#282), 6.9 result cache (#283), 6.11 usage metering (#284), 6.13 release pipeline (#281), 6.14 secrets ADR (#102). Moved to 6b: 6.1, 6.2, 6.3, 6.4, 6.5, 6.8, 6.10, 6.12 — see phase-6b-admin-panel.md. Predecessor: Phase 5 — UX quality. Successor: Phase 6b — Admin panel UX.

The scope sections below are the original Phase 6 specification, retained as the authoritative acceptance criteria for both the shipped 6a items and the 6b items (whose issues copy from here).

Goal

Make the Phase 2 multi-tenancy + RBAC plumbing visible and operable: admin panel, dataset catalog (user-facing + admin-curated), async ingest pipeline (demoted to Tier 3 per R4 B1, but still required to scale real datasets), external dataset polling, in-app notifications for async work, usage metering for observability, deployment/release pipeline ADR, audit log UI, and the predefined-results cache. This is the phase where a self-hoster can actually run the platform in a multi-user mode without shell access.

Entry conditions

  • Phase 5 closed: error taxonomy retrofit live, i18n scaffold in place, settings page exists (provides the surface admin toggles will attach to).
  • ADR-029 (DO Spaces layout) merged in Phase 3.
  • Scenario model + provenance in place from Phase 3.

Out of scope (deferred)

  • Managed hosting, K8s, ArgoCD, cert-manager — parked in phase-managed-hosting.md.
  • Stripe metering + billing — Tier 4.
  • Full scenario versioning (#P) — Tier 3, later phase.
  • Reports (PDF, templates) — Phase 7.
  • CLIMADA compute adapter, IngestAdapter ABC — Phase 7.

Scope

6.1 — Admin panel shell (#21)

Context. Vision #21 Tier 2. Dedicated /admin/* route tree behind a hard RBAC gate (requires admin role from Phase 2). Lazy-loaded bundle so regular users don't pay the download cost. Top-level tabs: Users, Orgs, Datasets, Jobs, Audit, System, Branding.

Acceptance criteria. - [ ] New route tree in climate-lama-ui/src/admin/, lazy-imported from the main router. - [ ] Route guard rejects non-admin users with a 403 page; API also enforces per endpoint (defence in depth). - [ ] Tab shell with stubs for each section — subsequent work items fill the panes. - [ ] Separate bundle-split confirmed (Vite output shows admin chunk only loads when visited). - [ ] Vitest: guard blocks non-admin; admin can navigate all tabs.

Cross-stack: UI + backbone (admin-scoped endpoints already mostly exist; this item audits coverage). Linked ADR: none. Suggested /analyze tier: 3 (Sonnet · high).

6.2 — Users + orgs management (admin panel)

Context. Vision #21 Tier 2 flesh-out. Admin can list/search users, invite new ones, assign roles, deactivate, reset API keys. For orgs: list, create, edit metadata (billing_email, plan_tier, data_source_allowlist), delete (soft).

Acceptance criteria. - [ ] Backbone endpoints: GET/POST/PATCH/DELETE /v1/admin/users, same for /v1/admin/orgs. RBAC: admin role required; platform-admin flag for cross-org operations (introduce is_platform_admin boolean on users if not already there). - [ ] Invite flow: admin creates user → email sent with signup link (piggybacks on Phase 2 email verification if present; otherwise prints link in dev mode). - [ ] UI panes: searchable user list, user detail with role edit + deactivate, org list with editable metadata. - [ ] Audit every write to the audit log (6.4 below). - [ ] Tests cover: role change, deactivate, org metadata edit, platform- admin gate on cross-org calls.

Cross-stack: backbone (endpoints + schema tweaks) + UI. Linked ADR: none. Suggested /analyze tier: 4 (Opus · high) — RBAC-adjacent, schema change, two repos.

6.3 — Datasets surface (#6 + #7)

Context. Vision #6 + #7 Tier 2. Unified "Datasets" UI with Catalog + Upload tabs. Exposures and hazards share the surface with hazard-specific filters. Drives both the user-facing browse (this item) and the admin- only curation (6.5).

Acceptance criteria. - [ ] New /datasets route with tabs "Hazards" + "Exposures", each with Catalog (shared + global) + Upload panes. - [ ] Backbone GET /v1/hazards and GET /v1/exposures support source, license, hazard_type, region, scenario, year_range query filters. - [ ] Each row shows name, source, license, date, type, preview action. - [ ] Preview endpoint GET /v1/hazards/{id}/preview and the exposure equivalent return first N rows + bbox + sample thumbnail URL (for rasters) or a point-density summary (for vectors). - [ ] Upload: CSV + XLSX (CLIMADA entity format) for exposures; existing HazardUploadModal.tsx extended with metadata fields and manifest- driven flow per ADR-029. - [ ] HDF5 + GeoTIFF uploads deferred; show a "Contact admin" link.

Cross-stack: backbone (filters + preview + uploads) + UI (route + components). Linked ADR: references ADR-029. Suggested /analyze tier: 4 (Opus · high) — two endpoints, filter/preview logic, two upload paths, new route.

6.4 — Audit log (write side + UI) (#E)

Context. Vision #E. Write side shipped in this phase (not Phase 3) now that admin-panel endpoints are the primary audit source. UI browsing lives in admin panel Audit tab.

Acceptance criteria. - [ ] Alembic migration: audit_log(id UUID, org_id FK, user_id FK, action TEXT, target_type TEXT, target_id UUID, metadata JSONB, created_at TIMESTAMP) with indexes on (org_id, created_at) and (target_type, target_id). - [ ] FastAPI middleware / dependency writes audit entries on every admin-panel write endpoint (6.2) and on every user-impactful action (scenario delete, API key rotation). Append-only — no updates, no deletes. - [ ] Admin UI Audit tab: paginated table, filter by user / action / target-type / date range; JSON detail modal per entry. - [ ] RBAC: admin sees own org; platform-admin can filter cross-org. - [ ] Tests: every admin endpoint writes exactly one audit row.

Cross-stack: backbone (schema + middleware) + UI (admin panel tab). Linked ADR: none. Suggested /analyze tier: 4 (Opus · high) — schema + middleware + cross-module.

6.5 — Dataset catalog (DO Spaces) — admin curation (#19)

Context. Vision #19 Tier 2. 3-dataset curated catalog (JRC flood, CMIP6 TC tracks, Worldpop population) with versioned JSON manifest per ADR-029 layout.

Acceptance criteria. - [ ] Catalog manifest catalog.v1.json in DO Spaces raw/ root, listing every dataset: {name, type, source, license, region, year, scenario, s3_url, sha256, manifest_url}. - [ ] Backbone service CatalogClient fetches manifest at boot, caches locally, refreshes on admin-triggered reload. - [ ] Admin UI Datasets tab (inside 6.1 shell): list catalog entries, view manifest, trigger refresh, "request ingest" action (hands off to 6.6 pipeline). - [ ] Each dataset entry ships a manifest.json at its leaf per ADR-029. - [ ] Self-hosters: env var CATALOG_URL points at their own bucket / catalog URL; same code path. - [ ] Seed the 3 catalog entries + their manifests in CI; fixtures under tests/fixtures/catalog/.

Cross-stack: backbone (CatalogClient + admin endpoints) + UI (admin tab) + data ops (seed authoring). Linked ADR: references ADR-029. Suggested /analyze tier: 4 (Opus · high) — new service + external I/O + versioning.

6.6 — Async ingest pipeline (#27, demoted)

Context. Vision #27 demoted to Tier 3 per R4 B1 but still required to make real datasets usable. Celery chord (parallel subtasks + callback) that consumes a DO-Spaces URI, validates, chunks, and writes to the DB. Stream bytes, never load full files in memory.

Acceptance criteria. - [ ] worker/ingest/pipeline.py defines the chord pattern: download → validate → chunk → N parallel writes → aggregate → commit. - [ ] Chunk size configurable per hazard type in settings. - [ ] ingest_jobs table tracks state per chord execution: id, dataset_id, status, progress_pct, logs TEXT, error_code, started_at, completed_at. - [ ] Admin UI: ingest jobs list with progress + logs + retry/cancel actions. - [ ] Memory test: 5GB netCDF fixture ingests without exceeding 1GB worker RSS. - [ ] Integration test: end-to-end ingest of a small fixture.

Cross-stack: worker + backbone (endpoints + schema) + UI (admin view). Linked ADR: none. Suggested /analyze tier: 5 (Opus · extra high) — chord pattern is load-bearing; concurrency/memory correctness matters; new subsystem.

6.7 — Admin download service + SSRF hardening (B4)

Context. Vision #27 download side + R4 B4. Admin triggers raw dataset downloads from external sources into DO Spaces. Must be resumable, streaming, observable; and SSRF-hardened via a per-org source allowlist + platform default list.

Acceptance criteria. - [ ] Celery task download_to_spaces(source_url, target_path, org_id) — streams bytes in chunks, records progress in downloads table, writes to raw/{source}/... per ADR-029. - [ ] downloads table: id, org_id, source_url, target_path, status, bytes_total, bytes_done, retry_count, error_code. - [ ] SSRF policy: platform default allowlist (CLIMADA data API, JRC, Copernicus CDS, NOAA, OSM) baked into config; per-org additions need platform-admin approval via 6.2 admin-user endpoint. - [ ] URL resolution guarded: resolve DNS, reject private IP ranges (RFC 1918, 127.0.0.0/8, ::1, link-local), reject redirects leaving the allowlist. - [ ] Admin UI: download manager surface with progress, logs, retry, cancel. - [ ] Short ADR "SSRF policy for admin download endpoints" capturing the allowlist rules — goes into DECISIONS.md with **Phase**: 6. - [ ] Integration test: malicious URL (file://, localhost, private IP) refused with coded error.

Cross-stack: worker + backbone + UI + docs. Linked ADR: authors a new ADR (number assigned when merged, e.g., ADR-031). Suggested /analyze tier: 5 (Opus · extra high) — security-sensitive per /analyze rubric; SSRF is a real attack surface.

6.8 — External dataset polling (#22)

Context. Vision #22 Tier 3. Scheduled mechanism watches external dataset publishers for new/updated raw datasets. Notifies admin (via 6.10 notifications) for review before ingestion.

Acceptance criteria. - [ ] DatasetSource ABC in src/climate_lama/ingest/sources/base.py with list_available(), get_metadata(id), download(id, dest). - [ ] Three first-party DatasetSource implementations: JRC European flood, Copernicus ERA5, Worldpop. Each in its own module. - [ ] Celery beat schedule runs each source weekly; diff against dataset_source_catalog table; on change, inserts a row in pending_ingests with status=awaiting_review. - [ ] Admin UI pending-ingests tab: review + approve (triggers 6.6 pipeline) + reject. - [ ] No auto-ingest in v1; human in the loop by design. - [ ] Unit tests with mocked source responses for each implementation.

Cross-stack: backbone (new module + Celery beat) + UI (admin tab). Linked ADR: references ADR-027 (adapter-by-design) — DatasetSource is one of the five extension points. Suggested /analyze tier: 4 (Opus · high) — three adapters, Celery beat + review workflow.

6.9 — Predefined results caching (#20)

Context. Vision #20 Tier 2. Deterministic SHA256 cache key; pre-warm popular scenarios; invalidate on input version bumps.

Acceptance criteria. - [ ] Cache key: SHA256(exposure_dataset_id || exposure_dataset_sha256 || hazard_dataset_id || hazard_dataset_sha256 || impact_function_id || impact_function_version || year || scenario || discount_rate || growth_rate). Measures NOT in key — applied post-cache as multiplier per vision #20 rationale. - [ ] Schema: result_cache(cache_key TEXT PK, result_id UUID FK, created_at, hit_count). - [ ] POST /v1/compute/impact checks cache → hit returns 200 with cache_hit: true; miss dispatches Celery as today. - [ ] Celery beat pre-warm job runs hand-curated list of 20 demo scenarios overnight. List lives in config/prewarm.yaml. - [ ] Impact function version bump invalidates dependent entries (already exists from Phase 3 #3.8). - [ ] Prometheus counter for cache hit rate. - [ ] Integration test: compute same request twice → second returns cache_hit: true.

Cross-stack: backbone (schema + lookup + pre-warm) + infra (Prometheus metric). Linked ADR: none. Suggested /analyze tier: 4 (Opus · high) — cache invalidation + key design.

6.10 — In-app notifications (#30)

Context. Vision #30 Tier 3. In-app notification surface (bell icon + dropdown) for async job completion, dataset availability, admin approvals. Minimum viable: polling, no WebSockets.

Acceptance criteria. - [ ] Schema: notifications(id, user_id, org_id, type, payload JSONB, created_at, read_at, dismissed_at). - [ ] Celery hook emits notifications on job completion (6.6 ingest, 6.7 download, compute impact) and admin approvals (6.8 pending-ingests). - [ ] GET /v1/users/me/notifications?unread=true endpoint with pagination. - [ ] UI bell icon with unread count; dropdown lists recent; click marks read; dismiss permanently removes. - [ ] Polling interval 30s by default, configurable in settings (5.7). - [ ] Taxonomy locked to 5 initial types: job_complete, dataset_available, system_announcement, admin_approval_needed, download_ready. Adding a type requires updating the ADR-028 code registry.

Cross-stack: backbone + UI. Linked ADR: references ADR-028 (codes for notification types). Suggested /analyze tier: 3 (Sonnet · high).

6.11 — Usage metering observability slice (#M)

Context. Vision #M Tier 3 observability slice (Stripe integration stays Tier 4). Instrument compute-seconds / runs / storage per tenant per billing period for admin visibility — no billing wiring.

Acceptance criteria. - [ ] Every compute job records compute_seconds, cpu_seconds, result_bytes to a usage_events table. - [ ] Aggregation view usage_by_period(org_id, period, compute_seconds, run_count, result_bytes) — monthly roll-up; period = first of month. - [ ] GET /v1/admin/usage?org_id=...&period=... endpoint. - [ ] Admin UI: usage tab with bar chart per month, CSV export. - [ ] Prometheus counter for compute-seconds labelled by org. - [ ] Schema designed so Stripe metered-billing API can read from it later without migration (Tier 4 plug-in).

Cross-stack: backbone + UI. Linked ADR: none. Suggested /analyze tier: 3 (Sonnet · high).

6.12 — Adaptation measures catalog UX (#10)

Context. Vision #10 catalog + selection. Phase 1 shipped the data model + CRUD API (#54-57); UI is still minimal (CostBenefitPanel.tsx). This item delivers the Measures page + scenario-level multi-select.

Acceptance criteria. - [ ] /measures route lists all predefined measures, filterable by hazard / region / source. Detail view shows cost, effectiveness curve, source, lifetime. - [ ] Scenario wizard gains a "select measures" step with multi-select chips per riskwise-v2 pattern. - [ ] Cost-benefit panel updated to render the selected measure set; the compute path already accepts measures from Phase 1. - [ ] Attribution (#3.10) exposed per measure. - [ ] Custom measure builder + save-as-set explicitly deferred (vision #10 Tier 3 later-in).

Cross-stack: UI + (minor) backbone filters on existing /v1/measures endpoint. Linked ADR: none. Suggested /analyze tier: 3 (Sonnet · high).

6.13 — Deployment ADR + release pipeline (#29)

Context. Vision #29 Tier 3. Write the ADR (K8s + ArgoCD + Helm stays Tier 4, only the path is documented) and ship the release pipeline: versioned image tags on git tags. Cheap now, pays off for both self- hosters and future managed hosting.

Acceptance criteria. - [ ] ADR drafted: self-hosters on Docker Compose indefinitely; managed hosting path sketched (K8s + Helm + ArgoCD + cert-manager) but not implemented; Docker images treated as the universal artifact. - [ ] .github/workflows/release.yml: on git tag v*, build backbone + UI Docker images, tag with the semver, push to GHCR. - [ ] Image tags follow v{major}.{minor}.{patch} + latest for the most recent release. - [ ] docker-compose.yml references ${CLIMATE_LAMA_TAG:-latest} so users can pin. - [ ] Docs-site page deployment/upgrading.md documents the tag cadence and upgrade path (Alembic migration flow). - [ ] **Phase**: 6 header line on the ADR.

Cross-stack: docs + CI. Linked ADR: authors a new ADR (e.g., ADR-032). Suggested /analyze tier: 3 (Sonnet · high).

6.14 — Secrets and config management ADR (#102)

Context. Phase 0 retrospective flagged that no ADR covers how secrets and config are managed across local dev, CI, and (future) hosted deploys. Pulled forward from the parked managed-hosting phase to land alongside the deployment ADR (§6.13) — both are ops decisions self-hosters need documented before any hosted deploy, and pairing them avoids re-doing the discussion when managed hosting revives.

Acceptance criteria. - [ ] ADR added to docs/DECISIONS.md covering: dev (.env + pydantic-settings), CI (GitHub Actions secrets → env vars), hosted path (per-environment secrets backend candidates — Hetzner vault, Doppler, 1Password Connect, AWS Secrets Manager — decision deferred until managed hosting unparks, alternatives captured). - [ ] Rotation story documented (who rotates, on what cadence, how rollouts pick up the new value). - [ ] Naming conventions for env vars codified (prefix, casing, scope markers). - [ ] Alternatives section covers committed-encrypted-secrets (sops, git-crypt) and explains why the env-var path was chosen instead. - [ ] **Phase**: 6 header line on the ADR. - [ ] Cross-referenced from phase-managed-hosting.md retrospective backlog table so the parked phase points at the resolved decision.

Cross-stack: backbone docs only. No code change. Linked ADR: authors a new ADR (e.g., ADR-033). Suggested /analyze tier: 2 (Sonnet · medium) — pure prose, existing template.

ADRs authored in this phase

ID Title Owner work item
ADR-031 (approx.) SSRF policy for admin download endpoints 6.7
ADR-032 (approx.) Deployment ADR — Compose, images, K8s path 6.13
ADR-034 Secrets and config management strategy 6.14

Numbers assigned at merge time; may shift if earlier phases author additional ADRs.

Exit criteria

  • All 14 work items closed.
  • Admin panel usable for a self-hoster to provision users, orgs, datasets, ingest jobs, audit log, usage views — no shell access required for common ops.
  • Admin-panel bundle confirmed lazy-loaded (non-admin users don't download it).
  • result_cache hit rate visible on the Grafana dashboard.
  • 3-dataset catalog seeded and browsable in the Datasets surface.
  • One full download → ingest → appear-in-catalog round-trip demonstrated end-to-end.
  • Release pipeline publishes versioned images on tag.

Risks

  • 6.6 ingest pipeline is the phase's most complex item. Memory bounds + partial-failure recovery are the usual traps. Run /build --review not auto-merge.
  • 6.7 SSRF is security-sensitive. Independent review required before merge; even a small gap in the allowlist validation can expose internal infra.
  • 6.8 polling depends on external services staying shape-stable. Expect adapter breakage when JRC / Copernicus change endpoints; budget maintenance.
  • 6.1 admin panel bundle-split: verify with production build output, not just dev mode. Dev mode serves everything.
  • 6.11 metering schema is the future billing foundation. Any design change after launch means a migration. Get it right up front.