Deployment Guide¶

The canonical reference for deploying Climate-Lama to production and operating the live environment. It answers three things: where everything lives (the inventory), how a change travels from your laptop to production (the end-to-end release flow), and the host-specific setup and day-2 operations.

The live production stack runs on a Hetzner host behind a shared Caddy reverse proxy (automatic TLS), with object storage on Hetzner Object Storage. The repo's docker-compose.prod.yml also ships a portable nginx + certbot path for a fresh host with no existing proxy (see TLS Certificates / First Deploy).

Production environment — at a glance¶

Single source of truth for "where is X / who owns Y". No secret values here — only locations and owners. Secret values live in .env.prod on the host (mode 600) and in your password manager; this table only says where to look.

Layer	What / where	Identifier	Managed in	Notes
Domain (registrar)	`climate-lama.online`	—	Namecheap	bought + auto-renewed at Namecheap
DNS (authoritative)	A records, edited by hand	—	Namecheap → Advanced DNS	NOT Hetzner DNS, NOT Cloudflare. `api` → server; `@` + `www` → server
Server	Hetzner VPS	`159.69.211.124` (`lamaface-dev-01`)	Hetzner Cloud console	single-tenant — runs only the Climate-Lama stack (see note below)
Edge / TLS	shared Caddy (`:80`/`:443`)	`/opt/caddy/Caddyfile`	on the host	automatic Let's Encrypt + renewal; reverse-proxies by container name on `caddy_net`
API	backbone `api` service	`https://api.climate-lama.online`	`/opt/climate-lama`	pinned image `…/climate-lama:${CLIMATE_LAMA_TAG}`
UI	`climate-lama-ui` SPA	`https://climate-lama.online` + `www`	`/opt/climate-lama` (overlay)	pinned `…/climate-lama-ui:${UI_TAG}`; same-origin, proxies `/v1` to api
Compute	`worker` (Celery) + `climate-lama-engine`	—	inside the worker image	engine is a pip dependency baked in, not a service; consumes the `compute` queue only (`--queues=compute --concurrency=1`): ingest chords, event impact, cost-benefit, reports, maintenance
Impact runs	`impact-worker` (Celery)	—	same worker image, own container	consumes the `impact` queue only (`--queues=impact --concurrency=3`), so runs and matrix cells compute side by side and never queue behind an ingest; un-profiled, rolled by the same `up -d`. See Celery workers and queues for the memory budget
Live feeds	`live-worker` (Celery)	—	same worker image, own container	consumes the `live` queue only (`--queues=live --concurrency=1`), so a feed tick never queues behind a multi-hour ingest; un-profiled, rolled by the same `up -d` — no separate image or `release.yml` entry
Database	PostgreSQL + PostGIS	container + `postgres_data` volume	on the host	back up with `pg_dump` (see Backups)
Cache / queue	Redis	container + `redis_data` volume	on the host	Celery broker + result backend
Object storage / datasets	Hetzner Object Storage	bucket `climate-lama-storage`, location `nbg1`	Hetzner Cloud console	endpoint `nbg1.your-objectstorage.com` — all datasets / tiles / reports live here
Container registry	GHCR (private)	`ghcr.io/cortomaltese3/climate-lama{,-worker,-ui}`	GitHub Packages	host authed via PAT `ghcr-pull-hetzner` (`read:packages`) — rotate before expiry: secret-rotation runbook
Docs site	MkDocs → Cloudflare Pages	`climate-lama.pages.dev`	Cloudflare	build/publish only; unrelated to the app runtime
Repos	backbone / UI / engine	`climate-lama` · `climate-lama-ui` · `climate-lama-engine`	GitHub	see Release flow

Production login (pointer, not a secret store): org northlane, user develama@protonmail.com (role org_admin). The password is held out-of-band (not committed); reset it with the psql snippet in Resetting a user's password (substitute the email).

Host file layout

/opt/climate-lama/ — docker-compose.prod.yml + docker-compose.hetzner.yml (overlay, both mode 644, delivered by the deploy workflow — see Compose file delivery) + .env.prod (mode 600, hand-managed)
/opt/caddy/ — the shared edge (docker-compose.yml + Caddyfile)

Note (2026-08-03): lama-chat — a previously co-tenant app on this host — was removed between 2026-07-28 (last seen in a Caddyfile backup) and 2026-08-02, and verified absent again on 2026-08-03: no lama-chat container in docker ps, no /opt/lama-chat/ directory, and no chat.lamaface.site block in the live /opt/caddy/Caddyfile (which is down to the two Climate-Lama vhosts only). It is treated as decommissioned. The box is single-tenant until further notice — if lama-chat (or any other app) is ever reintroduced, restore the co-tenant cautions this note replaced (don't disrupt its container/volume/Caddy block, and re-check disk headroom before any large operation).

Topology¶

                         Internet
                            │  :443 (TLS, auto Let's Encrypt)
                   ┌────────▼─────────┐
                   │   Caddy (edge)   │   /opt/caddy/Caddyfile
                   └──┬────────────┬──┘
        api.climate-lama.online    climate-lama.online / www
                   │                        │
            ┌──────▼──────┐          ┌───────▼───────┐
            │  api :8000  │◄──/v1────│  ui  (nginx)  │   same-origin SPA
            └──┬───┬───┬──┘  proxy   └───────────────┘
               │   │   └──────────────► Hetzner Object Storage  (datasets/tiles)
               │   └────► Redis ◄────── worker (Celery + climate-lama-engine)
               └────► PostgreSQL+PostGIS

Release pipeline: local to production¶

How a fix or feature travels from your laptop to the live site. You never edit code or build images on the server — the host only pulls immutable, CI-built images and switches to the new tag. Two artifacts ship independently: the backbone (api + worker) and the UI.

All host commands below assume this shorthand (the live host uses the Caddy overlay, so both -f files are always passed):

cd /opt/climate-lama
P="-f docker-compose.prod.yml -f docker-compose.hetzner.yml --env-file .env.prod"

Interactively, prefer /opt/climate-lama/dc — scripts/dc in this repo, installed on the host as /opt/climate-lama/dc (chmod +x). It supplies all three flags, so ./dc ps, ./dc logs -f api, ./dc exec -T api alembic current are always correct from any directory.

Dropping --env-file is the recurring mistake, and it fails misleadingly:

$ docker compose -f docker-compose.prod.yml exec -T api python scripts/create_app_role.py
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
error while interpolating services.api.image: required variable CLIMATE_LAMA_TAG
is missing a value: set CLIMATE_LAMA_TAG in .env.prod

.env.prod does define CLIMATE_LAMA_TAG. The env_file: key feeds container environments; it does not feed compose's own ${...} interpolation, and only --env-file does that. Silver lining: the failure is an interpolation error at parse time, so a mistyped down aborts before touching a container.

Dependency delivery after the lock pin¶

Since #984, both Docker images install the dependency set recorded in uv.lock, not a fresh resolve of pyproject.toml:

RUN uv export --frozen --no-emit-project [--extra worker] -o /tmp/requirements.txt     && uv pip install --system -r /tmp/requirements.txt     && uv pip install --system --no-deps -e "."

That is the point — an image is now reproducible, and the versions CI certified are the versions it ships. It also changes how upstream releases reach production, in a way that is silent if you are not expecting it:

A newly published dependency no longer reaches production on the next image build. The image installs the lock's pin until someone runs uv lock and merges the result.

This bites hardest for climate-lama-engine, because we publish it ourselves from a sibling repo. pyproject.toml declares climate-lama-engine>=0.5.0; before #984 the worker image re-resolved that constraint at build time, so tagging engine v0.6.0 and redeploying the backbone was enough to run it. Now the worker installs whatever uv.lock pins, and nothing reports the staleness: uv lock --check only asserts that the lock still satisfies pyproject.toml, and 0.5.0 satisfies >=0.5.0 forever.

So a backbone release that is meant to pick up a new engine needs an explicit step:

uv lock --upgrade-package climate-lama-engine   # or `uv lock` to refresh everything
git diff uv.lock                                # confirm the engine line moved

Land that as its own PR, let the trunk gate run on it, and then cut the release. The same applies to any dependency whose constraint is a floating >= — the engine is simply the one whose releases this team drives.

Version bumps must regenerate the lock too. uv.lock records the project's own version, so bumping pyproject.toml and src/climate_lama/__init__.py without re-running uv lock leaves the lock out of sync — which uv lock --check catches in ci.yml, i.e. on the release gate, after the PR has already merged.

Backbone (API / worker)¶

Preflight — prove every credential the runbook depends on, before doing any work. Each check below is a dependency of a later step, and the GHCR PAT expires; running them first turns an end-of-pipeline unauthorized into a minute-one failure. HOST is the prod box (root@159.69.211.124, or your ssh-config alias):

HOST=root@159.69.211.124

ssh -o BatchMode=yes "$HOST" true                  # 1. SSH auth to the host works
gh auth status                                     # 2. GitHub CLI token still valid
ssh "$HOST" 'docker manifest inspect \
  ghcr.io/cortomaltese3/climate-lama:latest >/dev/null'   # 3. GHCR reachable + host still logged in

→ verify: all three exit 0. Any failure → stop and fix it before step 1.

(1) fails → SSH key/agent problem; you cannot deploy at all.
(2) fails → gh auth login (needed to watch CI and cut the release).
(3) fails with unauthorized/denied → the host's GHCR PAT has expired or was revoked: re-run the docker login ghcr.io in Registry Authentication with a fresh read:packages token. Check 3 downloads no layers, so it is safe to repeat.
Change it locally — branch, implement, write tests, lint (ruff + mypy) per CLAUDE.md. → verify: tests green locally.
PR → CI → merge to main — open a PR; CI must be green before it merges. → verify: CI green, PR merged.
Cut a release tag — this is the step that builds and publishes the images:
```
git tag v0.5.0 && git push origin v0.5.0
```
release.yml builds …/climate-lama:v0.5.0 and …/climate-lama-worker:v0.5.0 and pushes them to GHCR. → verify: the release workflow is green and the tag exists in GHCR.
Deploy — primary path: dispatch the deploy workflow (see Automated deploy); it does steps 4–5 for you, including the migration:
```
gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0
```
→ verify: the run is green (it health-gates and identity-checks itself).

Fallback — by hand on the host (no git pull, no rebuild). Note this path does not refresh the compose files — only the workflow does that (see Compose file delivery) — so if the release changed the topology, copy them across yourself before rolling:

sed -i 's/^CLIMATE_LAMA_TAG=.*/CLIMATE_LAMA_TAG=v0.5.0/' .env.prod
docker compose $P pull api worker
docker compose $P up -d

→ verify: docker compose $P ps shows api/worker/impact-worker/live-worker healthy on the new tag. (pull api worker is enough for all of them: beat, impact-worker and live-worker share the worker image and the same pinned tag.) 5. Migrate — the workflow already did this; only needed on the manual fallback path (and only if the release added migrations). Run with the new image's alembic:

docker compose $P exec api alembic upgrade head

→ verify: alembic current == head.

Both paths roll the containers before migrating, so on a release that changes the schema live-worker and beat come up against the old one and a scheduled tick can fire in that window. The tick no-ops on a missing relation, but restart the two once migrations are at head so they run against the schema they were shipped for:

docker compose $P restart live-worker beat

→ verify: docker compose $P logs live-worker --tail=20 shows a fresh boot banner listing queue live. 6. Verify the deployed identity — the deploy is not done until this passes. Run it from a checkout of this repo on your laptop (it resolves the tag locally):

git fetch --tags
./scripts/verify-deploy.sh v0.5.0
# non-default target: ./scripts/verify-deploy.sh v0.5.0 https://api.example.com

The script checks /health/ready, reads the git_sha baked into the running image from /v1/info, resolves v0.5.0 to a commit locally, and compares the two. → verify: it exits 0 and prints ✓ identity: … serves v0.5.0.

Exit 1 tells you which failure you have: host unreachable, service not ready, git_sha mismatch (prints served vs expected — the host is still running the old image: re-check CLIMATE_LAMA_TAG, that pull actually fetched, and that up -d recreated the container), git_sha unknown (the image was built before deploy identity existed, or without the build-arg), or the tag not existing locally (git fetch --tags).

A bare liveness curl is a weaker check and must not be used as the gate — a healthy old container passes it, which is exactly how an unbumped tag or a no-op pull goes unnoticed. Keep it only as a quick eyeball:

curl -s https://api.climate-lama.online/health/live   # {"status":"healthy"}
curl -s https://api.climate-lama.online/v1/info       # version, environment, git_sha, features

7. Rollback if needed — set CLIMATE_LAMA_TAG back to the previous tag, pull, up -d, and alembic downgrade if the schema moved. See Rollback Procedure.

There is a brief downtime during up -d (no rolling restart yet).

UI (climate-lama-ui)¶

Same shape, different repo and workflow:

1–2. Change + merge to main in climate-lama-ui (CI green). 3. Tag a release: git tag v0.2.0 && git push origin v0.2.0 → publish.yml builds …/climate-lama-ui:v0.2.0. (Map-tile URLs are baked at build time — tracked in #355.) 4. Deploy — primary path: dispatch the deploy workflow (see Automated deploy). A UI-only dispatch never touches api/worker and never runs migrations:

gh workflow run deploy-prod.yml -f ui_tag=v0.2.0

Fallback — by hand on the host:

sed -i 's/^UI_TAG=.*/UI_TAG=v0.2.0/' .env.prod
docker compose $P pull ui && docker compose $P up -d ui

5. Verify the deployed identity — the deploy is not done until this passes (issue #1315). The UI has no /v1/info-style endpoint, so a bare liveness curl is a weaker check and must not be used as the gate — a healthy old UI container returns 200 just as happily as a new one, which is exactly how a silent no-op UI deploy goes unnoticed. The automated dispatch above already runs this (deploy-prod.yml's Verify deployed identity (UI) step captures the Vite-hashed bundle filename at ${UI_BASE_URL}/ before the roll and fails if the same filename is still served afterwards). On the manual fallback path, mirror it by hand or with the script:

before="$(curl -s https://climate-lama.online/ | grep -o 'assets/index-[A-Za-z0-9_-]*\.js' | head -n1)"
# ... roll the container (the sed + pull + up -d above) ...
VERIFY_DEPLOY_PREV_UI_BUNDLE="$before" ./scripts/verify-deploy.sh <current-backbone-tag>

→ verify: it prints ✓ ui identity: … bundle changed … -> …. (verify-deploy.sh still requires a backbone tag argument — pass the one already deployed, unchanged, if this is a UI-only release; steps 1–4 pass trivially and step 5 is the one that matters here.)

Like the backbone check, this warns instead of failing — it does not count as a pass — whenever either read comes back empty: no bundle filename found before the roll (before above is empty; the automated workflow logs this from its own Capture UI bundle hash (pre-deploy) step) or none found after it. A green run means both reads succeeded and the filename changed; a warning means try again, not "it passed."

A bare liveness curl is still useful as a quick eyeball, just not as the gate:

curl -s https://climate-lama.online/health/live

Deploy identity (what `verify-deploy.sh` relies on)¶

Production deliberately lags main, so the only meaningful post-deploy invariant is served identity == the release tag being deployed. It is enforceable because the commit travels with the image:

release.yml builds the api/worker images with --build-arg GIT_SHA=<commit> (and the matching org.opencontainers.image.revision / …version labels).
docker/core.Dockerfile / docker/worker.Dockerfile turn that into ENV GIT_SHA, so it is readable in the container (docker compose $P exec api printenv GIT_SHA).
Settings.git_sha reads that env var and GET /v1/info returns it next to version and environment.
scripts/verify-deploy.sh compares it with git rev-list -n1 <tag>.

Images built without the build-arg (any local docker compose build, or any image published before this was wired up) report git_sha: "unknown". That is not an error at runtime — but verify-deploy.sh treats it as unverifiable and exits 1, on purpose. Do not set GIT_SHA in .env.prod: compose's env_file would override the baked value and the check would validate a hand-typed string instead of the image.

:latest is mutable (rebuilt on every push to main). Always pin a tag in .env.prod so production is reproducible and roll-back-able. The engine (climate-lama-engine) ships inside the worker image — it has no separate release/deploy step; bumping it is a backbone dependency change + new backbone tag.

UI identity is checked differently (issue #1315). The UI has no /v1/info-style endpoint, and a climate-lama-ui release tag lives in a sibling repo, so nothing in this repo's git history can resolve it to an "expected" value the way a backbone tag resolves to a commit. The check is therefore relative, not absolute: capture the Vite-hashed bundle filename served at the UI's apex (assets/index-<hash>.js) before the roll, and fail if the exact same filename is still served after it.

deploy-prod.yml's Verify deployed identity (UI) step does this automatically — Capture UI bundle hash (pre-deploy) runs immediately before Deploy UI and stamps the value into $GITHUB_ENV for the later step to compare against.
scripts/verify-deploy.sh mirrors it for the manual fallback path, gated on VERIFY_DEPLOY_PREV_UI_BUNDLE (the filename captured before the roll) the same way the CLIMADA sidecar check below it is gated on its own precondition: with that variable unset, the UI check makes no network call at all and just prints a skip reason, so the script's documented <expected-tag> [base-url] invocation behaves exactly as before this check existed. Set it to opt in, along with VERIFY_DEPLOY_UI_BASE_URL (default https://climate-lama.online) if the UI is not on the default domain.
A rebuild that produces a byte-identical bundle is the one case this cannot catch — a stronger /version.json-stamped variant is noted in issue #1315 as a possible follow-up, not implemented here.

Prerequisites¶

Production deployment uses Docker Compose; the live host terminates TLS with the shared Caddy (the bundled nginx is the portable alternative).

Docker ≥ 24.0 (docker --version)
Docker Compose v2 (docker compose version)
A server with ports 80 and 443 open
A domain name pointing to the server (for Let's Encrypt TLS)

Registry Authentication¶

The api/worker images (ghcr.io/cortomaltese3/climate-lama and …-worker) are private on GHCR, so the host must authenticate to the registry once before it can pull. This is a one-time setup per host — Docker caches the credential in ~/.docker/config.json and reuses it for every later pull, including updates. (CI is unaffected: the smoke workflows build images from source and never pull from GHCR.)

Create a GitHub Personal Access Token (classic) with the single read:packages scope: GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) → Generate new token (classic). Name it e.g. ghcr-pull-hetzner, set an expiry, check only read:packages, and copy the ghp_… value.

Log in on the host, as the same user that runs docker compose (use root / sudo if you deploy with sudo docker compose):

echo "ghp_xxxxxxxxxxxx" | docker login ghcr.io -u CortoMaltese3 --password-stdin
chmod 600 ~/.docker/config.json   # the token is stored base64, not encrypted

Verify (once the target release tag is published — see release process):
```
docker pull ghcr.io/cortomaltese3/climate-lama:v0.4.0
```

Token rotation¶

The PAT expires on the date you chose. Set a reminder to regenerate it and re-run the docker login above before it lapses — otherwise the next update deploy fails with unauthorized on docker pull, on a host you rarely touch. See ADR-038.

The step-by-step procedure — mint, log in, verify the new token can actually read all three private repositories, then revoke the old one — is docs/ops/secret-rotation.md §1. That runbook also covers the prod-host deploy keypair, the application secrets in .env.prod (DB passwords, MinIO credentials, APP_SECRET_KEY, API_KEY_PEPPER) and the SDK_SMOKE_* repo secrets.

TLS Certificates¶

Self-signed (development / smoke-test)¶

Generate a self-signed cert into nginx/certs/:

mkdir -p nginx/certs
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout nginx/certs/server.key \
  -out nginx/certs/server.crt \
  -subj "/CN=localhost"

Let's Encrypt (production)¶

Install certbot on the host.
Stop nginx if running: docker compose -f docker-compose.prod.yml stop nginx

Obtain a certificate:

certbot certonly --standalone -d your-domain.example.com

Copy the issued files into nginx/certs/:

cp /etc/letsencrypt/live/your-domain.example.com/fullchain.pem nginx/certs/server.crt
cp /etc/letsencrypt/live/your-domain.example.com/privkey.pem   nginx/certs/server.key

Restart nginx: docker compose -f docker-compose.prod.yml start nginx

Renew manually when needed (certbot renew), then repeat step 4-5. Automate with a cron job:

0 3 * * * certbot renew --quiet && \
  cp /etc/letsencrypt/live/<domain>/fullchain.pem /path/to/project/nginx/certs/server.crt && \
  cp /etc/letsencrypt/live/<domain>/privkey.pem   /path/to/project/nginx/certs/server.key && \
  docker compose -f /path/to/project/docker-compose.prod.yml restart nginx

First Deploy¶

Clone the repository onto the server and cd into it. The UI is a separate deployable in the climate-lama-ui repo; it consumes this backbone's API over HTTP and is deployed independently (see that repo's own deployment instructions).
Create the environment file:
```
cp .env.example.prod .env.prod
chmod 600 .env.prod
# Edit .env.prod — fill in all secrets, set CORS_ORIGINS to your domain,
# and set CLIMATE_LAMA_TAG to the release you want to deploy.
```
The host's /opt/climate-lama/.env.prod is the structural point of truth for which keys exist and in what order; .env.example.prod mirrors it, and the dev .env.example mirrors the same 33-key spine with dev values, so the two templates diff line-for-line. tests/test_env_template_parity.py fails when a key is added to one template and not the other. Add a key to .env.prod on the host and it belongs in both templates too.

.env.prod and its .env.prod.bak.* backups are mode 0600 on the host (root-only): they hold the DB password, the API-key pepper and the object-store credentials in cleartext.

The template defaults the MINIO_* block to Hetzner Object Storage (see Hetzner Object Storage). The prod compose ships no embedded object store — MINIO_ENDPOINT must point at a real S3-compatible bucket. For a fully self-contained eval stack use the dev docker-compose.yml, which bundles MinIO.

Generate TLS certificates (see TLS Certificates above).
Start all services:
```
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d
```
Prod pulls pre-built, CI-published images from GHCR — it never builds from source. The images are private, so the host must be authenticated to GHCR first (one-time; see Registry Authentication above). The api and worker services are pinned to :${CLIMATE_LAMA_TAG}; compose fails fast if that variable is unset. The api/worker images are published by release.yml on every v* tag — make sure the tag in CLIMATE_LAMA_TAG has actually been released before deploying. See ADR-038.

Run database migrations:

docker compose -f docker-compose.prod.yml exec api alembic upgrade head

Smoke test:
```
curl -k https://<host>/health/live
# Expected: {"status": "healthy"}
```
Remove -k once you have a valid certificate. Then confirm which build is serving, not just that something is (see Deploy identity):
```
./scripts/verify-deploy.sh v0.4.0 https://<host>
```
Provision an administrator — a fresh deployment has no admin until you create one. See Environment bootstrap (organization + admin) for the one-shot path (scripts/bootstrap_env.py), or Bootstrapping the first platform admin (#828) for the repeatable scripts/create_user.py path this production deployment actually uses.

Database roles and row-level security¶

Status: APPLIED on production 2026-08-04. api, worker and beat all connect as the unprivileged climate_lama_app role, so RLS is enforced rather than inert; the maintenance lane keeps the privileged role for the two jobs that need it. Closed on #556, which carries the preflight matrix and the post-flip evidence.

Until that date the flip described below was not actually performable from .env.prod — the compose file hardcoded the runtime DSNs and silently overrode the file, so following the old procedure produced a healthy stack still running as the superuser. APP_DATABASE_URL / APP_DATABASE_SYNC_URL exist to fix that; see the callout under "The two-role model". The procedure below is kept as the reference for a fresh host — it is what a new deployment must still do.

The two-role model¶

Row-level security is enabled and forced on every org-scoped table (migration 0022 and its successors), with one policy shape everywhere:

CREATE POLICY org_isolation ON <table>
  USING (org_id = current_setting('app.current_org_id', true)::uuid)

There is exactly one GUC (app.current_org_id) and no bypass GUC, no admin policy, no WITH CHECK clause — because the policy is FOR ALL without one, USING also gates INSERT/UPDATE. Cross-org admin surfaces work today because the tables they touch (organizations, users, api_keys, refresh_tokens, audit_events, usage_by_period) are deliberately exempt from RLS, not because anything bypasses a policy.

None of that has any effect while the application connects as a SUPERUSER role with BYPASSRLS, which is what POSTGRES_USER (the cluster bootstrap role) is. Hence two roles:

Role	Attributes	Used by
`climate_lama` (`POSTGRES_USER`)	superuser, owns the schema	Alembic migrations, `scripts/bootstrap_env.py`, the backfills and ingest scripts, `pg_dump`/`pg_restore`, psql inspection, and the maintenance lane below

"Alembic migrations" is not automatic — it is arranged. The deploy workflow runs docker compose exec api alembic upgrade head, i.e. inside the api container, which is precisely the process the flip repoints at climate_lama_app. Migrations are DDL and climate_lama_app holds no ownership and no CREATE ON SCHEMA, so nothing would keep them on the owner by itself. migrations/env.py therefore resolves its DSN from MAINTENANCE_DATABASE_SYNC_URL first, falling back to DATABASE_SYNC_URL (see src/climate_lama/db/migration_url.py). That is why step 2 of the flip — setting the maintenance DSN to the owner — is not optional and must come before step 3: it is what keeps migrations, the usage_by_period refresh and the result_cache purge on the owner once the app DSN moves. Skip it and the flip looks fine, then the next deploy fails at alembic upgrade head. | climate_lama_app | LOGIN NOSUPERUSER NOBYPASSRLS NOCREATEDB NOCREATEROLE | the api, worker and beat runtime DSNs |

The application DSN is pure configuration (DATABASE_URL / DATABASE_SYNC_URL, read by src/climate_lama/config.py), so the flip is an env change plus a restart — no image rebuild.

Which variable actually takes effect — read this before editing anything. docker-compose.prod.yml declares DATABASE_URL and DATABASE_SYNC_URL in the environment: block of api, worker and beat, and Docker Compose gives environment: precedence over env_file:. So a DATABASE_URL= / DATABASE_SYNC_URL= line in .env.prod is inert — it is read from the file and then overridden. Editing it changes nothing, the stack restarts healthy, and you would conclude RLS is enforced while the app is still connecting as the cluster superuser. That is the failure this section used to walk an operator straight into. Both lines were deleted from the host's .env.prod on 2026-08-26 and .env.example.prod does not ship them; do not add them back.

The live knobs are APP_DATABASE_URL and APP_DATABASE_SYNC_URL, which Compose interpolates into those same entries:
DATABASE_URL: ${APP_DATABASE_URL:-postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}}
DATABASE_SYNC_URL: ${APP_DATABASE_SYNC_URL:-postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}}
Unset or empty → the owner DSN derived from POSTGRES_*, i.e. exactly the behaviour every existing deployment has today. Set → that value wins. The names are deliberately distinct from the legacy inert lines so "the variable I edited" and "the variable that took effect" can never diverge.

POSTGRES_USER / POSTGRES_PASSWORD are not a substitute: they initialize and authenticate the postgres container itself and drive scripts/backup.sh / scripts/restore.sh. Repointing them at climate_lama_app breaks the database and the backups, not just the app.

MAINTENANCE_DATABASE_SYNC_URL is different again: it has no environment: entry in any service, so env_file: .env.prod supplies it directly and setting it in .env.prod does take effect.

The two runtime lanes¶

Everything the API and the worker do falls into one of exactly two lanes.

Org-scoped lane — the default, and where all but two tasks live. Every session that acts on behalf of one organization stamps SELECT set_config('app.current_org_id', :org_id, true) on its transaction immediately after opening it, via db/rls.py::set_org_guc / set_org_guc_sync. Two properties of that call are load-bearing:

the org id is a bound parameter, never interpolated (a SET LOCAL statement takes no bind parameters — asyncpg would send = $1 and Postgres would raise a syntax error; tests/lint_no_set_local_org_guc.py pins this);
is_local => true makes the setting transaction-local, so it cannot leak onto the next tenant's checkout of a pooled connection — and, by the same token, it is discarded at every COMMIT. Code that commits and keeps working on the same session must set it again.

Work that is cross-org in aggregate but decomposable per org is written as a loop of org-scoped transactions: the org list comes from the RLS-exempt organizations table (db/rls.py::list_org_ids_sync) and each org's work runs under its own GUC. That is how the ingest watchdog sweeps, surface retention, and the "which org owns this row?" lookups in convert_to_cog / build_dataset_cog / download_to_spaces (db/rls.py::resolve_owning_org_sync) are expressed.

Maintenance lane — narrow, explicit, two users. Reserved for work that cannot be expressed per org at all. It is configured by MAINTENANCE_DATABASE_SYNC_URL; when that is unset it falls back to DATABASE_SYNC_URL, which is exactly today's behaviour, so dev, tests and the current production deployment are unaffected until you flip.

Maintenance-lane user	Why it cannot be org-scoped
`worker/tasks.py::refresh_usage_view` → `UsageEventRepository.refresh_view`	`REFRESH MATERIALIZED VIEW` needs ownership of `usage_by_period`, which `climate_lama_app` deliberately lacks — and granting it ownership would not help, because an owner without `BYPASSRLS` refreshes the view to zero rows (`usage_events` is FORCE-RLS)
`worker/tasks.py::purge_orphan_cache_entries`	`result_cache` carries no `org_id`, so "orphaned" is only definable against the union of every org's `impact_results`; scoped to one org, every other org's cache rows would look orphaned

That table is the whole list. Anything else needing the privileged role is a bug, not a third lane. purge_orphan_cache_entries additionally keeps its emptiness guard: if impact_results reads empty against a non-empty result_cache it refuses to purge and logs an error, which is what makes a misconfigured maintenance DSN safe rather than destructive.

Grant set¶

scripts/create_app_role.py applies exactly this, idempotently:

GRANT CONNECT ON DATABASE <db>                       TO climate_lama_app;
GRANT USAGE   ON SCHEMA public                       TO climate_lama_app;
GRANT SELECT, INSERT, UPDATE, DELETE
      ON ALL TABLES IN SCHEMA public                 TO climate_lama_app;
GRANT USAGE, SELECT
      ON ALL SEQUENCES IN SCHEMA public              TO climate_lama_app;
GRANT SELECT ON <each materialized view in public>   TO climate_lama_app;
ALTER DEFAULT PRIVILEGES FOR ROLE climate_lama IN SCHEMA public
      GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO climate_lama_app;
ALTER DEFAULT PRIVILEGES FOR ROLE climate_lama IN SCHEMA public
      GRANT USAGE, SELECT ON SEQUENCES               TO climate_lama_app;

Deliberately not granted: ownership, TRUNCATE, REFERENCES, TRIGGER, CREATE ON SCHEMA. The ALTER DEFAULT PRIVILEGES pair is what makes future migrations' tables reachable without re-running the script — but Postgres does not apply default privileges to materialized views, so re-run the script after any migration that adds one.

Running the bootstrap¶

Run it as the privileged role (the current DATABASE_SYNC_URL), before changing anything. The default is a dry run that prints every statement.

# on the prod host, from /opt/climate-lama
read -rs -p "new app-role password: " APP_PW; echo
docker compose -f docker-compose.prod.yml exec -T \
  -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" api \
  python scripts/create_app_role.py            # dry run — prints the statements

docker compose -f docker-compose.prod.yml exec -T \
  -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" api \
  python scripts/create_app_role.py --apply

Re-running is safe: it re-asserts the attributes, re-applies the grants and rotates the password. Verify:

SELECT rolname, rolsuper, rolbypassrls, rolcanlogin
  FROM pg_roles WHERE rolname LIKE 'climate_lama%';
-- climate_lama_app must show f | f | t

The env flip¶

Applied on production 2026-08-04 (#556); this is the reference procedure for a fresh host. Do these in order. Step 0 is a precondition; steps 1–2 are safe on their own; step 3 is the flip.

0. Assert the deployed image actually contains the code this flip depends on. This flip is not self-contained — it only works if the running image carries the commits that teach the application about the two-role model. Run:

git fetch origin
for ref in 05b3b7b 504132f 590ad17 c6c8cab; do
  scripts/assert_deployed_contains.sh "$ref" || echo "BLOCKED: $ref is not deployed"
done

commit	PR	what the flip needs it for
`05b3b7b`	#576	parameterised `set_config` org GUC
`504132f`	#577	non-superuser role support, `resolve_owning_org_sync`, seed guard
`590ad17`	#581	org-scoped GUC lane + maintenance lane in the worker
`c6c8cab`	#633	Alembic on the maintenance lane

If any line reports BLOCKED, stop and deploy first. This is not hypothetical: on 2026-08-04 the flip was applied while prod ran v0.6.0, which predated all four. MAINTENANCE_DATABASE_SYNC_URL was set on the host but did not exist anywhere in the deployed source, so the maintenance lane was inert, every scheduled task ran on the runtime DSN as the unprivileged role, and the ingest chord stamped no org GUC (#638).

The generic lesson — this is not specific to the RLS flip. A configuration change that depends on application code needs a deployed-code assertion, not just a health check. Every check run after that flip was green (/health/*, /v1/info, an unauthenticated 401, alembic current) because every one of them exercises a read path that the old image gets right. A health check tells you the container is alive; it does not tell you the container knows about the setting you just added. Any runbook step of the shape "set an env var and restart" gets a step 0 like this one.

1. Create the role — scripts/create_app_role.py --apply, as above, and confirm pg_roles shows climate_lama_app as f | f | t.

2. Point the maintenance lane at the privileged user, before flipping the app DSN. Add one line to .env.prod — spelled out, not interpolated, because env_file values are literal strings and ${...} inside them is not expanded:

MAINTENANCE_DATABASE_SYNC_URL=postgresql+psycopg2://<POSTGRES_USER>:<POSTGRES_PASSWORD>@postgres:5432/<POSTGRES_DB>

Setting it while the app DSN is still the privileged role is a no-op — both point at the same user — which is exactly why it goes first. Three things must already be on the privileged lane at the moment the app DSN changes underneath them:

the nightly refresh_usage_view and purge_orphan_cache_entries;
alembic upgrade head, which the deploy workflow runs inside the api container. migrations/env.py prefers MAINTENANCE_DATABASE_SYNC_URL over DATABASE_SYNC_URL for exactly this reason. Leave it unset and the flip itself succeeds, but the next deploy dies on the first migration that issues DDL — the failure lands one deploy after its cause.

Restart api, worker and beat so they pick it up, and confirm it landed in all three:

docker compose -f docker-compose.prod.yml --env-file .env.prod up -d api worker beat
for svc in api worker beat; do
  docker compose -f docker-compose.prod.yml --env-file .env.prod exec -T "$svc" \
    printenv MAINTENANCE_DATABASE_SYNC_URL
done

Then prove migrations still resolve to the owner — before the app DSN moves, so a wrong answer here costs nothing:

docker compose -f docker-compose.prod.yml --env-file .env.prod exec -T api \
  alembic current      # must print the current revision and `(head)`

3. Flip the app DSN — add APP_DATABASE_URL / APP_DATABASE_SYNC_URL to .env.prod. This is the whole flip; docker-compose.prod.yml needs no edit, and neither does anything else in the repo:

APP_DATABASE_URL=postgresql+asyncpg://climate_lama_app:<APP_PW>@postgres:5432/<POSTGRES_DB>
APP_DATABASE_SYNC_URL=postgresql+psycopg2://climate_lama_app:<APP_PW>@postgres:5432/<POSTGRES_DB>

Leave POSTGRES_USER / POSTGRES_PASSWORD alone — they stay the owner, because they also initialize the postgres container and drive the backup scripts. Do not bother editing the legacy DATABASE_URL / DATABASE_SYNC_URL lines: as explained above they are inert under this compose file.

Both variables must be set together, and both must name the same role — the async DSN serves the api, the sync DSN serves the worker, and a half-flip leaves the two runtimes disagreeing about which role they connect as.

Verify the render before restarting anything. config resolves the file exactly the way up will, at zero risk:

docker compose -f docker-compose.prod.yml -f docker-compose.hetzner.yml \
  --env-file .env.prod config | grep -E 'DATABASE_(SYNC_)?URL'
# every api/worker/beat DATABASE_URL and DATABASE_SYNC_URL must now read
# climate_lama_app, and MAINTENANCE_DATABASE_SYNC_URL must still read the owner

Also confirm any one-shot bootstrap service is either absent from this deployment's compose file (production's — no CL_BOOTSTRAP_ADMIN_* here at all, see Environment bootstrap) or, if present, points at a privileged DSN. scripts/bootstrap_env.py refuses under the unprivileged app role with exit code 2 — a loud refusal rather than a half-bootstrapped organization — the same guard the retired demo seeder grew after issue #556.

4. Restart only the app services — the database is untouched:

docker compose -f docker-compose.prod.yml up -d api worker beat
curl -fsS https://api.climate-lama.online/health/ready     # exercises a real DB round-trip

5. Verify. Four checks, in this order:

# a) the role really is unprivileged, and the app connects as it
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname LIKE 'climate_lama%';"
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT usename, count(*) FROM pg_stat_activity WHERE datname = current_database()
     AND usename IS NOT NULL GROUP BY 1;"
# api/worker connections must now show up under climate_lama_app

b) one ingest end to end. Upload a small hazard through POST /v1/hazards/ingest and poll GET /v1/hazards/ingest-jobs/{id} to succeeded. This is the chord's whole session surface (stage → validate → plan → write → aggregate → commit) in one shot; a missed GUC shows up as a job that never leaves its state or a dataset committed with zero centroids.
c) one matrix run. build_reference_surfaces with a 2×2 matrix — the task that exposes the "GUC dies at COMMIT" failure, since only cells after the first would misbehave.
d) the nightly tasks. Trigger them by hand rather than waiting for beat, and read the logs:

docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "from climate_lama.worker.tasks import refresh_usage_view, purge_orphan_cache_entries; \
             print(refresh_usage_view.apply().get(), purge_orphan_cache_entries.apply().get())"

purge_orphan_cache_entries returning {"skipped": true} means the guard fired — the maintenance DSN is not privileged. Fix step 2 before proceeding. Then confirm GET /v1/admin/usage still returns non-empty aggregates, which is the check that the matview refresh saw rows. - e) migrations still reach the owner. The one check whose failure would otherwise surface at the next deploy rather than now:

docker compose -f docker-compose.prod.yml exec -T api alembic current
# must print the revision and `(head)` — a permission error here means
# MAINTENANCE_DATABASE_SYNC_URL is missing from the api service (step 2)
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT usename, application_name FROM pg_stat_activity
     WHERE datname = current_database() AND usename IS NOT NULL;"

alembic current is read-only, so it passes under either role — it proves connectivity, not privilege. For the real proof, run the next deploy's alembic upgrade head and watch it succeed, or confirm the api container's MAINTENANCE_DATABASE_SYNC_URL names the owner (step 2's printenv loop).

Rollback is the inverse and takes one restart: comment out (or blank) the APP_DATABASE_URL / APP_DATABASE_SYNC_URL lines in .env.prod — the :- defaults then resolve back to the POSTGRES_* owner DSN — and up -d api worker beat again. MAINTENANCE_DATABASE_SYNC_URL can stay — it then simply names the same role as the app DSN. The climate_lama_app role can be left in place; it is inert when nothing connects as it.

Pre-flip blockers — cleared¶

The blocker list audited on 2026-08-03 has been implemented; each item below is now the shipped behaviour rather than a warning. Every one of them was a silent behaviour change under the new role, not a permission error, which is why they gated the flip.

Was	Now
The ingest chord opened ~20 sessions with no GUC across `worker/ingest/{pipeline,stage_source,validate_source,plan_chunks,write_chunk,aggregate_and_commit}.py`, all touching FORCE-RLS `ingest_jobs` / `ingest_chunks`	Every one of those sessions stamps the org GUC on its transaction. `write_chunk`'s chunk helpers now take `org_id` explicitly — `IngestChunkRepository` keys on `ingest_job_id` alone, so nothing else carried the org
`worker/ingest/watchdog.py` (`_find_wedged_jobs`, `_find_wedged_writing_jobs`) scanned cross-org with no GUC → permanent no-op	Both candidate scans iterate `organizations` and run one org-scoped transaction per org
`build_reference_surfaces` set the GUC once, then COMMITted inside its loop → every cell after the first ran GUC-less	The GUC is re-set at the top of every matrix-cell iteration
Unfiltered "peek" reads on `hazard_datasets` in `convert_to_cog` / `build_dataset_cog`	`db/rls.py::resolve_owning_org_sync` locates the owning org by probing each org's own scope; the follow-up row read is then org-scoped
`REFRESH MATERIALIZED VIEW CONCURRENTLY usage_by_period` needs ownership, and a non-`BYPASSRLS` owner would refresh to zero rows	Moved to the maintenance lane (`MAINTENANCE_DATABASE_SYNC_URL`)
`purge_orphan_cache_entries` would empty `result_cache` wholesale	Moved to the maintenance lane, keeping the #577 emptiness guard
`CL_SEED_DEMO=true` ran the cross-org seeder inside the API process as the app role	Moot: the demo layer — seeder, fixtures, `CL_SEED_DEMO` and `CL_DEMO_USER_PASSWORD` — was deleted whole by #892 (ADR-077), and both keys were removed from the host's `.env.prod` on 2026-08-26. There is no seeding code path left to guard

Found beyond that list and fixed here:

download_to_spaces had the same unfiltered peek on downloads (FORCE-RLS since migration 0041) that convert_to_cog had; the download would have looked deleted. Same resolve_owning_org_sync treatment.
The climate_lama_surface_* ORG-plane gauges were computed with one unscoped rollup at the end of the retention sweep, which under the new role would have reported only the last org visited. refresh_surface_gauges now takes the sweep's org list and sums the ORG plane one RLS scope at a time; the org-less REFERENCE plane is still read once.

Everything else audited clean: login/refresh/API-key paths, /health*, core/audit.py, core/notifications.py, api/v1/admin/*, worker/dataset_polling.py and worker/dataset_ingest.py (they touch pending_ingests, which carries no RLS), and db/repositories/org_download_allowlist_repository.py (which carries its own explicit org_id predicate as belt-and-braces).

Verifying isolation¶

tests/db/test_app_role_rls_isolation.py is the load-bearing proof: it creates its own throwaway NOSUPERUSER / NOBYPASSRLS role, seeds two orgs, and shows that unfiltered SELECTs against hazard_datasets / hazard_centroids return nothing without the GUC and only the caller's own rows with it. It skips when no PostgreSQL is reachable and runs for real in CI.

tests/test_worker/test_rls_lanes.py is its DB-free companion: it pins lane selection — which session factory each task opens and whether it stamps the GUC — which is the property that regresses when someone adds a new session to the worker. Add a case there when you add a session.

Environment bootstrap (organization + admin)¶

scripts/bootstrap_env.py is the supported path from empty, migrated database to somebody can log in (issue #889). It is content-free — one organization and one admin user, no datasets, no demo tenant — and it is what a clean clone, a new dev machine and a CI run all use. It does not replace the DB-role bootstrap above (create_app_role.py, "Running the bootstrap"), which provisions the Postgres role rather than the tenant.

# Non-interactive (compose / CI). Credentials come from the environment.
CL_BOOTSTRAP_ADMIN_EMAIL=you@your-org.example \
CL_BOOTSTRAP_ADMIN_PASSWORD='...' \
  uv run python scripts/bootstrap_env.py

# Interactive: prompts for anything not supplied.
uv run python scripts/bootstrap_env.py

# Add the analyst/viewer accounts the admin-gating check needs — same command,
# different email and role. Re-running is how the roster grows.
uv run python scripts/bootstrap_env.py --email analyst@your-org.example --role analyst

docker-compose.yml runs it as the bootstrap one-shot after migrations, so docker compose --profile fullstack up from a clean clone with CL_BOOTSTRAP_ADMIN_* set in .env yields an admin who can POST /v1/auth/login. The variables are documented in .env.example; there is no default email or password anywhere, and placeholder values (the change-me family, this repo's own example addresses, <angle-bracketed> templates) are refused rather than persisted.

Exit	Meaning
`0`	Bootstrapped, already bootstrapped (idempotent no-op), or skipped because no credentials were configured and stdin is not a TTY.
`1`	Misconfiguration — placeholder or policy-violating credentials, only half the credential pair set, unknown role, or the email already belongs to another org. Nothing written.
`2`	The connected DB role neither is a superuser nor has `BYPASSRLS`. Creating an org and then writing rows into it is cross-org work, so under `climate_lama_app` it would half-apply silently (issue #556).

Pass --require-credentials in CI to turn the exit-0 skip into a hard failure, so a misconfigured job fails at bootstrap rather than much later as an unexplained 401.

Production is already bootstrapped. The real tenant is northlane (issue #858), so .env.example.prod keeps CL_BOOTSTRAP_ADMIN_* commented out and docker-compose.prod.yml carries no bootstrap service. Provision or promote a production account with scripts/create_user.py instead — see "Bootstrapping the first platform admin (#828)".

Verifying a clean clone — the end-to-end walk¶

scripts/smoke_e2e.py walks the real user journey against a running stack and is both the go/no-go gate inside sdk-smoke.yml and the executable form of the demoable MVP slice in CLAUDE.md (issue #890). Nothing about it is demo machinery: it provisions through scripts/bootstrap_env.py, loads a real GADM boundary through scripts/load_admin_boundaries.py, ingests through the async ingest API, computes through the Celery queue, and asserts a non-empty result linked to a saved scenario.

Since issue #1045 the walk runs this once per country leg — Greece, then Cyprus, on the same org — rather than once overall. Each leg has its own GADM boundaries, region, reference raster, exposure CSV and ingested-row names, so the two never collide on a lookup. Once both legs have ingested, a cross-pin check proves the isolation is real: it asks GET /v1/risk/lookup — with no region filter — at each leg's own tile centre and asserts the answer names that leg's hazard dataset, never the other one's. That only holds because

1047's coverage rule restricts which datasets a coordinate can resolve to¶

those whose footprint actually covers it; a caller-supplied region hint would prove nothing.

Since issue #1029 the walk carries a third, optional leg on the orthogonal axis: an event replay of one real observed earthquake — the USGS ShakeMap PGA field of the 2020-10-30 Samos M7.0, vendored as tests/fixtures/reference_datasets/earthquake_event_pga.tif (#1028). It opens a live event, ingests that raster through the ingest API's single-event mode (#1020/#1021: an event block and a live_event_id instead of return periods), computes it on the deterministic-event lane against the Greek leg's own exposure dataset, and asserts a positive event loss with a non-empty per-asset vector, the event's own valid_at, the event lane's grade note, and no ead/aai anywhere in the payload. It is the only automated exercise the event lane gets end to end.

The event leg runs only when SDK_SMOKE_EVENT_TIF (or --event-tif) names a raster, and it needs LIVE_MODE_ENABLED=true on the api and worker services. Leave the variable unset and the walk is exactly the two-country walk above; set it against a stack with live mode off and the leg fails loudly rather than skipping — a gated leg that no-ops on a misconfiguration is a green run that proved nothing.

# 1. Clean clone, credentials in .env (CL_BOOTSTRAP_ADMIN_* — see above). Add
#    LIVE_MODE_ENABLED=true to .env for the event leg (omit it and the walk
#    runs the two country legs only), then:
docker compose --profile fullstack up -d --wait

# 2. Stage the tracked reference rasters where the worker can read them. The
#    ./data bind mount is gitignored, and the worker sees it as /data.
mkdir -p data
cp tests/fixtures/reference_datasets/river_flood.tif data/
cp tests/fixtures/reference_datasets/river_flood_cyprus.tif data/
cp tests/fixtures/reference_datasets/earthquake_event_pga.tif data/   # event leg only

# 3. Walk it. Compose's `bootstrap` one-shot already provisioned the account, so
#    point the walk's credentials at it and skip that stage. Drop the
#    SDK_SMOKE_EVENT_TIF line to walk the two country legs only.
SDK_SMOKE_EMAIL="$CL_BOOTSTRAP_ADMIN_EMAIL" \
SDK_SMOKE_PASSWORD="$CL_BOOTSTRAP_ADMIN_PASSWORD" \
SDK_SMOKE_ORG_SLUG="${CL_BOOTSTRAP_ORG_SLUG:-default}" \
SDK_SMOKE_EVENT_TIF=/data/earthquake_event_pga.tif \
ADMIN_LOADER_DATABASE_URL=postgresql://climate_lama:password@localhost:5433/climate_lama \
  uv run python scripts/smoke_e2e.py --skip-bootstrap

A pass prints one E2E_READY_GRC hazard_id=… exposure_id=… result_id=… line and one E2E_READY_CYP hazard_id=… exposure_id=… result_id=… line — plus, when the event leg ran, one E2E_READY_EVT_GRC live_event_id=… snapshot_id=… event_loss=… line, and, when the prod-shaped seed ran (--prod-shaped-dir), one E2E_READY_PROD_SHAPED portfolio_m3=… … ssp_hazard_id=… earthquake_hazard_id=… admin1_ids=… admin2_id=… line and one E2E_READY_GRC_FUTURE hazard_id=… result_id=… ead=… today_ead=… line — then exits 0; any other outcome exits 1 with the failing leg and stage named (e.g. [CYP] …, [EVT_GRC] …, [GRC_FUTURE] …). The future-cell leg (GRC_FUTURE.1-GRC_FUTURE.3, issue #1563) is demo slice step 3's "scenario × horizon cells": it computes SSP2-4.5 · 2050 by sending the GRC baseline's id with scenario and horizon_year (no haz_type, no region), the way Explore addresses a future cell, and passes only when the result names the seeded SSP dataset (hazard_id) and its EAD is above the Greek leg's Today EAD (today_ead; the SSP slice's bands are 1.5 × the baseline's). The Greek leg's stages map onto the MVP slice one-for-one — configure (ingest the hazard, upload the exposure, assign centroids, save the scenario — steps GRC.2-GRC.5 in the walk's own output), compute (GRC.6), see and save the result (GRC.7), and the map half of "see results rendered" as a real raster tile through the backbone tile proxy (GRC.8, issue #1002) — so a green walk is the slice's verification, and the UI check on top of it is "log in and open that scenario". The Cyprus leg repeats the same stages (CYP.2-CYP.8) under its own names and proves a second country coexists cleanly. The event leg's four stages (EVT_GRC.1-EVT_GRC.4) are the replay lane rather than the slice: open the watch, ingest the observed snapshot, compute it, assert the result.

The tile-render stage needs the titiler service, which a plain docker compose up starts. It is the only automated check anywhere that the raster chain (proxy → titiler → GDAL /vsis3/ → object store) actually serves pixels; see docs/architecture/raster-serving.md, "Automated coverage".

Two knobs matter when pointing it somewhere else:

--skip-bootstrap / --skip-boundaries drop the two stages that need direct database access. --skip-boundaries applies to every leg. With both passed the walk is pure HTTP and can be aimed at a remote stack with --base-url.
--region must name one of the boundaries --boundary loads. Writing any admin_boundaries row arms the region vocabulary (issue #721), so a mismatch would 422 the ingest — the walk checks the two against each other, for every leg, up front and refuses to start rather than failing minutes later. --region/--boundary/--hazard-tif/--exposure-csv all override the Greek leg only; the Cyprus leg's fixtures are fixed.
--event-tif (or SDK_SMOKE_EVENT_TIF) arms the event leg and is the only thing that does. It replays over the Greek leg, whose exposure it computes against, so --legs CYP with an event raster named is refused up front rather than asserting a positive loss against a footprint that contains none of that leg's assets. The stack must have LIVE_MODE_ENABLED=true; one without it fails the leg on its first call rather than skipping it.

Resetting a user's password¶

There is no scripts/reset_password.py — the supported entry points (scripts/bootstrap_env.py, scripts/create_user.py) only ever set a password at creation and never touch an existing user's, by design (see bootstrap_environment()'s docstring on password). To change one, reset it via psql, substituting the target email and a real password:

HASH=$(docker compose exec -T api python -c \
  "from climate_lama.core.security import hash_password; print(hash_password('<new-password>'))")
docker compose exec postgres psql -U climate_lama -d climate_lama \
  -c "UPDATE users SET password_hash = '$HASH' WHERE email = '<user-email>';"

On production, run the equivalent against the prod compose file and the privileged climate_lama role (see Provisioning an org + user below for the docker compose $P shorthand).

Wiping and re-provisioning a dev stack from scratch¶

docker compose down -v   # drops the postgres_data volume
docker compose up -d     # re-creates containers, runs migrations, and re-runs the
                          # `bootstrap` one-shot against the now-empty database

With CL_BOOTSTRAP_ADMIN_* set in .env this reaches "someone can log in" with no further steps — see Environment bootstrap (organization + admin) above. There is no seeded content to re-seed: every dataset a fresh stack needs comes from a real ingest, e.g. Verifying a clean clone — the end-to-end walk above, or scripts/ingest_scenario_hazards.py for the full scenario-conditioned catalog (see docs/quickstart/orsa-walkthrough.md).

Update Flow¶

Deploying a new release is steps 3–6 of Release → Production: cut a tag (CI publishes the image), bump CLIMATE_LAMA_TAG (or UI_TAG) in .env.prod, pull, up -d, then alembic upgrade head if the schema moved. No git pull or rebuild on the host — the image is already built by CI. Rolling restart is not configured yet, so there is a brief downtime during up -d.

The primary path is the deploy workflow below — it performs exactly those steps over SSH. The by-hand commands stay documented as the fallback for when the workflow is unavailable (missing secrets, GitHub outage, host firewall).

Automated deploy (GitHub Actions)¶

.github/workflows/deploy-prod.yml applies a release to the live host. It does not build images — release.yml (backbone) and the UI repo's publish.yml already did that; this only points .env.prod at an already-published tag and rolls the containers.

Trigger: workflow_dispatch only. Deploying to prod is an explicit, auditable click — never a side effect of a push, tag, or release. Do not add another trigger.

gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0                    # backbone only
gh workflow run deploy-prod.yml -f ui_tag=v0.2.0                          # UI only
gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0 -f ui_tag=v0.2.0   # coordinated

Input	Effect
`backbone_tag`	bump `CLIMATE_LAMA_TAG` → `pull api worker` → `up -d --remove-orphans` → `alembic upgrade head` → assert `alembic current` is at head
`ui_tag`	bump `UI_TAG` → `pull ui` → `up -d --no-deps ui`. No migrations.

Both are optional but at least one is required — an empty dispatch fails immediately with a clear message. The two deployables are versioned separately, so a UI-only deploy must not disturb the backbone: the workflow passes --no-deps and then asserts the api/worker container IDs are byte-identical either side of the roll.

What it guarantees, in order:

Tag inputs are charset-restricted ([A-Za-z0-9][A-Za-z0-9._-]{0,63}) before they reach a remote shell — they are interpolated into SSH commands.
GHCR preflight runs before any .env.prod edit. docker manifest inspect (on the host, no layers downloaded) proves every requested image exists. The ordering is the point: a half-applied env bump pointing at an image that was never published leaves prod pinned to something unpullable.
Both compose files are uploaded to the host before the containers roll, so the topology prod runs is the topology main describes — see Compose file delivery below.
.env.prod is backed up to .env.prod.bak.<UTC timestamp> before every edit (last 10 kept), and the bump is read back and verified.
concurrency: deploy-prod (queue, never cancel) serialises dispatches so two runs cannot interleave mid-edit on the host.
Orphaned containers are reconciled — the backbone roll is up -d --remove-orphans, so a service dropped from the compose file also disappears from the host. See Orphaned containers below for why this is not optional.
Health gate — backbone: /health/live must report healthy; UI: the apex must return HTTP 200. Both retry for up to 3 minutes.
Deploy identity — a healthy old container passes the health gate just as happily as a new one, so that alone cannot prove either tag landed (see Deploy identity):
backbone — the git_sha baked into the running image is compared with the tag's commit. A definite mismatch fails the run; an image that predates the GIT_SHA build-arg only warns.
UI (issue #1315) — the Vite-hashed bundle filename captured immediately before the UI rolls is compared with the same read afterwards; an unchanged filename fails the run. Either read coming back empty (unreachable, non-200, or the asset naming changed) only warns — it is not treated as "changed" and does not pass the run.
Superseded images are pruned (issue #579) once the gates above have passed — see Image retention below.
On failure, docker compose ps and the last 200 log lines for api/worker/ui are printed into the job output.

Compose file delivery¶

The workflow delivers docker-compose.prod.yml and docker-compose.hetzner.yml to /opt/climate-lama/; it does not merely assert they are there. Before issue #998 it only checked for their presence, so a merged topology change reached production solely if a human remembered to copy it across — and the host's docker-compose.prod.yml drifted from main for ~18 days without any deploy noticing.

.env.prod is not uploaded. It holds secrets, is correctly absent from the repo, and stays hand-managed on the host.

Which ref the compose files come from. They come from the ref the workflow was dispatched on (the actions/checkout at the top of the job), not from backbone_tag / ui_tag. The images are tag-pinned; the topology is ref-pinned. In the normal case — deploying the newest tag from main — the two coincide. They diverge when you roll back to an older tag from main, which ships main's topology against the older image. When you want the tag's own topology, dispatch on the tag:

gh workflow run deploy-prod.yml --ref v0.17.0 -f backbone_tag=v0.17.0

How the upload behaves, per file:

Runs after the GHCR preflight (a dispatch naming unpublished tags fails before it rewrites the host's topology) and before every docker compose ... up -d, so the roll is what consumes the new files.
The file is scp'd to <name>.incoming beside the live copy and then mv'd into place. mv within one filesystem is atomic, so a docker compose command racing the deploy can never read a half-written topology.
A sha256 computed on the runner is re-checked on the host, and both files are verified before either is installed — the base file and its overlay are one topology, so a truncated overlay must not leave the base already swapped. Any mismatch aborts the deploy with both live files untouched and nothing left staged.
If the host copy is already byte-identical, nothing is written beyond a mode normalisation — repeated deploys do not mint redundant backups.
Otherwise the previous copy is backed up to <name>.bak.<UTC timestamp> (cp -p, then an explicit chmod), and the last 10 are kept. This is the .env.prod.bak.* convention with one deliberate difference each way: the mode is 644, not 600 (compose files carry no secrets — what must not fork is setting the mode explicitly rather than letting cp -p inherit it, which is how the world-readable .env.prod backups of #999 were minted), and the prune orders backups by name, not by ls -t (cp -p preserves the source mtime, so a freshly minted backup can sort older than its predecessors and be the one deleted; the .bak.<UTC timestamp> name is the reliable key).
The installed mode is 644. docker-compose.prod.yml sat at 0755 on the host — a leftover of the hand-run sync_compose.sh from PR #632 — and the first upload normalises it.

tests/test_deploy_workflow_compose_upload.py is the executable guard: it parses the workflow and fails if the upload step is removed, reordered after the roll, made non-atomic, or stripped of its backup.

Orphaned containers¶

Removing a service from docker-compose.prod.yml does not remove its container from the host. Compose only ever acts on services it can see in the merged files, so a deleted service becomes an orphan: still running, still restarting on boot (restart: unless-stopped), still holding whatever connections it held — and now invisible to every pull, up, and down you run, because none of them enumerate it.

This is not hypothetical. martin was retired in #562 and its tables dropped in migration 0067; the container went on running for seven weeks. It surfaced only after the #556 RLS flip, when a connection census showed the retired service holding 17 connections as the cluster owner while api, worker and beat held 2 as the unprivileged role — a service with no consumer had become the single largest holder of BYPASSRLS connections to the production database, and the only application process RLS did not constrain (#634).

The deploy workflow now passes --remove-orphans on the backbone roll, which closes this going forward. The flag is scoped to the compose project label (com.docker.compose.project=climate-lama), so a co-tenant stack on the same host is out of reach by construction — the shared Caddy runs as project caddy and is not a candidate. The UI-only path deliberately keeps up -d --no-deps ui with no sweep, so a UI deploy still cannot disturb anything else.

To audit the host by hand — every container carrying the project label, against the services the compose files actually declare:

cd /opt/climate-lama
for c in $(docker ps -a --format '{{.Names}}'); do
  printf '%-28s %s\n' "$c" \
    "$(docker inspect -f '{{index .Config.Labels "com.docker.compose.project"}}' "$c")"
done
./dc config --services | sort

Anything labelled climate-lama that is missing from the service list is an orphan. Remove it targeted (docker stop <name> && docker rm <name>) rather than with up -d --remove-orphans when you are doing this outside a deploy: the sweep form also reconciles the seven live services, and recreating postgres for the sake of deleting an unrelated container is a stateful restart you did not need. Drop the image afterwards too — the post-deploy prune only ever enumerates this project's own three repositories, so a retired third-party image is never a prune candidate.

Rollback is a re-dispatch with the previous tag. If the schema moved, roll it back by hand first (docker compose $P exec api alembic downgrade <revision>) — the workflow only ever upgrades. See Rollback Procedure.

Image retention on the host¶

Every deploy leaves the image it superseded on disk, and nothing used to remove it: the 38 GB host volume reached 93% on 2026-08-02 with tags accumulated since June. The final step of a successful deploy keeps the tag just deployed plus the one before it for climate-lama, climate-lama-worker and climate-lama-ui, and removes older tags of those three repositories.

Keeping n-1 is the point: it is the rollback reserve, which is exactly what a blanket docker image prune -a would destroy (the caveat raised in #539). A rollback re-dispatch to the previous tag therefore needs no re-pull.

The step is written to under-delete rather than over-delete:

Only this project's three repositories are ever listed. Infra images (postgres, redis, caddy, titiler, martin, …) are not enumerated at all, so they cannot be removed by construction.
Scoped to what was deployed. A UI-only deploy prunes only UI tags; a backbone-only deploy only backbone/worker tags.
Matched by image ID, not by tag string. Every alias of a kept image (e.g. :latest) survives, and an alias can never be mistaken for the previous release. Anything a container references — running or stopped — is skipped, and docker image rm is called without -f, so the daemon refuses in-use images as a second line of defence.
If the deployed tag is not on the host, nothing is pruned for that repository — the step warns rather than guessing which tags are stale.
It cannot fail the deploy. The step is continue-on-error and its script always exits 0: reclaiming disk must never turn a good deploy red. Free space before and after is printed in the job log.

Re-dispatching the same tag is a near-no-op up -d, so a retry after a transient failure is safe.

Required secrets¶

These live in GitHub Actions, never in the repo. They belong to the prod-host environment (Settings → Environments → New environment → prod-host → Add secret) — a dedicated environment, not the Cloudflare-Pages climate-lama (Production) one, so host access can carry its own protection rules (e.g. required reviewers on a deploy).

Secret	Value
`DEPLOY_SSH_HOST`	Hostname or IP of the prod box (e.g. `159.69.211.124`)
`DEPLOY_SSH_USER`	SSH user owning `/opt/climate-lama` and the host's GHCR login (today `root`)
`DEPLOY_SSH_KEY`	Private half of a dedicated deploy keypair, OpenSSH format, trailing newline included. Least privilege — generate a fresh key for this workflow, never reuse a personal one.
`DEPLOY_SSH_KNOWN_HOSTS`	Output of `ssh-keyscan -H <host>`; pins the host key so the connection cannot be silently MITM'd

Optional variables (same environment, or repo-level) override the defaults /opt/climate-lama, 22, https://api.climate-lama.online, https://climate-lama.online: DEPLOY_PATH, DEPLOY_SSH_PORT, PROD_API_BASE_URL, PROD_UI_BASE_URL.

No GHCR credential is added. Every pull — and the preflight existence check — runs on the host and reuses its existing docker login ghcr.io for all three images (climate-lama, climate-lama-worker, climate-lama-ui), so the PAT in Registry Authentication stays the single copy. When that PAT expires the preflight is what fails, before anything is touched.

One-time host setup:

ssh-keygen -t ed25519 -f gha-deploy -C "gha-deploy@climate-lama" -N ""
ssh-copy-id -i gha-deploy.pub root@159.69.211.124   # or append to authorized_keys
ssh-keyscan -H 159.69.211.124                       # -> DEPLOY_SSH_KNOWN_HOSTS
# gha-deploy (private) -> DEPLOY_SSH_KEY, then delete the local copy

The host's firewall must accept SSH from GitHub-hosted runners (their egress IPs are dynamic — see GitHub's meta API). A self-hosted runner or a fixed bastion is the alternative if the box is IP-restricted.

In-flight Celery messages: deploy workers before producers¶

Celery serialises an entire chain into its first message, so an ingest chord dispatched before the deploy keeps executing the old task signatures until it drains — the payload on the broker is frozen at dispatch time and is not re-serialised when a link runs. Two rules follow, and neither is optional when a release changes a task's parameters:

Deploy the worker before the producer (API). A new API dispatching a message with parameters an old worker does not accept fails that task with a TypeError — the chord dies mid-flight and nothing notices until a watchdog sweep (the 6h chord timeout for a job stuck in chunking; the shorter wedged- writing TTL if the skew hits write_chunk instead). This direction cannot be fixed in code — the receiver is the old build — so ordering is the mitigation. Draining the ingest queue first — no ingest_jobs row in pending/running/chunking/writing/aggregating/committed (committed is still in flight: aggregate_and_commit walks through it to succeeded inside one task) — removes the window entirely, and is worth doing for a release that changes an ingest task.
New parameters must be optional, and their defaults must not carry meaning. The reverse direction (an old, short message on a new worker) is a plain argument default, so it is fixable in code and is fixed: plan_chunks resolves its bbox from ingest_jobs.bbox (#453) and its legacy job_id from the jobs.params->>'ingest_job_id' back-link (#454), treating both arguments as pure overrides. A default that instead means something — bbox=None once meaning "plan against the raw file" — turns a rolling deploy into a silent behaviour change: in that case a spurious E_INGEST_PLAN_REJECTED on a job whose clipped extent was comfortably under the cell ceiling.

Hetzner shared-host deployment (behind shared Caddy)¶

The live production box (api.climate-lama.online) is a Hetzner host that already runs a shared Caddy owning :80/:443 and fronting the Climate-Lama api/ui vhosts (the box is single-tenant as of 2026-08-03 — see the decommission note in the host file layout above — but the Caddy-in-front-of-compose pattern is designed to front additional apps if the host ever gains one), with object storage on Hetzner Object Storage. That topology is captured in a version-controlled overlay, docker-compose.hetzner.yml, layered on top of the base prod compose:

docker compose -f docker-compose.prod.yml -f docker-compose.hetzner.yml \
  --env-file .env.prod up -d

What the overlay does (and deliberately nothing more):

api joins the external caddy_net network with aliases climate-lama-api (what Caddy reverse-proxies for the API host) and api (what the UI container's nginx targets via proxy_pass http://api:8000).
ui runs the published climate-lama-ui image (pinned with UI_TAG), also on caddy_net, served by Caddy at the apex/www.
the bundled nginx TLS terminator is parked behind a donotstart profile so it never contends for :80/:443 with the shared Caddy.

Object storage and healthchecks are not in the overlay — they are handled by the base docker-compose.prod.yml plus .env.prod: MINIO_ENDPOINT=nbg1.your-objectstorage.com selects Hetzner Object Storage, and the base file ships no embedded object store to park. The caddy_net network must already exist on the host (it is created by the shared Caddy stack; external: true).

DNS records (Namecheap)¶

All records are A records added by hand in Namecheap → Domain List → Manage climate-lama.online → Advanced DNS → Host Records. Namecheap is the authoritative DNS for this domain — there is no Hetzner or Cloudflare DNS zone for it.

Host	Type	Value	Serves
`api`	A	`159.69.211.124`	the backbone API
`@`	A	`159.69.211.124`	the UI (apex)
`www`	A	`159.69.211.124`	the UI (www)

A new record resolves within minutes (set TTL to 5 min while iterating). Caddy can only issue a certificate once the name resolves to the box — add the DNS record first, confirm it resolves (nslookup <host> 1.1.1.1), then add the Caddy block.

Caddy wiring (exposing a host)¶

The shared Caddy routes by container name on caddy_net. To expose a new host, add a block to /opt/caddy/Caddyfile and reload — Caddy auto-issues the Let's Encrypt cert (and auto-renews it):

api.climate-lama.online {
    reverse_proxy climate-lama-api:8000
}

climate-lama.online, www.climate-lama.online {
    reverse_proxy climate-lama-ui:80
}

cp /opt/caddy/Caddyfile /opt/caddy/Caddyfile.bak.$(date +%s)          # back up first
docker exec caddy caddy validate --adapter caddyfile --config /etc/caddy/Caddyfile
docker exec caddy caddy reload   --adapter caddyfile --config /etc/caddy/Caddyfile  # graceful; other vhosts stay up
docker logs --since 1m caddy | grep -i certificate                    # watch issuance

Provisioning an org + user¶

scripts/create_user.py provisions a user in an existing org (it does not create orgs):

docker compose $P exec api python scripts/create_user.py \
  --email you@example.com --role org_admin --org-slug <slug>
# prompts for a password — policy: >= 12 chars, >= 1 uppercase, >= 1 digit

Bootstrapping the first platform admin (#828)¶

GET /v1/admin/* (e.g. the external-dataset review queue, pending-ingests) is gated on the users.is_platform_admin flag, not an org role — a fresh deployment starts with zero platform admins, and that endpoint family is unreachable until one exists. Migration 0014 always seeds a default organization, so this works on a genuinely fresh deployment with no manual SQL:

docker compose $P exec api python scripts/create_user.py \
  --email you@example.com --role org_admin --platform-admin --org-slug default

--platform-admin is idempotent and repeatable: if --email already has an account in the org given by --org-slug (e.g. it was provisioned earlier without the flag), re-running the same command promotes it in place rather than no-op'ing — this is the supported alternative to hand-writing an UPDATE users SET is_platform_admin = true WHERE email = ... against the database. Re-running without --platform-admin remains a plain no-op, as before.

Email is only unique per org, not globally — if the account you mean to promote lives in a different org than --org-slug selects, the script refuses (rather than silently minting a second identity for the same address) and tells you which --org-slug to use instead. Get the org right: e.g. production's admin lives in org northlane (see Production login), not default.

To create the org first (and, on images that predate create_user.py, the user too) use the repositories directly:

docker compose $P exec -T api python - <<'PY'
import asyncio
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession
from climate_lama.config import get_settings
from climate_lama.db.repositories.organization_repository import OrganizationRepository

async def main():
    eng = create_async_engine(get_settings().database_url)
    async with async_sessionmaker(eng, class_=AsyncSession)() as s, s.begin():
        repo = OrganizationRepository(s)
        org = await repo.get_by_slug("northlane") or await repo.create(name="Northlane", slug="northlane")
        print("org", org.id)
    await eng.dispose()
asyncio.run(main())
PY

Login is multi-tenant — clients send org_slug alongside email/password.

Optional: UI as a compose service (dev / smoke-test)¶

For a one-command "full stack up" experience — for demos or smoke-tests, not production — the dev docker-compose.yml defines a ui service behind the fullstack profile. The service pulls the published image from GHCR and does not build from source.

# Start the backend only (default):
docker compose up

# Start backend + UI (pulls ghcr.io/cortomaltese3/climate-lama-ui):
docker compose --profile fullstack up

# Pin a specific UI version (default is :latest):
UI_TAG=v0.1.0 docker compose --profile fullstack up

The UI is reachable at http://localhost:3000 and proxies /v1/ to the API container. For production deployments the UI is deployed from its own repo; this profile is not used by docker-compose.prod.yml.

Docs Site (Cloudflare Pages)¶

The MkDocs Material docs site is built by GitHub Actions and deployed to Cloudflare Pages. The live site is at https://climate-lama.pages.dev.

How it works¶

Once per weekday (05:00 UTC cron), plus on demand via gh workflow run docs.yml --ref main: the site is built from main (mkdocs build --strict) and deployed to production (climate-lama.pages.dev). There is no per-PR preview and no per-push deploy — see CI.md.
On every ci.yml trunk gate: the Checks job runs the same strict build (issue #1267), so a docs change that would break the deploy turns the gate red.
Both install pyproject.toml's docs extra; a local check is uv run --extra docs mkdocs build --strict.

The workflow lives at .github/workflows/docs.yml.

One-time setup (already done)¶

These steps were completed when the pipeline was first wired up. Documented here so they can be reproduced if credentials need to be rotated.

1. Create a Cloudflare account

Free at cloudflare.com. No credit card required.

2. Create a Pages project

Cloudflare Dashboard → Workers & Pages → Create → Pages → Connect to Git → authorise GitHub → select the climate-lama repo → set project name to climate-lama. Skip the build configuration (GitHub Actions handles the build).

3. Get credentials

Account ID: visible in the right sidebar of any Workers & Pages page, or in the dashboard URL (https://dash.cloudflare.com/<account-id>).
API Token: My Profile → API Tokens → Create Token → Create Custom Token. Add one permission row: Account / Cloudflare Pages / Edit. Leave all other settings at defaults.

4. Add secrets to the GitHub repo

Repository Settings → Secrets and variables → Actions → New repository secret:

Secret name	Value
`CLOUDFLARE_API_TOKEN`	The token created in step 3
`CLOUDFLARE_ACCOUNT_ID`	The account ID from step 3

Rotating credentials¶

If the API token is revoked or expires:

Create a new token in the Cloudflare dashboard (same permissions as above).
Update the CLOUDFLARE_API_TOKEN secret in GitHub repo settings.
The next push to main will use the new token automatically.

Triggering a deploy manually¶

Push any v* tag pointing to the commit you want deployed:

git tag v0.4.0 && git push origin v0.4.0

To test without a real release, use a pre-release tag and delete it afterwards:

git tag v0.4.0-rc1 && git push origin v0.4.0-rc1
# verify the deploy, then clean up
git tag -d v0.4.0-rc1 && git push origin :refs/tags/v0.4.0-rc1

Custom domain (optional)¶

To serve the site from a custom domain (e.g. docs.climate-lama.io):

Cloudflare Dashboard → climate-lama Pages project → Custom domains → Set up a custom domain → enter the domain.
If the domain is managed in Cloudflare DNS, the CNAME is added automatically. Otherwise, add a CNAME record pointing to climate-lama.pages.dev at your DNS provider.
Update site_url in mkdocs.yml to the new domain.

Backup Strategy¶

scripts/backup.sh and scripts/restore.sh (issue #388) replace the old manual pg_dump snippet with a scripted, scheduled backup and a documented, exercised restore. Both are plain bash, no extra host dependencies beyond docker + docker compose — the Postgres dump runs via docker compose exec, and the object-store step runs mc in a throwaway pgsty/mc container (MC_IMAGE), so it works the same way against either object-store backend the app supports (both speak the S3 API behind the same MINIO_* env vars):

Hetzner Object Storage (prod default) — MINIO_ENDPOINT is a public FQDN.
Embedded/local MinIO (dev docker-compose.yml) — MINIO_ENDPOINT is a bare compose service name (e.g. minio:9000); the script joins the stack's docker network to resolve it.

The MinIO images and the re-pull step¶

Upstream MinIO no longer publishes anonymously pullable images (its quay.io and Docker Hub repositories both refuse an anonymous pull since 2026-09), so the stack runs the pgsty community fork (ADR-082). One pin per role, tag + digest, byte-identical in every consumer:

Server — docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Z@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372, used by docker-compose.yml's minio service and ci.yml's slow-job MinIO service.
Client — docker.io/pgsty/mc:RELEASE.2026-09-16T00-00-00Z@sha256:cfc83108c3abb371f8fb84d99c1fdc88f8c237e022409b0081fb7c0a3be634dd, used by docker-compose.yml's minio-init, the MC_IMAGE default of scripts/backup.sh and scripts/restore.sh, the hzmc alias under Hetzner Object Storage and the slow CI job's mc binary.

The bucket name, the MINIO_* variables and the S3 API are unchanged. The images keep the upstream layout (minio server … entrypoint, mc inside the server image for the healthcheck, a shell in the client image), so no command changed with them.

Prod re-pull (after the first deploy that carries this change). Prod runs no embedded MinIO — only the client image reaches it, through the backup/restore scripts and hzmc. A host that still caches the old quay.io image keeps working until its cache is pruned, so pre-pull the new pin once on the prod host and check it answers:

docker pull docker.io/pgsty/mc:RELEASE.2026-09-16T00-00-00Z@sha256:cfc83108c3abb371f8fb84d99c1fdc88f8c237e022409b0081fb7c0a3be634dd
docker run --rm docker.io/pgsty/mc:RELEASE.2026-09-16T00-00-00Z@sha256:cfc83108c3abb371f8fb84d99c1fdc88f8c237e022409b0081fb7c0a3be634dd --version
# -> mc version RELEASE.2026-09-16T00-00-00Z …

Nothing restarts: every mc call is a throwaway container, so the next backup, restore or hzmc call runs the new pin (and would pull it on first use — the pre-pull only moves a registry failure from a scheduled backup to deploy time).

Dev stacks pick the images up with docker compose pull minio minio-init && docker compose up -d minio minio-init. The minio_data volume carries over: a bucket and an object written by the old quay.io server pin read back unchanged through the new one.

What is (and isn't) covered¶

Each run writes a timestamped directory under --backup-root (default ./backups):

backups/20260727_030000/
  db.dump         # pg_dump -Fc (Postgres custom format, restorable with pg_restore)
  objects/        # mc mirror of the configured MINIO_BUCKET
  MANIFEST.txt    # what was backed up, from where, at what git sha

Covered: the Postgres database, and the configured object-store bucket. Not covered — nothing is backed up silently:

.env.prod (DB password, API keys, object-store credentials) — never copied into a backup directory. Reprovision it from your secrets manager on restore.
nginx/Caddy TLS config, host firewall/DNS setup.
Redis (queue/cache state) — disposable; in-flight jobs are re-run, not restored.

MANIFEST.txt restates this coverage list on every run so it travels with the backup itself, not just this doc.

Running a backup¶

scripts/backup.sh                                    # defaults: docker-compose.prod.yml, .env.prod, ./backups, 14-day retention, 1024 MB free-space floor
scripts/backup.sh --retention-days 30                 # keep a month
scripts/backup.sh --compose-file docker-compose.yml --env-file .env   # local/dev stack
scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024   # DB-only, the scheduled-run flags (see below)
scripts/backup.sh --skip-objects --retention-days 7 --push-remote        # ...and ship the dump off-host

The dump is verified before the run is called a success¶

pg_dump exiting 0 is not proof of a restorable archive — a dump truncated by a full disk, a killed container or a broken pipe still leaves a plausible-looking file behind. Every run therefore reads its own dump back with pg_restore --list (run inside the postgres container, so no host-side Postgres client is needed) and fails unless the archive's table of contents enumerates at least --min-table-data TABLE DATA entries — default 10; production measured 66 on 2026-09-06 (the pre-migration dump cited under Scheduling below). A dump that fails either check is renamed to db.dump.FAILED-VERIFICATION, so scripts/restore.sh cannot mistake it for a usable backup, and the script exits non-zero the night it happens rather than the day someone needs it.

Note the bound: this reads the archive's table of contents. It catches an unreadable, non-pg_dump or implausibly small archive; it does not prove every row inside a well-formed archive restores. A restore into a scratch database is the stronger check and is still a manual drill (see below).

Off-host copy (`--push-remote`)¶

By default a backup lands only on the host it protects, which covers accidental data loss but not loss of the host. --push-remote also uploads db.dump and MANIFEST.txt to <--remote-prefix>/<timestamp>/ in the same S3-compatible bucket the app already uses — same MINIO_* credentials and the same throwaway-mc-container plumbing as the mirror step, so it needs no new machinery or secrets. --remote-prefix defaults to backups.

It is opt-in: without the flag the script behaves exactly as before.
MINIO_* becomes required as soon as --push-remote is passed, even alongside --skip-objects.
The objects/ mirror is never uploaded — it is a copy of that same bucket.
A failed upload fails the run (exit 1) but never discards the verified local dump.
Remote copies are not pruned by --retention-days. Expire them with a bucket lifecycle rule.

This matters more than the object-store side suggests: exposure_datasets has no path column, so the 717k exposure rows are PostGIS geometry in Postgres and exist in no bucket. Losing the bucket costs rasters that can be re-ingested from upstream; losing the host costs data that exists nowhere else.

Retention¶

Two numbers, both correct, and they are not the same number. backup.sh's own default is 14 days (RETENTION_DAYS="${RETENTION_DAYS:-14}", scripts/backup.sh) — that is what you get running the script bare, and it is the figure quoted under Running a backup above. The systemd timer installed on the prod host overrides it to 7 by passing --retention-days 7 (see Scheduling below). So "14-day" describes the tool and "7-day" describes production; the sizing arithmetic below is stated against the installed 7.

Backups older than --retention-days (default 14) are pruned at the end of a successful run. A directory under --backup-root is prunable when it is recognisably a backup: its name matches the script's own YYYYMMDD_HHMMSS pattern, or it contains a db.dump or a MANIFEST.txt. The second clause is what hand-taken pre-deploy dumps (predeploy-v0.5.0, pre-v0.6.0, pre-org-migration-…) look like; under the old name-only rule they were never pruned and had accumulated 1.2 GB past the window in production (#771). Anything else parked under --backup-root is still never touched, and the directory the current run just wrote is excluded unconditionally. Pass --prune-timestamped-only to restore the legacy name-only rule.

--skip-objects dumps the database only and skips mirroring the object-store bucket (mirrors the flag scripts/restore.sh already has). Before doing any docker/postgres work, the script also refuses to start with less than --min-free-mb (default 1024) free at --backup-root, so an unattended run on a disk shared with another app fails loudly with the shortfall named, instead of silently filling the disk.

Scheduling (prod host)¶

docker compose has no built-in scheduler, so this runs at the host level. The systemd timer below is the mechanism actually installed on the prod host — DB-only (--skip-objects), 7-day retention. A cron entry works identically and is documented as a fallback, but it is not what is running; if you switch to it, disable the timer first so a backup doesn't run twice.

Sizing, from the 2026-09-06 pre-migration dump — /opt/climate-lama/backups/20260906_181653/db.dump, the most recent measurement and the one this section is now stated from. That DB-only dump is 602 MB with 66 TABLE DATA entries, so the timer's 7-day retention holds about 4.2 GB at steady state. Free space was last measured at ~18 GB of a 38 GB volume (52% used) on 2026-08-03, after lama-chat's removal — so 4.2 GB of backups fits with room to spare, and every run still clears backup.sh's MIN_FREE_MB floor of 1024 MB (which the timer also passes explicitly) and its MIN_TABLE_DATA floor of 10 by a wide margin (66 ≫ 10).

The database has grown 2.3× since the previous measurement — 602 MB on 2026-09-06 against ~257 MB on 2026-08-02 — which is exactly why this arithmetic is dated rather than stated as a standing fact. Re-measure it after a data-loading phase, and re-measure free space alongside it: the floor makes an over-full disk fail loudly instead of silently, but it does not create space. At the current rate the 7-day window would need another 4× of database growth before free space, rather than the floor, became the binding constraint.

systemd timer (installed) — /etc/systemd/system/climate-lama-backup.service:

[Unit]
Description=Climate-Lama backup (DB only, object store skipped)

[Service]
Type=oneshot
WorkingDirectory=/opt/climate-lama
ExecStart=/opt/climate-lama/scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024

/etc/systemd/system/climate-lama-backup.timer:

[Unit]
Description=Run Climate-Lama backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl enable --now climate-lama-backup.timer

A failed run is visible via the unit's own exit status — systemctl status climate-lama-backup.service and journalctl -u climate-lama-backup.service show the script's FAIL line (stderr is captured by journald automatically for a systemd-managed service), so a human checking either surfaces it even without dedicated alerting.

cron (documented fallback, not installed) — as the deploy user, crontab -e:

# Climate-Lama backup, daily at 03:00
0 3 * * * cd /opt/climate-lama && ./scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024 >> /var/log/climate-lama-backup.log 2>&1

Object-store mirroring is skipped on the scheduled run (the bucket lives on Hetzner Object Storage with its own durability); run scripts/backup.sh without --skip-objects by hand for an occasional full mirror.

Host-side step, not done by merging code (#771): the installed unit above does not yet pass --push-remote, so the scheduled dump still lands only on the host it protects. Appending --push-remote to the unit's ExecStart (then systemctl daemon-reload) is what closes the host-loss gap; until that is done on the box, the capability exists in the script and is unused in production.

Still out of scope (tracked as follow-ups, not built here): alerting on backup failure, bucket versioning (#857), and an automated restore-into-a-scratch-database drill.

Status (#533): the flags and guard above are implemented and tested in this repo. Installing the timer on the production host — and confirming it produces a dated backup directory without being run by hand — is a separate, host-side step tracked on that issue; it is not done by merging this doc change.

Restore procedure¶

scripts/restore.sh <backup-dir> [--compose-file <path>] [--env-file <path>] [--yes]

<backup-dir> is one of the timestamped directories backup.sh produced (e.g. ./backups/20260727_030000). The script is destructive by design — it prompts for confirmation (--yes to skip, required when not run at a tty) before:

pg_restore --clean --if-exists — drops and recreates database objects from db.dump.
mc mirror — overwrites matching keys in the target bucket from objects/.

Use --skip-db / --skip-objects to restore only one half.

After any DB restore, re-run scripts/create_app_role.py --apply if the RLS role flip is live. pg_restore --clean drops and recreates every object as POSTGRES_USER, and the recreated objects carry no grants to climate_lama_app — ALTER DEFAULT PRIVILEGES applies to objects created after it was set, not retroactively to a restore. The api and worker would come back up throwing permission denied for table … on everything. The role itself survives the restore (roles are cluster-level, not in the dump); only its grants are lost.

backup.sh and restore.sh themselves are unaffected by the flip: they exec into the postgres container and authenticate as POSTGRES_USER over the container-local socket, never through DATABASE_URL.

Exercised restore drill¶

Run once against a scratch stack (local docker-compose.yml, isolated project/network, torn down afterward — no prod data involved) to prove the procedure actually works, not just that it parses. Transcript, lightly trimmed:

$ docker compose -f docker-compose.yml up -d postgres minio minio-init
 Container climate_lama_postgres  Started
 Container climate_lama_minio     Healthy
 Container climate_lama_minio_init Started

# seed data to lose
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama \
    -c "CREATE TABLE backup_drill (id serial PRIMARY KEY, note text); \
        INSERT INTO backup_drill (note) VALUES ('issue-388-restore-drill');"
CREATE TABLE
INSERT 0 1
$ mc cp drill-object.txt local/climate-lama/drill-object.txt
`drill-object.txt` -> `local/climate-lama/drill-object.txt`

# back up
$ scripts/backup.sh --compose-file docker-compose.yml --env-file .env --backup-root ./backups
-> dumping database "climate_lama" (pg_dump -Fc)...
✓ database dump: ./backups/20260727_055915/db.dump (8.0K)
-> mirroring object store bucket "climate-lama" (minio:9000)...
✓ object store mirror: ./backups/20260727_055915/objects (1 object(s))
✓ manifest: ./backups/20260727_055915/MANIFEST.txt
✓ backup complete: ./backups/20260727_055915

# simulate data loss
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "DROP TABLE backup_drill;"
DROP TABLE
$ mc rm local/climate-lama/drill-object.txt
Removed `local/climate-lama/drill-object.txt`.
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "SELECT * FROM backup_drill;"
ERROR:  relation "backup_drill" does not exist

# restore
$ scripts/restore.sh ./backups/20260727_055915 --compose-file docker-compose.yml --env-file .env --yes
This will overwrite data using ./backups/20260727_055915:
  - DROP and recreate objects in database "climate_lama" (pg_restore --clean)
  - overwrite matching keys in bucket "climate-lama" (minio:9000)
-> restoring database "climate_lama" from ./backups/20260727_055915/db.dump...
✓ database restored from ./backups/20260727_055915/db.dump
-> restoring object store bucket "climate-lama" from ./backups/20260727_055915/objects...
✓ object store restored from ./backups/20260727_055915/objects
✓ restore complete from ./backups/20260727_055915

# verify
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "SELECT * FROM backup_drill;"
 id |          note
----+-------------------------
  1 | issue-388-restore-drill
(1 row)

$ mc cat local/climate-lama/drill-object.txt
issue-388 backup/restore drill object

Both the dropped table (and its row) and the deleted object came back exactly as backed up. Scratch stack torn down with docker compose -f docker-compose.yml down -v afterward — nothing above touched prod data or credentials.

Hetzner Object Storage (production object storage)¶

Production deployments use Hetzner Object Storage as the S3-compatible backend for the same MINIO_* env vars the backbone uses locally. The canonical bucket layout is defined by ADR-029 (written for DO Spaces, but S3-generic — it applies to any backend unchanged).

Current bucket: climate-lama-storage in location nbg1 (endpoint https://nbg1.your-objectstorage.com).

Migration note (2026-07): the previous backend, DigitalOcean Spaces (climate-lama-space @ fra1), is decommissioned. A mirror of its last contents sits in the new bucket under the legacy climate-lama-space/ prefix (a handful of June hazard HDF5s with no DB rows referencing them) — safe to delete once confirmed orphaned.

Credentials¶

Hetzner S3 credentials are scoped to the Hetzner Cloud project, not the bucket — one access/secret key pair reaches every bucket in the project. Generate or revoke them in the Hetzner Cloud console under the project's Object Storage section, and put them in .env.prod only (mode 600 on the host; never committed).

One-time setup (done 2026-07-28)¶

Documented here so the bucket can be reprovisioned or a second location added. The bucket itself is created in the Hetzner Cloud console. Everything else is plain S3 API — the commands below use mc (MinIO client) in a throwaway container so the host needs nothing beyond docker; run them on the prod host so the credentials are read from .env.prod instead of appearing in a shell history:

alias hzmc='docker run --rm --env-file /opt/climate-lama/.env.prod \
  --entrypoint sh docker.io/pgsty/mc:RELEASE.2026-09-16T00-00-00Z@sha256:cfc83108c3abb371f8fb84d99c1fdc88f8c237e022409b0081fb7c0a3be634dd -c'

1. Create the canonical prefix markers

S3 has no real folders — empty .keep objects make the ADR-029 prefix hierarchy (raw/, processed/, processed/public/, tiles/, reports/) visible in the console and give lifecycle rules a stable target:

hzmc 'mc alias set hz "https://$MINIO_ENDPOINT" "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" &&
  for p in raw processed processed/public tiles reports; do
    printf "" | mc pipe hz/$MINIO_BUCKET/$p/.keep
  done && mc ls hz/$MINIO_BUCKET'

2. Lifecycle rule for tiles/ — pending

Tiles are regenerable (ADR-029 invariant 4) — expire them after 30 days to keep storage bounded. Hetzner implements the S3 lifecycle API (verified: GetBucketLifecycleConfiguration answers NoSuchLifecycleConfiguration rather than rejecting the call), but the rule has not been applied yet:

hzmc 'mc alias set hz "https://$MINIO_ENDPOINT" "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" &&
  mc ilm rule add --expire-days 30 --prefix tiles/ hz/$MINIO_BUCKET &&
  mc ilm rule ls hz/$MINIO_BUCKET'

3. CORS — none configured, none currently needed

The UI never fetches the bucket directly: tiles and reports are served through the backbone's read-through proxy (/v1/tiles/...), so browser CORS rules on the bucket are unnecessary today. Hetzner answers the bucket-CORS S3 API (verified via mc cors get), so if a browser-direct path is ever introduced (e.g. presigned-URL fetches from the SPA), add rules with mc cors set for https://climate-lama.online + www at that point.

4. Configure environment variables

Set in .env.prod (and .env locally when testing against the live bucket):

MINIO_ENDPOINT=nbg1.your-objectstorage.com
MINIO_ACCESS_KEY=<hetzner-s3-access-key>
MINIO_SECRET_KEY=<hetzner-s3-secret-key>
MINIO_BUCKET=climate-lama-storage
MINIO_SECURE=true
TITILER_S3_PREFIX=s3://climate-lama-storage

The backbone reads these from src/climate_lama/config.py. Hetzner Object Storage is S3-compatible, so no code change is needed to switch from MinIO. The setting that toggles HTTPS is MINIO_SECURE (the minio Python client's secure= flag) — set it true here. There is no MINIO_USE_SSL variable; because Settings ignores unknown env keys, a misnamed one is silently dropped and the client falls back to plain HTTP.

docker-compose.prod.yml no longer hardcodes MINIO_ENDPOINT; it reads the whole MINIO_* block from .env.prod, so pointing at the external bucket is purely a config change. The prod compose ships no embedded object store — the external bucket is required; the embedded MinIO exists only in the dev docker-compose.yml.

Geocoding in production¶

The geocoding compose profile (nominatim in docker-compose.prod.yml) is opt-in and off by default. With it off, GET /v1/geocode returns 503 E_GEOCODE_UNAVAILABLE, the caller's explicit signal to fall back to manual pin-drop (ADR-044) — the decision record is issue #534, reaffirmed by #1322.

Phase 16 (docs/plan/phase-16-consistency.md, D8) reverses that for prod on new evidence: the owner re-reported the Lookup address box as not working after #1322 closed. Since #1374 the deploy keeps the profile on whenever .env.prod carries GEOCODER_ADVERTISED=true; switching it on is the owner-run host action #1377, following Enable address search below. Until that runs, the 503 remains what production answers.

GEOCODER_ENABLED stays "true" on the api service regardless — flipping it to false would erase a diagnostic signal: with it true, the 503's internal reason distinguishes a genuine Nominatim outage ("connection") from a deliberate disable ("disabled").

The two geocoder flags¶

Because GEOCODER_ENABLED is held "true" for that diagnostic reason on a host that runs no Nominatim, it is not a truthful answer to "can this deployment geocode?". Since issue #1336 the two questions have two flags, and GET /v1/info advertises the conjunction:

Flag	Question it answers	Prod value	Effect
`GEOCODER_ENABLED` (`Settings.geocoder_enabled`)	Is the geocoding code path switched on?	`"true"` (literal, in `docker-compose.prod.yml`)	Gates `core/geocoding.py`: off ⇒ `GeocodeUnavailableError("disabled")` before any network call
`GEOCODER_ADVERTISED` (`Settings.geocoder_advertised`)	Should a client mount an address-entry surface at all?	`false` (compose default `${GEOCODER_ADVERTISED:-false}`) until #1377 flips it	Advertising only. Gates no code path; it cannot make a geocode succeed or fail

GET /v1/info reports features.geocoding = geocoder_enabled AND geocoder_advertised (core/geocoding.py::geocoding_advertised, read per request through get_settings()). So production advertises false, which is what #1322 decided (pin-drop-only, reaffirming #534) — while the 503's reason keeps saying "connection" for whoever calls GET /v1/geocode directly. Advertising can never outrun the master switch: with GEOCODER_ENABLED off, the advertised value is false whatever GEOCODER_ADVERTISED says.

The flag has no reader yet. Nothing consumes features.geocoding today — gating the UI's /lookup address box on it is climate-lama-ui#424, still open. Until that lands, the address box renders regardless and this flag changes nothing a user sees; it is the backbone half of the contract, published first so the UI has something truthful to read.

Flipping it on. GEOCODER_ADVERTISED=true in /opt/climate-lama/.env.prod is the whole change on the config side (the compose environment: entry interpolates from the --env-file .env.prod the deploy passes) — and since #1374 the same line is what makes every deploy keep nominatim running. Follow Enable address search: it orders the flag, the import and the api recreate so the flag never advertises a geocoder that is not serving. Advertising a geocoder that is not running is the one failure this flag exists to prevent.

Enable address search¶

The deploy side (#1374). Each of the three P= assemblies in deploy-prod.yml — the backbone roll, the UI roll and the on-failure dump — appends the geocoding profile when .env.prod matches ^GEOCODER_ADVERTISED=true$, read on the host inside each SSH step exactly like the climada gate described below. There is no new .env.prod key: the gate reads the key that already flips features.geocoding, so advertising address search and keeping nominatim alive through up -d --remove-orphans are one decision. Two guards ride along:

The backbone step refuses to deploy (::error::, before it edits .env.prod) when a line sets GEOCODER_ADVERTISED to anything other than exactly true, false or nothing. Compose and the api's pydantic bool accept far more — True, 1, "true", a trailing # comment, an export prefix, spaces around = — any of which would advertise a geocoder the profile gate does not keep. The error names the key, never its value.
The UI step's container-id proof includes nominatim, so a UI deploy that restarted it mid-import fails loudly. With the profile off both sides read the same and the check is inert.

The backbone step also prints -> compose flags: …, the run log's record of which opt-in profiles a roll carried.

Switching it on is a host action (owner-run, #1377). In this order, from /opt/climate-lama:

Check headroom. df -h / and free -h. The first boot imports the Greece OSM extract into nominatim's own PostgreSQL — multi-GB and memory-hungry (see What standing Nominatim up would actually cost). NOMINATIM_PASSWORD must already be set in .env.prod; it is (Host-side step — done). Record the numbers on #1377.
Set the flag first. Edit the line in .env.prod to exactly GEOCODER_ADVERTISED=true — lowercase, unquoted, no trailing comment. It goes first so a deploy that lands during the import keeps nominatim instead of reaping it. The running api does not see the edit yet: a container reads .env.prod interpolation only when it is recreated (step 5).
Start the import. ./dc --profile geocoding up -d nominatim.
Wait for the Greece import. ./dc --profile geocoding ps nominatim until it reads (healthy); follow it with ./dc --profile geocoding logs -f nominatim. The healthcheck allows a start_period of 45 minutes, then 10 retries 30 s apart, so an import that runs longer shows unhealthy while it is still working — trust the logs over the status. Record wall time, disk and RAM on #1377. Avoid dispatching a backbone deploy in this window: it recreates the api, which would advertise features.geocoding = true while /v1/geocode still answers 503.
Recreate the api so it reads the flag. ./dc up -d api — compose sees the changed GEOCODER_ADVERTISED value and recreates api alone (a few seconds of api restart).
Verify.

curl -fsS https://api.climate-lama.online/v1/info          # "features": {"geocoding": true, ...}
curl -fsS -H "Authorization: Bearer $TOKEN" \
  "https://api.climate-lama.online/v1/geocode?q=Thessaloniki"  # 200 with candidates

/v1/geocode needs a token with at least the viewer role. 7. The next deploy keeps it. Its run log shows -> compose flags: … ending in the geocoding profile, and ./dc --profile geocoding ps nominatim shows the same container still up afterwards.

Coverage. The extract is Greece only (PBF_URL: …/greece-latest.osm.pbf), so Cyprus addresses do not resolve. A merged GRC+CYP extract or a second instance is out of scope for Phase 16 (see its plan's "Out of scope").

Switching it off. Set GEOCODER_ADVERTISED=false and run ./dc up -d api first, so the api stops advertising before the geocoder goes away; then ./dc --profile geocoding stop nominatim. The nominatim_data volume keeps the imported database until it is removed explicitly.

Why the 503 is safe to ship¶

src/climate_lama/core/geocoding.py:172-173 — when Settings.geocoder_enabled is False (the library default — config.py:779), geocode_address/reverse_geocode raise GeocodeUnavailableError("disabled") before any network call.
src/climate_lama/core/geocoding.py:264-265 — when the provider can't be reached (a compose service that isn't running fails DNS resolution immediately), the adapter catches httpx.HTTPError — the base class of ConnectError, ConnectTimeout, and ReadTimeout — and raises GeocodeUnavailableError("connection", ...). Every call is additionally bounded by geocoder_timeout_seconds (config.py:781, default 5.0), so a hung provider can't hang the request either.
Both paths map to the same 503 E_GEOCODE_UNAVAILABLE at src/climate_lama/main.py:393. A client that handles the disabled case already handles the not-yet-provisioned case — there is no separate failure mode to build for.

What standing Nominatim up would actually cost¶

mediagis/nominatim imports an OSM extract into its own PostgreSQL instance on first boot — not a config toggle. Even a Greece-only extract is a multi-GB, long-running, memory-hungry import. The production host had 13.4 GB free of a 38 GB volume when measured on 2026-08-02, and that one box runs the entire climate-lama stack plus Caddy, so an unattended multi-GB import competes with the live service for both disk and memory. Standing the profile up for real needs a disk assessment (likely a dedicated volume) first, or routing through a hosted/third-party geocoder instead — the adapter is provider-agnostic by design (ADR-044), so either is a legitimate answer without reversing the fail-closed default.

Update (2026-08-03): the co-tenant/disk-contention framing above is now void — lama-chat has been decommissioned (see the note in the host inventory near the top of this doc), the host is single-tenant, and free disk has grown to ~18 GB of the same volume (52% used), up from the 13.4 GB measured on 2026-08-02. This retires the rationale above, not the decision: geocoding stays disabled-by-fallback in production until the owner explicitly revisits it — a memory-hungry OSM import competing with the live service is still a real cost worth a deliberate decision, just no longer one made scarcer by a co-tenant.

Update (Phase 16, D8): the owner has revisited it — geocoding is being enabled on prod through Enable address search (#1374 deploy side, #1377 host action).

Measured (2026-09-16, #1377): the Greece import was run on the rescaled host (150 GB volume, 94 GB free, 15.6 GB RAM, 8 vCPU). The container was healthy 26 min after up -d nominatim. The nominatim_data volume is 8.0 GB (disk used went from 51 GB to 60 GB). Sampled container memory peaked at ~3.0 GB during the import and settled at ~1.2 GB once serving; sampling began 22 min in, so the true peak may be higher. The 13.4 GB / 38 GB figures above describe the pre-rescale host.

Host-side step — done¶

.env.prod carries NOMINATIM_PASSWORD (set on the host on 2026-08-14), and docker compose ... config now emits zero warnings.

The reason it has to be set even though the nominatim service is never started: Compose interpolates variables for every service defined in the file at parse time, regardless of which profiles are active, so an unset NOMINATIM_PASSWORD used to make every invocation on the host print

level=warning msg="The \"NOMINATIM_PASSWORD\" variable is not set. Defaulting to a blank string."

Keep the key set in .env.prod — .env.example.prod ships it as a change-me-… placeholder for that reason. This was the last host-side step on issue #534.

CLIMADA sidecar on the deploy path (opt-in profile)¶

Same shape as geocoding above: the climada compose profile (climada-sidecar in docker-compose.prod.yml, ADR-042) is opt-in and off by default. Setting CLIMADA_SIDECAR_URL in .env.prod (unset or a bare CLIMADA_SIDECAR_URL= both count as off) makes deploy-prod.yml build the sidecar image before the api/worker roll and keep the --profile climada flag on every docker compose call in that job, so --remove-orphans never tears it down; leaving the key unset makes every deploy byte-identical to today. The deploy also uploads the sidecar's build context (docker/climada-sidecar/) to the host on every run, which is what that build reads — see What the deploy does when the key is set. scripts/verify-deploy.sh probes the sidecar's /health too, but only when run from the host itself (the sidecar is never exposed over HTTP — see the compose file's comment). This paragraph is a pointer only; the full operator runbook (enabling the profile, sizing its memory limit in docker-compose.hetzner.yml, first activation) is tracked separately as ops work (issue #1051) and not duplicated here.

Enabling the CLIMADA engine (opt-in profile)¶

Operator command reference for the climada compose profile the section above describes. First activation on the live host — a real captured docker compose ... up transcript and a real docker stats reading — is ops work tracked separately as issue #1051. This section documents the mechanism and the commands; it is not that walkthrough.

The two keys¶

.env.prod and .env.example.prod both declare CLIMADA_SIDECAR_URL and CLIMADA_SIDECAR_TIMEOUT_SECONDS below the templates' shared tail marker (# --- Optional - omit to keep the default ---) — they are optional keys by construction, absent from the 33-key spine tests/test_env_template_parity.py enforces across both templates. Leaving both unset — or CLIMADA_SIDECAR_URL= present but empty, which counts the same — keeps the engine registered-but-unavailable: a comparison that omits engines reports it in excluded_engines with reason "unavailable" and continues with whatever else is available, while a request naming climada explicitly fails fast with 400 E_ENGINE_UNAVAILABLE before anything is queued. Setting CLIMADA_SIDECAR_URL=http://climada-sidecar:8100 (the in-network service name from docker-compose.prod.yml) turns the engine on; CLIMADA_SIDECAR_TIMEOUT_SECONDS (default 900) bounds the sidecar's compute time per job.

What the deploy does when the key is set¶

deploy-prod.yml's backbone step reads CLIMADA_SIDECAR_URL from .env.prod on the host, inside the SSH session — it is never uploaded to or read on the runner. A non-empty value folds --profile climada into that step's $P for the rest of the run and builds the sidecar image before the api/worker roll: a failed CLIMADA build fails the deploy outright rather than surfacing later as a silently-missing orphaned service. The UI-deploy step and the on-failure log dump each re-derive the same check independently — a variable set in one SSH step does not survive into the next — so --profile climada stays on every $P invocation across the whole job and --remove-orphans never tears the sidecar down. Leaving the key unset makes every deploy byte-identical to today.

The deploy delivers the build context the host builds from. The sidecar is built on the host and never pulled from a registry (ADR-042), and compose resolves its build.context: docker/climada-sidecar against the deploy directory — so /opt/climate-lama/docker/climada-sidecar/ has to hold Dockerfile and server.py. Nothing else puts them there: /opt/climate-lama is hand-assembled, the workflow runs no git pull on the host, and neither backbone image copies docker/climada-sidecar/. Until issue #1115 the deploy uploaded only the two compose files, so the directory was simply absent and the pre-build died on a missing context — which, with CLIMADA_SIDECAR_URL set, failed every backbone deploy, not just a sidecar-intent one. The Upload the CLIMADA sidecar build context to the host step now ships those two files (~48 KB) with the same sha256-verified staging the compose files get: mkdir -p the target directory, scp to .incoming, verify the hash that left the runner, back up any previous host copy as .bak.<UTC timestamp> (keeping the last 10), then install with an atomic mv. It runs unconditionally, because the profile is only knowable on the host while scp runs on the runner; that is harmless while the profile is off and keeps the host's copy fresh for whenever it is switched on. So a change to docker/climada-sidecar/ reaches production by dispatching a deploy — not by editing the host by hand.

Building it once by hand¶

docker compose $P --profile climada build climada-sidecar

($P is the Release → Production shorthand: -f docker-compose.prod.yml -f docker-compose.hetzner.yml --env-file .env.prod.)

This builds from ${DEPLOY_DIR}/docker/climada-sidecar/, whatever that directory currently holds. After bumping the CLIMADA version pinned in docker/climada-sidecar/Dockerfile (or changing server.py), dispatch a deploy first — that is what refreshes the host's copy (see above) — and only then rebuild. Running the build by hand against a stale host copy silently rebuilds the old pin, and the deploy's own pre-build will do the rebuild for you anyway once the profile is on.

Rollback semantics¶

docker-compose.prod.yml pins the sidecar to image: climate-lama-climada-sidecar:local — built on the host, never pushed to any registry (ADR-042: CLIMADA must not leave this private stack). Because that tag carries no ${IMAGE_NS} prefix, the post-deploy image prune — scoped to the GHCR namespace — never touches it, so there is no rollback-by-retag for this one service the way CLIMATE_LAMA_TAG rolls back api/worker. A rebuild is the only way to change what :local points at:

docker compose $P --profile climada build climada-sidecar
docker compose $P up -d climada-sidecar

The Rollback Procedure below still applies normally to the CLIMADA adapter code inside api/worker (it ships with the CLIMATE_LAMA_TAG image like everything else); only the sidecar image itself sits outside that mechanism.

Celery workers and queues¶

Three queues, one worker image (celery_app.py::task_routes, pinned by tests/test_worker/test_celery_routes.py):

Queue	Prod consumer	Concurrency	Carries
`compute`	`worker`	1	ingest chords, `compute_event_impact`, cost-benefit, reports, maintenance
`impact`	`impact-worker`	3	`compute_impact`: every run and every matrix or comparison cell (issue #1569)
`live`	`live-worker`	1	the live-feed tick and polls

The concurrencies live in docker-compose.hetzner.yml. The image's default command and the dev worker consume compute,impact, so any single-worker stack (dev, the UI repo's smoke stack, which extends the dev worker, and sdk-smoke.yml) still serves impact runs. Every compute_impact dispatch names queue=IMPACT_QUEUE (core/job_dispatch.py), because an explicit queue overrides the route table.

Memory budget. impact-worker runs three pool processes, so its peak is up to three concurrent engine runs. The host budget is: 3 × one impact process's peak, plus the worker, live-worker, api and postgres peaks, stays under the host's 15 GiB. No measured per-process peak exists yet. The needs-prod-proof step of issue #1569 records it after the first deploy: two parallel scripts/verify_engine_agreement.py runs with docker stats --no-stream sampled during them, the arithmetic pasted in the release record. If the sum does not fit, lower --concurrency in docker-compose.hetzner.yml.

Memory footprint and host budget¶

The production host is a Hetzner CX43 — 8 vCPU / 15.25 GiB RAM, measured live on 2026-08-29. (A nearby comment in docker-compose.hetzner.yml, on the worker's concurrency override, records the earlier "2 vCPU / 3.7 GiB" box. #1042 flagged it stale, and #1569 was the deliberate follow-up: impact runs moved to impact-worker at concurrency 3, see Celery workers and queues.) Steady-state containers (api, worker, beat, postgres, redis, titiler) measured ~2.87 GiB on the same date, leaving roughly ~8 GiB of headroom on the host against the sidecar's mem_limit: 4g ceiling in docker-compose.hetzner.yml.

That 8 GiB of headroom is proof the ceiling fits on this host, not proof 4g is correctly sized — docker-compose.hetzner.yml already flags the ceiling itself as an [ASSUMPTION]: CLIMADA's actual import RSS has never been measured (no Docker runtime was available in either the environment that set mem_limit: 4g or the one that wrote this section). Capturing a real docker stats climada-sidecar reading at first activation, and tightening or raising mem_limit against it, is exactly the ops work tracked by issue #1051.

Proving it works¶

uv run python scripts/verify_engine_agreement.py \
    --base-url http://localhost:8000 --hazard <hazard_dataset_id> \
    [--exposure <exposure_dataset_id>] [--impf <impact_function_id>]

Runs one scenario through every available engine and exits 0 on agreement within tolerance. See RUNBOOK.md → "Multi-engine agreement verification" for the full exit-code contract and the triage entry for a sidecar that reports itself unavailable or fails a job.

Enabling the Oasis engine (opt-in profile)¶

The oasis compose profile (oasis-sidecar in docker-compose.prod.yml, ADR-084) serves X-Engine: oasis the same way the CLIMADA profile above serves climada: opt-in, off by default, built on the host and never pushed to any registry. Everything in the CLIMADA section applies with the names swapped. This section records the names and the one place the mechanism differs. First activation on the live host, and the real docker stats reading its memory limit is checked against, are ops work tracked as issue #1499.

The Oasis keys¶

.env.example.prod declares OASIS_SIDECAR_URL (commented out), OASIS_SIDECAR_TIMEOUT_SECONDS=900 and OASIS_SIDECAR_TOOL_TIMEOUT_SECONDS=900 below the shared tail marker (# --- Optional - omit to keep the default ---), so all three stay outside the 33-key spine tests/test_env_template_parity.py enforces. Unset, or a bare OASIS_SIDECAR_URL= line, keeps oasis registered but unavailable. Setting OASIS_SIDECAR_URL=http://oasis-sidecar:8200 (the in-network service name and port) turns it on, and docker-compose.prod.yml passes both keys through to api and worker. OASIS_SIDECAR_TIMEOUT_SECONDS (default 900, matching Settings.oasis_sidecar_timeout_seconds) is how long the worker's HTTP client waits for one job before it records E_ENGINE_UNAVAILABLE. One job runs up to five Oasis tools in sequence (csvtobin three times, evepy, gulmc), so a per-tool bound alone would let the sidecar compute on for five times that bound after the worker gave up. The prod compose file therefore passes the same key to the sidecar as OASIS_SIDECAR_JOB_TIMEOUT_SECONDS, a deadline all of the job's tools draw from: each tool gets what is left of it, and no tool starts once it is spent. OASIS_SIDECAR_TOOL_TIMEOUT_SECONDS (default 900) caps any single tool inside that deadline. It can only tighten one tool, never extend the job.

What the deploy does when the Oasis key is set¶

deploy-prod.yml reads OASIS_SIDECAR_URL from .env.prod on the host, in every step that assembles $P (backbone, UI, the on-failure dump), and appends --profile oasis when it is non-empty. The backbone step then builds oasis-sidecar before the api/worker roll, so a failed build fails the deploy, and up -d --remove-orphans keeps the sidecar running because it reuses that $P. The run log shows -> compose flags: … --profile oasis and -> building oasis-sidecar (OASIS_SIDECAR_URL is set). With the key unset every deploy stays byte-identical to one without this profile.

The Upload the Oasis sidecar build context to the host step ships the build context to ${DEPLOY_DIR}/docker/oasis-sidecar/ on every deploy, unconditionally, like the CLIMADA one. The difference: it does not name the files. git ls-files enumerates every tracked file under docker/oasis-sidecar/ at run time, they travel as one tar archive whose sha256 is checked on the host before anything is unpacked, and each file is then installed with the CLIMADA step's discipline (unchanged files skipped, previous copies kept as .bak.<UTC timestamp>, last 10 kept, atomic mv). A module added to the directory therefore ships on the next deploy with no workflow edit. Two things differ from CLIMADA, so that the host's context always holds exactly the tracked set and nothing stale can reach an image whatever the Dockerfile copies. The backups go to the sibling ${DEPLOY_DIR}/docker/oasis-sidecar.bak/, outside the build context. A file deleted from the repo is backed up there and removed from the live context.

scripts/verify-deploy.sh probes the sidecar's /health from the worker container (step 7), but only when run from the deploy directory on the host with the key set. Anywhere else it prints a skip reason.

Building the Oasis sidecar once by hand¶

docker compose $P --profile oasis build oasis-sidecar
docker compose $P --profile oasis up -d oasis-sidecar

This builds the Dockerfile's runtime stage (no reference-model data) from whatever ${DEPLOY_DIR}/docker/oasis-sidecar/ holds. After changing the oasislmf pin or any file in that directory, dispatch a deploy first so the host's copy is fresh, then rebuild. Once the key is set, the deploy's own pre-build does this for you.

Oasis rollback semantics¶

The sidecar is pinned to image: climate-lama-oasis-sidecar:local, outside the ${IMAGE_NS}-scoped post-deploy prune, so there is no rollback by retag. To turn the engine off, remove the OASIS_SIDECAR_URL line from .env.prod (or leave it empty) and dispatch a deploy, so api and worker stop offering oasis, then stop the container explicitly:

docker compose $P --profile oasis rm -sf oasis-sidecar

To roll the sidecar itself back, copy the previous .bak.* files from docker/oasis-sidecar.bak/ over their live names in docker/oasis-sidecar/ (or deploy an older ref of this repo) and rebuild with the two commands in Building the Oasis sidecar once by hand.

The Oasis adapter code in api/worker ships with the CLIMATE_LAMA_TAG image and follows the normal Rollback Procedure.

Oasis memory limit¶

docker-compose.hetzner.yml caps the sidecar at mem_limit: 2g, marked [ASSUMPTION] in that file: no oasislmf ground-up run's RSS had been measured when it was set. With the CLIMADA sidecar's 4g it totals 6 GiB, inside the ~8 GiB of measured headroom on the CX43 (see Memory footprint and host budget). Issue #1499 records the first real docker stats reading and adjusts the limit.

Live Event Mode on prod (opt-in keys)¶

Same shape as the CLIMADA section above: two optional keys, both below each template's own tail marker (# --- Optional - omit to keep the default --- in .env.example.prod; # --- Dev-only / tuning --- in .env.example — the two files use different marker text for the same structural boundary), absent from the 33-key spine tests/test_env_template_parity.py enforces.

The two keys¶

LIVE_MODE_ENABLED (issue #1018, Phase 13 W0 "Dark landing") is Settings.live_mode_enabled's default-off master switch for the live-event-replay feature line; docker-compose.prod.yml passes it through as ${LIVE_MODE_ENABLED:-false}. FIRMS_MAP_KEY is a narrower, operator pack-side key: it authenticates scripts/packs/firms_bt_event.py's calls to the FIRMS Area CSV API when fetching the Evia 2021 wildfire replay offline of any ingest request. The platform itself never reads it — there is no Settings.firms_map_key — so leaving it unset changes no runtime behaviour; it only blocks the pack's own fetch subcommand (plan and rasterize work without it). It is free but registration-gated (https://firms.modaps.eosdis.nasa.gov/api/area/ → "Get MAP_KEY").

Host-side step — done¶

Both keys were hand-edited into .env.prod on 2026-09-02, below the 33-key spine: LIVE_MODE_ENABLED=true and FIRMS_MAP_KEY=…. Both reached the running api/worker containers the same day. Live mode has been on since; the UI's Events section still stays dark (per the dark-landing design record) until a live event exists for a portfolio — no live event has been created on prod yet. FIRMS_MAP_KEY on the host does not by itself mean a FIRMS download has run: as of this writing no operator has executed scripts/packs/firms_bt_event.py fetch on prod (tracked on issue #1030).

Staging a replay series on prod¶

A replay feed releases a directory of archived provider rasters on their original clock. The repo vendors one series (a single Evia 2021 acquisition day) for the smoke stack; a prod walk that wants a moving map needs a multi-observation series, and prod has none until an operator stages one. This is the recipe the Phase 15 phase-end walk had to derive (mid-review F-5).

Where it goes. live-worker and worker both mount ./data:/data:ro, so a series placed at <deploy dir>/data/live_series/<name>/ on the host is /data/live_series/<name> inside the containers, and that container path is what series_dir takes. The API validates series_dir lexically against the configured roots — settings.dataset_source_download_dir and settings.data_mount_root (/data) — so anything outside them is refused at feed-create time, before any filesystem call. /data is read-only in the containers: stage on the host, never from inside one.

The layout. manifest.json plus the rasters it names, one entry per observation, in release order — the schema is in CLIMATE_LAMA.md § "The three feed kinds". Every entry's valid_at must be the provider's real observation instant (it is written onto the snapshot verbatim), file must be a bare filename in the same directory, and source / license / citation / source_url are required — source is capped at 255 characters and a longer one is refused here rather than dying on the insert later.

# On the prod host, in the deploy directory.
mkdir -p data/live_series/evia_2021_daily
# copy the daily rasters + a manifest.json naming them, then:
./dc exec -T live-worker ls -l /data/live_series/evia_2021_daily </dev/null

The Phase 15 walk built its six-observation series from the Phase 13 daily FIRMS rasters already on the host (data/firms-evia/<date>/*_bt.tif), taking each entry's valid_at from that day's source_manifest.json.

Cadence versus the tick. The feed's cadence_seconds decides whether a feed is due; live_feed_tick_seconds (prod: 300) is the floor at which anything is polled at all. A feed configured at 120 s therefore releases one observation every 300 s while the UI still says "Every 2 minutes" — a reporting defect, not a loop defect (#1444). Size the cadence at or above the deployment's tick unless you want that skew. Once the cursor passes the last entry, every tick logs live_feed.replay.series_exhausted and releases nothing.

Branding defaults (white-labelling an unconfigured org)¶

GET /v1/orgs/me/branding serves nine BRAND_DEFAULT_* settings (src/climate_lama/config.py, issues #1190 and #1321) to any organization that has configured no branding of its own — every organization on a fresh install. core/branding.py assembles them from Settings rather than from literals in the endpoint, so a deploy can white-label that never-configured payload (its two accent colours, its app name, theme mode, locale) without a code change. An org that has stored any branding of its own is served that stored blob whole instead — the endpoint never merges these defaults into it, not even per-field (see docs/CLIMATE_LAMA.md, "GET /v1/orgs/me/branding — defaults instead of 404").

Both templates document all nine keys as a commented "Branding defaults" block below their own tail marker, same structural region as the CLIMADA and Live Event Mode blocks above — every line already reproduces the built-in default, so the block is informational until an operator uncomments and edits a line. Five keys carry a real platform default (BRAND_DEFAULT_PRIMARY_COLOR, BRAND_DEFAULT_SECONDARY_COLOR, BRAND_DEFAULT_APP_NAME, BRAND_DEFAULT_THEME_MODE, BRAND_DEFAULT_LOCALE); the remaining four (BRAND_DEFAULT_LOGO_URL, BRAND_DEFAULT_FAVICON_URL, BRAND_DEFAULT_ORG_DISPLAY_NAME, BRAND_DEFAULT_SUPPORT_URL) are None by design — a logo, a favicon, an org display name and a support URL identify one specific organization, and the platform has no default for one, so the template leaves them blank rather than inventing a value. tests/test_env_template_branding.py asserts every brand_default_* Settings field has a line in both templates, with the non-None five matched against the live Settings default rather than a hand-typed guess.

API connection pool and the vector tile guard¶

Six keys size the API's database pool and bound its vector tiles (issue

1804, `src/climate_lama/config.py`). All six are optional: unset, the API¶

runs on the defaults below. Both templates declare them at those defaults below their tail marker.

Key	Default	Meaning
`DB_POOL_SIZE`	`5`	Connections each API process keeps open.
`DB_MAX_OVERFLOW`	`10`	Further connections a process opens under load, so its pool is `DB_POOL_SIZE + DB_MAX_OVERFLOW` (15).
`DB_POOL_TIMEOUT_SECONDS`	`30`	How long a request waits for a pooled connection before it is answered `503 E_SERVICE_BUSY`.
`VECTOR_TILE_STATEMENT_TIMEOUT_MS`	`5000`	PostgreSQL cancels a vector tile statement that runs past this; the tile is `503 E_SERVICE_BUSY` with `details.reason` `tile_query_timeout`.
`VECTOR_TILE_MAX_CONCURRENCY`	`8`	Vector tile requests in flight per API process; one more waits for a slot.
`VECTOR_TILE_SLOT_WAIT_MS`	`2000`	How long a tile request past the cap waits for a slot, holding no pooled connection, before it is refused as `503 E_SERVICE_BUSY` with `details.reason` `tile_concurrency_cap`. `0` refuses it at once.

A tile past the cap waits, because a refused tile stays undrawn. One map view asks for more vector tiles at once than the cap (12 a source against 8), and the map does not ask again for a tile answered 503: it draws that tile on the next pan or zoom. So a request past the cap waits up to VECTOR_TILE_SLOT_WAIT_MS for a tile in flight to finish. It waits before it checks out a connection, so waiting tiles cost the pool nothing.

The cap stays below the pool. VECTOR_TILE_MAX_CONCURRENCY must be less than DB_POOL_SIZE + DB_MAX_OVERFLOW, or tile requests alone could hold every connection — the outage of 2026-10-12, when fifteen slow tiles left every other route answering 503. The API refuses to start on a cap at or above the pool (a Settings validation error naming vector_tile_max_concurrency), so raise the pool before the cap. At the defaults tiles hold at most 8 of 15 connections and 7 stay free.

The pool is per API process and only the API's async engine reads it; the workers' engines are unchanged. PostgreSQL's max_connections has to cover the pool of every API process plus the workers, so raising DB_POOL_SIZE or DB_MAX_OVERFLOW is a database-side decision too. The keys reach the api container through env_file (.env in docker-compose.yml, .env.prod in docker-compose.prod.yml); to read the values in force:

docker compose run --rm --no-deps api python -c "from climate_lama.config import get_settings; s = get_settings(); print(s.db_pool_size, s.db_max_overflow, s.db_pool_timeout_seconds, s.vector_tile_statement_timeout_ms, s.vector_tile_max_concurrency, s.vector_tile_slot_wait_ms)"
# 5 10 30 5000 8 2000

What the two refusals mean on a live host, and what to look at first: RUNBOOK.md, the E_SERVICE_BUSY entry.

Rollback Procedure¶

Identify the last good release tag (e.g. v0.3.0).
Point .env.prod at it: set CLIMATE_LAMA_TAG=v0.3.0.
Pull and restart — no rebuild, the image already exists in GHCR ($P is the both-files shorthand from Release → Production):
```
docker compose $P pull api worker
docker compose $P up -d
```
A one-step rollback needs no network: the deploy workflow deliberately keeps the previous release's images on the host (see Image retention), so the pull is a no-op. Going back two or more releases does re-pull from GHCR.

Roll back migrations if the schema changed:

docker compose $P exec api alembic downgrade -1

Repeat -1 for each migration to undo, or target a specific revision:

docker compose $P exec api alembic downgrade <revision>

Verify the rollback landed — same identity check, with the previous tag:
```
./scripts/verify-deploy.sh v0.3.0
```
A rollback that silently kept serving the bad image is the failure mode this catches; /health/live cannot.

Monitoring (optional)¶

A Grafana + Prometheus overlay is available for local observability. It layers on top of docker-compose.yml without modifying it.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

Prometheus UI: http://localhost:9090 — scrapes api:8000/metrics every 15s and loads the JobFailureRateHigh alert rule.
Grafana: http://localhost:3000 — login admin / admin. The preconfigured Climate-Lama dashboard is auto-provisioned with request rate, 5xx error rate, job completions, and p95 job duration panels.

Logs¶

On the live host use the overlay shorthand $P (from Release → Production) so the ui service is included:

# All services
docker compose $P logs -f

# Single service
docker compose $P logs -f api

Deployment Guide¶

Production environment — at a glance¶

Topology¶

Release pipeline: local to production¶

Dependency delivery after the lock pin¶

Backbone (API / worker)¶

UI (climate-lama-ui)¶

Deploy identity (what verify-deploy.sh relies on)¶

Prerequisites¶

Registry Authentication¶

Token rotation¶

TLS Certificates¶

Self-signed (development / smoke-test)¶

Let's Encrypt (production)¶

First Deploy¶

Database roles and row-level security¶

The two-role model¶

The two runtime lanes¶

Grant set¶

Running the bootstrap¶

The env flip¶

Pre-flip blockers — cleared¶

Verifying isolation¶

Environment bootstrap (organization + admin)¶

Verifying a clean clone — the end-to-end walk¶

1047's coverage rule restricts which datasets a coordinate can resolve to¶

Resetting a user's password¶

Wiping and re-provisioning a dev stack from scratch¶

Update Flow¶

Automated deploy (GitHub Actions)¶

Compose file delivery¶

Orphaned containers¶

Image retention on the host¶

Required secrets¶

In-flight Celery messages: deploy workers before producers¶

Hetzner shared-host deployment (behind shared Caddy)¶

DNS records (Namecheap)¶

Caddy wiring (exposing a host)¶

Provisioning an org + user¶

Bootstrapping the first platform admin (#828)¶

Optional: UI as a compose service (dev / smoke-test)¶

Docs Site (Cloudflare Pages)¶

How it works¶

One-time setup (already done)¶

Rotating credentials¶

Triggering a deploy manually¶

Custom domain (optional)¶

Backup Strategy¶

The MinIO images and the re-pull step¶

What is (and isn't) covered¶

Running a backup¶

The dump is verified before the run is called a success¶

Off-host copy (--push-remote)¶

Retention¶

Scheduling (prod host)¶

Restore procedure¶

Exercised restore drill¶

Hetzner Object Storage (production object storage)¶

Credentials¶

One-time setup (done 2026-07-28)¶

Geocoding in production¶

The two geocoder flags¶

Enable address search¶

Why the 503 is safe to ship¶

What standing Nominatim up would actually cost¶

Host-side step — done¶

CLIMADA sidecar on the deploy path (opt-in profile)¶

Enabling the CLIMADA engine (opt-in profile)¶

The two keys¶

What the deploy does when the key is set¶

Building it once by hand¶

Rollback semantics¶

Celery workers and queues¶

Memory footprint and host budget¶

Proving it works¶

See also¶

Enabling the Oasis engine (opt-in profile)¶

The Oasis keys¶

What the deploy does when the Oasis key is set¶

Building the Oasis sidecar once by hand¶

Deploy identity (what `verify-deploy.sh` relies on)¶

Off-host copy (`--push-remote`)¶

1804, `src/climate_lama/config.py`). All six are optional: unset, the API¶