Deployment Guide¶

The canonical reference for deploying Climate-Lama to production and operating the live environment. It answers three things: where everything lives (the inventory), how a change travels from your laptop to production (the end-to-end release flow), and the host-specific setup and day-2 operations.

The live production stack runs on a Hetzner host behind a shared Caddy reverse proxy (automatic TLS), with object storage on Hetzner Object Storage. The repo's docker-compose.prod.yml also ships a portable nginx + certbot path for a fresh host with no existing proxy (see TLS Certificates / First Deploy).

Production environment — at a glance¶

Single source of truth for "where is X / who owns Y". No secret values here — only locations and owners. Secret values live in .env.prod on the host (mode 600) and in your password manager; this table only says where to look.

Layer	What / where	Identifier	Managed in	Notes
Domain (registrar)	`climate-lama.online`	—	Namecheap	bought + auto-renewed at Namecheap
DNS (authoritative)	A records, edited by hand	—	Namecheap → Advanced DNS	NOT Hetzner DNS, NOT Cloudflare. `api` → server; `@` + `www` → server
Server	Hetzner VPS	`159.69.211.124` (`lamaface-dev-01`)	Hetzner Cloud console	single-tenant — runs only the Climate-Lama stack (see note below)
Edge / TLS	shared Caddy (`:80`/`:443`)	`/opt/caddy/Caddyfile`	on the host	automatic Let's Encrypt + renewal; reverse-proxies by container name on `caddy_net`
API	backbone `api` service	`https://api.climate-lama.online`	`/opt/climate-lama`	pinned image `…/climate-lama:${CLIMATE_LAMA_TAG}`
UI	`climate-lama-ui` SPA	`https://climate-lama.online` + `www`	`/opt/climate-lama` (overlay)	pinned `…/climate-lama-ui:${UI_TAG}`; same-origin, proxies `/v1` to api
Compute	`worker` (Celery) + `climate-lama-engine`	—	inside the worker image	engine is a pip dependency baked in, not a service
Database	PostgreSQL + PostGIS	container + `postgres_data` volume	on the host	back up with `pg_dump` (see Backups)
Cache / queue	Redis	container + `redis_data` volume	on the host	Celery broker + result backend
Object storage / datasets	Hetzner Object Storage	bucket `climate-lama-storage`, location `nbg1`	Hetzner Cloud console	endpoint `nbg1.your-objectstorage.com` — all datasets / tiles / reports live here
Container registry	GHCR (private)	`ghcr.io/cortomaltese3/climate-lama{,-worker,-ui}`	GitHub Packages	host authed via PAT `ghcr-pull-hetzner` (`read:packages`) — rotate before expiry: secret-rotation runbook
Docs site	MkDocs → Cloudflare Pages	`climate-lama.pages.dev`	Cloudflare	build/publish only; unrelated to the app runtime
Repos	backbone / UI / engine	`climate-lama` · `climate-lama-ui` · `climate-lama-engine`	GitHub	see Release flow

Seeded login (pointer, not a secret store): org northlane, user develama@protonmail.com (role org_admin). The password is held out-of-band (not committed); reset it with the psql snippet in Demo data seeder (substitute the email).

Host file layout

/opt/climate-lama/ — docker-compose.prod.yml + docker-compose.hetzner.yml (overlay) + .env.prod (mode 600)
/opt/caddy/ — the shared edge (docker-compose.yml + Caddyfile)

Note (2026-08-03): lama-chat — a previously co-tenant app on this host — was removed between 2026-07-28 (last seen in a Caddyfile backup) and 2026-08-02, and verified absent again on 2026-08-03: no lama-chat container in docker ps, no /opt/lama-chat/ directory, and no chat.lamaface.site block in the live /opt/caddy/Caddyfile (which is down to the two Climate-Lama vhosts only). It is treated as decommissioned. The box is single-tenant until further notice — if lama-chat (or any other app) is ever reintroduced, restore the co-tenant cautions this note replaced (don't disrupt its container/volume/Caddy block, and re-check disk headroom before any large operation).

Topology¶

                         Internet
                            │  :443 (TLS, auto Let's Encrypt)
                   ┌────────▼─────────┐
                   │   Caddy (edge)   │   /opt/caddy/Caddyfile
                   └──┬────────────┬──┘
        api.climate-lama.online    climate-lama.online / www
                   │                        │
            ┌──────▼──────┐          ┌───────▼───────┐
            │  api :8000  │◄──/v1────│  ui  (nginx)  │   same-origin SPA
            └──┬───┬───┬──┘  proxy   └───────────────┘
               │   │   └──────────────► Hetzner Object Storage  (datasets/tiles)
               │   └────► Redis ◄────── worker (Celery + climate-lama-engine)
               └────► PostgreSQL+PostGIS

Release pipeline: local to production¶

How a fix or feature travels from your laptop to the live site. You never edit code or build images on the server — the host only pulls immutable, CI-built images and switches to the new tag. Two artifacts ship independently: the backbone (api + worker) and the UI.

All host commands below assume this shorthand (the live host uses the Caddy overlay, so both -f files are always passed):

cd /opt/climate-lama
P="-f docker-compose.prod.yml -f docker-compose.hetzner.yml --env-file .env.prod"

Interactively, prefer /opt/climate-lama/dc — scripts/dc in this repo, installed on the host as /opt/climate-lama/dc (chmod +x). It supplies all three flags, so ./dc ps, ./dc logs -f api, ./dc exec -T api alembic current are always correct from any directory.

Dropping --env-file is the recurring mistake, and it fails misleadingly:

$ docker compose -f docker-compose.prod.yml exec -T api python scripts/create_app_role.py
WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.
error while interpolating services.api.image: required variable CLIMATE_LAMA_TAG
is missing a value: set CLIMATE_LAMA_TAG in .env.prod

.env.prod does define CLIMATE_LAMA_TAG. The env_file: key feeds container environments; it does not feed compose's own ${...} interpolation, and only --env-file does that. Silver lining: the failure is an interpolation error at parse time, so a mistyped down aborts before touching a container.

Backbone (API / worker)¶

Preflight — prove every credential the runbook depends on, before doing any work. Each check below is a dependency of a later step, and the GHCR PAT expires; running them first turns an end-of-pipeline unauthorized into a minute-one failure. HOST is the prod box (root@159.69.211.124, or your ssh-config alias):

HOST=root@159.69.211.124

ssh -o BatchMode=yes "$HOST" true                  # 1. SSH auth to the host works
gh auth status                                     # 2. GitHub CLI token still valid
ssh "$HOST" 'docker manifest inspect \
  ghcr.io/cortomaltese3/climate-lama:latest >/dev/null'   # 3. GHCR reachable + host still logged in

→ verify: all three exit 0. Any failure → stop and fix it before step 1.

(1) fails → SSH key/agent problem; you cannot deploy at all.
(2) fails → gh auth login (needed to watch CI and cut the release).
(3) fails with unauthorized/denied → the host's GHCR PAT has expired or was revoked: re-run the docker login ghcr.io in Registry Authentication with a fresh read:packages token. Check 3 downloads no layers, so it is safe to repeat.
Change it locally — branch, implement, write tests, lint (ruff + mypy) per CLAUDE.md. → verify: tests green locally.
PR → CI → merge to main — open a PR; CI must be green before it merges. → verify: CI green, PR merged.
Cut a release tag — this is the step that builds and publishes the images:
```
git tag v0.5.0 && git push origin v0.5.0
```
release.yml builds …/climate-lama:v0.5.0 and …/climate-lama-worker:v0.5.0 and pushes them to GHCR. → verify: the release workflow is green and the tag exists in GHCR.
Deploy — primary path: dispatch the deploy workflow (see Automated deploy); it does steps 4–5 for you, including the migration:
```
gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0
```
→ verify: the run is green (it health-gates and identity-checks itself).

Fallback — by hand on the host (no git pull, no rebuild):

sed -i 's/^CLIMATE_LAMA_TAG=.*/CLIMATE_LAMA_TAG=v0.5.0/' .env.prod
docker compose $P pull api worker
docker compose $P up -d

→ verify: docker compose $P ps shows api/worker healthy on the new tag. 5. Migrate — the workflow already did this; only needed on the manual fallback path (and only if the release added migrations). Run with the new image's alembic:

docker compose $P exec api alembic upgrade head

→ verify: alembic current == head. 6. Verify the deployed identity — the deploy is not done until this passes. Run it from a checkout of this repo on your laptop (it resolves the tag locally):

git fetch --tags
./scripts/verify-deploy.sh v0.5.0
# non-default target: ./scripts/verify-deploy.sh v0.5.0 https://api.example.com

The script checks /health/ready, reads the git_sha baked into the running image from /v1/info, resolves v0.5.0 to a commit locally, and compares the two. → verify: it exits 0 and prints ✓ identity: … serves v0.5.0.

Exit 1 tells you which failure you have: host unreachable, service not ready, git_sha mismatch (prints served vs expected — the host is still running the old image: re-check CLIMATE_LAMA_TAG, that pull actually fetched, and that up -d recreated the container), git_sha unknown (the image was built before deploy identity existed, or without the build-arg), or the tag not existing locally (git fetch --tags).

A bare liveness curl is a weaker check and must not be used as the gate — a healthy old container passes it, which is exactly how an unbumped tag or a no-op pull goes unnoticed. Keep it only as a quick eyeball:

curl -s https://api.climate-lama.online/health/live   # {"status":"healthy"}
curl -s https://api.climate-lama.online/v1/info       # version, environment, git_sha

7. Rollback if needed — set CLIMATE_LAMA_TAG back to the previous tag, pull, up -d, and alembic downgrade if the schema moved. See Rollback Procedure.

There is a brief downtime during up -d (no rolling restart yet).

UI (climate-lama-ui)¶

Same shape, different repo and workflow:

1–2. Change + merge to main in climate-lama-ui (CI green). 3. Tag a release: git tag v0.2.0 && git push origin v0.2.0 → publish.yml builds …/climate-lama-ui:v0.2.0. (Map-tile URLs are baked at build time — tracked in #355.) 4. Deploy — primary path: dispatch the deploy workflow (see Automated deploy). A UI-only dispatch never touches api/worker and never runs migrations:

gh workflow run deploy-prod.yml -f ui_tag=v0.2.0

Fallback — by hand on the host:

sed -i 's/^UI_TAG=.*/UI_TAG=v0.2.0/' .env.prod
docker compose $P pull ui && docker compose $P up -d ui

5. → verify: curl -s https://climate-lama.online/health/live and load the site.

Deploy identity (what `verify-deploy.sh` relies on)¶

Production deliberately lags main, so the only meaningful post-deploy invariant is served identity == the release tag being deployed. It is enforceable because the commit travels with the image:

release.yml builds the api/worker images with --build-arg GIT_SHA=<commit> (and the matching org.opencontainers.image.revision / …version labels).
docker/core.Dockerfile / docker/worker.Dockerfile turn that into ENV GIT_SHA, so it is readable in the container (docker compose $P exec api printenv GIT_SHA).
Settings.git_sha reads that env var and GET /v1/info returns it next to version and environment.
scripts/verify-deploy.sh compares it with git rev-list -n1 <tag>.

Images built without the build-arg (any local docker compose build, or any image published before this was wired up) report git_sha: "unknown". That is not an error at runtime — but verify-deploy.sh treats it as unverifiable and exits 1, on purpose. Do not set GIT_SHA in .env.prod: compose's env_file would override the baked value and the check would validate a hand-typed string instead of the image.

:latest is mutable (rebuilt on every push to main). Always pin a tag in .env.prod so production is reproducible and roll-back-able. The engine (climate-lama-engine) ships inside the worker image — it has no separate release/deploy step; bumping it is a backbone dependency change + new backbone tag.

Prerequisites¶

Production deployment uses Docker Compose; the live host terminates TLS with the shared Caddy (the bundled nginx is the portable alternative).

Docker ≥ 24.0 (docker --version)
Docker Compose v2 (docker compose version)
A server with ports 80 and 443 open
A domain name pointing to the server (for Let's Encrypt TLS)

Registry Authentication¶

The api/worker images (ghcr.io/cortomaltese3/climate-lama and …-worker) are private on GHCR, so the host must authenticate to the registry once before it can pull. This is a one-time setup per host — Docker caches the credential in ~/.docker/config.json and reuses it for every later pull, including updates. (CI is unaffected: the smoke workflows build images from source and never pull from GHCR.)

Create a GitHub Personal Access Token (classic) with the single read:packages scope: GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) → Generate new token (classic). Name it e.g. ghcr-pull-hetzner, set an expiry, check only read:packages, and copy the ghp_… value.

Log in on the host, as the same user that runs docker compose (use root / sudo if you deploy with sudo docker compose):

echo "ghp_xxxxxxxxxxxx" | docker login ghcr.io -u CortoMaltese3 --password-stdin
chmod 600 ~/.docker/config.json   # the token is stored base64, not encrypted

Verify (once the target release tag is published — see release process):
```
docker pull ghcr.io/cortomaltese3/climate-lama:v0.4.0
```

Token rotation¶

The PAT expires on the date you chose. Set a reminder to regenerate it and re-run the docker login above before it lapses — otherwise the next update deploy fails with unauthorized on docker pull, on a host you rarely touch. See ADR-038.

The step-by-step procedure — mint, log in, verify the new token can actually read all three private repositories, then revoke the old one — is docs/ops/secret-rotation.md §1. That runbook also covers the prod-host deploy keypair, the application secrets in .env.prod (DB passwords, MinIO credentials, APP_SECRET_KEY, API_KEY_PEPPER) and the SDK_SMOKE_* repo secrets.

TLS Certificates¶

Self-signed (development / smoke-test)¶

Generate a self-signed cert into nginx/certs/:

mkdir -p nginx/certs
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout nginx/certs/server.key \
  -out nginx/certs/server.crt \
  -subj "/CN=localhost"

Let's Encrypt (production)¶

Install certbot on the host.
Stop nginx if running: docker compose -f docker-compose.prod.yml stop nginx

Obtain a certificate:

certbot certonly --standalone -d your-domain.example.com

Copy the issued files into nginx/certs/:

cp /etc/letsencrypt/live/your-domain.example.com/fullchain.pem nginx/certs/server.crt
cp /etc/letsencrypt/live/your-domain.example.com/privkey.pem   nginx/certs/server.key

Restart nginx: docker compose -f docker-compose.prod.yml start nginx

Renew manually when needed (certbot renew), then repeat step 4-5. Automate with a cron job:

0 3 * * * certbot renew --quiet && \
  cp /etc/letsencrypt/live/<domain>/fullchain.pem /path/to/project/nginx/certs/server.crt && \
  cp /etc/letsencrypt/live/<domain>/privkey.pem   /path/to/project/nginx/certs/server.key && \
  docker compose -f /path/to/project/docker-compose.prod.yml restart nginx

First Deploy¶

Clone the repository onto the server and cd into it. The UI is a separate deployable in the climate-lama-ui repo; it consumes this backbone's API over HTTP and is deployed independently (see that repo's own deployment instructions).
Create the environment file:
```
cp .env.example.prod .env.prod
# Edit .env.prod — fill in all secrets, set CORS_ORIGINS to your domain,
# and set CLIMATE_LAMA_TAG to the release you want to deploy (e.g. v0.4.0).
```
The template defaults the MINIO_* block to Hetzner Object Storage (see Hetzner Object Storage). The prod compose ships no embedded object store — MINIO_ENDPOINT must point at a real S3-compatible bucket. For a fully self-contained eval stack use the dev docker-compose.yml, which bundles MinIO.
Generate TLS certificates (see TLS Certificates above).
Start all services:
```
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d
```
Prod pulls pre-built, CI-published images from GHCR — it never builds from source. The images are private, so the host must be authenticated to GHCR first (one-time; see Registry Authentication above). The api and worker services are pinned to :${CLIMATE_LAMA_TAG}; compose fails fast if that variable is unset. The api/worker images are published by release.yml on every v* tag — make sure the tag in CLIMATE_LAMA_TAG has actually been released before deploying. See ADR-038.

Run database migrations:

docker compose -f docker-compose.prod.yml exec api alembic upgrade head

Smoke test:
```
curl -k https://<host>/health/live
# Expected: {"status": "healthy"}
```
Remove -k once you have a valid certificate. Then confirm which build is serving, not just that something is (see Deploy identity):
```
./scripts/verify-deploy.sh v0.4.0 https://<host>
```
(Optional) Seed demo data — see Demo data seeder below.

Database roles and row-level security¶

Status: APPLIED on production 2026-08-04. api, worker and beat all connect as the unprivileged climate_lama_app role, so RLS is enforced rather than inert; the maintenance lane keeps the privileged role for the two jobs that need it. Closed on #556, which carries the preflight matrix and the post-flip evidence.

Until that date the flip described below was not actually performable from .env.prod — the compose file hardcoded the runtime DSNs and silently overrode the file, so following the old procedure produced a healthy stack still running as the superuser. APP_DATABASE_URL / APP_DATABASE_SYNC_URL exist to fix that; see the callout under "The two-role model". The procedure below is kept as the reference for a fresh host — it is what a new deployment must still do.

The two-role model¶

Row-level security is enabled and forced on every org-scoped table (migration 0022 and its successors), with one policy shape everywhere:

CREATE POLICY org_isolation ON <table>
  USING (org_id = current_setting('app.current_org_id', true)::uuid)

There is exactly one GUC (app.current_org_id) and no bypass GUC, no admin policy, no WITH CHECK clause — because the policy is FOR ALL without one, USING also gates INSERT/UPDATE. Cross-org admin surfaces work today because the tables they touch (organizations, users, api_keys, refresh_tokens, audit_events, usage_by_period) are deliberately exempt from RLS, not because anything bypasses a policy.

None of that has any effect while the application connects as a SUPERUSER role with BYPASSRLS, which is what POSTGRES_USER (the cluster bootstrap role) is. Hence two roles:

Role	Attributes	Used by
`climate_lama` (`POSTGRES_USER`)	superuser, owns the schema	Alembic migrations, `scripts/seed_demo.py`, the backfills and ingest scripts, `pg_dump`/`pg_restore`, psql inspection, and the maintenance lane below

"Alembic migrations" is not automatic — it is arranged. The deploy workflow runs docker compose exec api alembic upgrade head, i.e. inside the api container, which is precisely the process the flip repoints at climate_lama_app. Migrations are DDL and climate_lama_app holds no ownership and no CREATE ON SCHEMA, so nothing would keep them on the owner by itself. migrations/env.py therefore resolves its DSN from MAINTENANCE_DATABASE_SYNC_URL first, falling back to DATABASE_SYNC_URL (see src/climate_lama/db/migration_url.py). That is why step 2 of the flip — setting the maintenance DSN to the owner — is not optional and must come before step 3: it is what keeps migrations, the usage_by_period refresh and the result_cache purge on the owner once the app DSN moves. Skip it and the flip looks fine, then the next deploy fails at alembic upgrade head. | climate_lama_app | LOGIN NOSUPERUSER NOBYPASSRLS NOCREATEDB NOCREATEROLE | the api, worker and beat runtime DSNs |

The application DSN is pure configuration (DATABASE_URL / DATABASE_SYNC_URL, read by src/climate_lama/config.py), so the flip is an env change plus a restart — no image rebuild.

Which variable actually takes effect — read this before editing anything. docker-compose.prod.yml declares DATABASE_URL and DATABASE_SYNC_URL in the environment: block of api, worker and beat, and Docker Compose gives environment: precedence over env_file:. So the DATABASE_URL= / DATABASE_SYNC_URL= lines that exist in the live .env.prod are inert — they are read from the file and then overridden. Editing them changes nothing, the stack restarts healthy, and you would conclude RLS is enforced while the app is still connecting as the cluster superuser. That is the failure this section used to walk an operator straight into.

The live knobs are APP_DATABASE_URL and APP_DATABASE_SYNC_URL, which Compose interpolates into those same entries:
DATABASE_URL: ${APP_DATABASE_URL:-postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}}
DATABASE_SYNC_URL: ${APP_DATABASE_SYNC_URL:-postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}}
Unset or empty → the owner DSN derived from POSTGRES_*, i.e. exactly the behaviour every existing deployment has today. Set → that value wins. The names are deliberately distinct from the legacy inert lines so "the variable I edited" and "the variable that took effect" can never diverge.

POSTGRES_USER / POSTGRES_PASSWORD are not a substitute: they initialize and authenticate the postgres container itself and drive scripts/backup.sh / scripts/restore.sh. Repointing them at climate_lama_app breaks the database and the backups, not just the app.

MAINTENANCE_DATABASE_SYNC_URL is different again: it has no environment: entry in any service, so env_file: .env.prod supplies it directly and setting it in .env.prod does take effect.

The two runtime lanes¶

Everything the API and the worker do falls into one of exactly two lanes.

Org-scoped lane — the default, and where all but two tasks live. Every session that acts on behalf of one organization stamps SELECT set_config('app.current_org_id', :org_id, true) on its transaction immediately after opening it, via db/rls.py::set_org_guc / set_org_guc_sync. Two properties of that call are load-bearing:

the org id is a bound parameter, never interpolated (a SET LOCAL statement takes no bind parameters — asyncpg would send = $1 and Postgres would raise a syntax error; tests/lint_no_set_local_org_guc.py pins this);
is_local => true makes the setting transaction-local, so it cannot leak onto the next tenant's checkout of a pooled connection — and, by the same token, it is discarded at every COMMIT. Code that commits and keeps working on the same session must set it again.

Work that is cross-org in aggregate but decomposable per org is written as a loop of org-scoped transactions: the org list comes from the RLS-exempt organizations table (db/rls.py::list_org_ids_sync) and each org's work runs under its own GUC. That is how the ingest watchdog sweeps, surface retention, and the "which org owns this row?" lookups in convert_to_cog / build_dataset_cog / download_to_spaces (db/rls.py::resolve_owning_org_sync) are expressed.

Maintenance lane — narrow, explicit, two users. Reserved for work that cannot be expressed per org at all. It is configured by MAINTENANCE_DATABASE_SYNC_URL; when that is unset it falls back to DATABASE_SYNC_URL, which is exactly today's behaviour, so dev, tests and the current production deployment are unaffected until you flip.

Maintenance-lane user	Why it cannot be org-scoped
`worker/tasks.py::refresh_usage_view` → `UsageEventRepository.refresh_view`	`REFRESH MATERIALIZED VIEW` needs ownership of `usage_by_period`, which `climate_lama_app` deliberately lacks — and granting it ownership would not help, because an owner without `BYPASSRLS` refreshes the view to zero rows (`usage_events` is FORCE-RLS)
`worker/tasks.py::purge_orphan_cache_entries`	`result_cache` carries no `org_id`, so "orphaned" is only definable against the union of every org's `impact_results`; scoped to one org, every other org's cache rows would look orphaned

That table is the whole list. Anything else needing the privileged role is a bug, not a third lane. purge_orphan_cache_entries additionally keeps its emptiness guard: if impact_results reads empty against a non-empty result_cache it refuses to purge and logs an error, which is what makes a misconfigured maintenance DSN safe rather than destructive.

Grant set¶

scripts/create_app_role.py applies exactly this, idempotently:

GRANT CONNECT ON DATABASE <db>                       TO climate_lama_app;
GRANT USAGE   ON SCHEMA public                       TO climate_lama_app;
GRANT SELECT, INSERT, UPDATE, DELETE
      ON ALL TABLES IN SCHEMA public                 TO climate_lama_app;
GRANT USAGE, SELECT
      ON ALL SEQUENCES IN SCHEMA public              TO climate_lama_app;
GRANT SELECT ON <each materialized view in public>   TO climate_lama_app;
ALTER DEFAULT PRIVILEGES FOR ROLE climate_lama IN SCHEMA public
      GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO climate_lama_app;
ALTER DEFAULT PRIVILEGES FOR ROLE climate_lama IN SCHEMA public
      GRANT USAGE, SELECT ON SEQUENCES               TO climate_lama_app;

Deliberately not granted: ownership, TRUNCATE, REFERENCES, TRIGGER, CREATE ON SCHEMA. The ALTER DEFAULT PRIVILEGES pair is what makes future migrations' tables reachable without re-running the script — but Postgres does not apply default privileges to materialized views, so re-run the script after any migration that adds one.

Running the bootstrap¶

Run it as the privileged role (the current DATABASE_SYNC_URL), before changing anything. The default is a dry run that prints every statement.

# on the prod host, from /opt/climate-lama
read -rs -p "new app-role password: " APP_PW; echo
docker compose -f docker-compose.prod.yml exec -T \
  -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" api \
  python scripts/create_app_role.py            # dry run — prints the statements

docker compose -f docker-compose.prod.yml exec -T \
  -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" api \
  python scripts/create_app_role.py --apply

Re-running is safe: it re-asserts the attributes, re-applies the grants and rotates the password. Verify:

SELECT rolname, rolsuper, rolbypassrls, rolcanlogin
  FROM pg_roles WHERE rolname LIKE 'climate_lama%';
-- climate_lama_app must show f | f | t

The env flip¶

Applied on production 2026-08-04 (#556); this is the reference procedure for a fresh host. Do these in order. Step 0 is a precondition; steps 1–2 are safe on their own; step 3 is the flip.

0. Assert the deployed image actually contains the code this flip depends on. This flip is not self-contained — it only works if the running image carries the commits that teach the application about the two-role model. Run:

git fetch origin
for ref in 05b3b7b 504132f 590ad17 c6c8cab; do
  scripts/assert_deployed_contains.sh "$ref" || echo "BLOCKED: $ref is not deployed"
done

commit	PR	what the flip needs it for
`05b3b7b`	#576	parameterised `set_config` org GUC
`504132f`	#577	non-superuser role support, `resolve_owning_org_sync`, seed guard
`590ad17`	#581	org-scoped GUC lane + maintenance lane in the worker
`c6c8cab`	#633	Alembic on the maintenance lane

If any line reports BLOCKED, stop and deploy first. This is not hypothetical: on 2026-08-04 the flip was applied while prod ran v0.6.0, which predated all four. MAINTENANCE_DATABASE_SYNC_URL was set on the host but did not exist anywhere in the deployed source, so the maintenance lane was inert, every scheduled task ran on the runtime DSN as the unprivileged role, and the ingest chord stamped no org GUC (#638).

The generic lesson — this is not specific to the RLS flip. A configuration change that depends on application code needs a deployed-code assertion, not just a health check. Every check run after that flip was green (/health/*, /v1/info, an unauthenticated 401, alembic current) because every one of them exercises a read path that the old image gets right. A health check tells you the container is alive; it does not tell you the container knows about the setting you just added. Any runbook step of the shape "set an env var and restart" gets a step 0 like this one.

1. Create the role — scripts/create_app_role.py --apply, as above, and confirm pg_roles shows climate_lama_app as f | f | t.

2. Point the maintenance lane at the privileged user, before flipping the app DSN. Add one line to .env.prod — spelled out, not interpolated, because env_file values are literal strings and ${...} inside them is not expanded:

MAINTENANCE_DATABASE_SYNC_URL=postgresql+psycopg2://<POSTGRES_USER>:<POSTGRES_PASSWORD>@postgres:5432/<POSTGRES_DB>

Setting it while the app DSN is still the privileged role is a no-op — both point at the same user — which is exactly why it goes first. Three things must already be on the privileged lane at the moment the app DSN changes underneath them:

the nightly refresh_usage_view and purge_orphan_cache_entries;
alembic upgrade head, which the deploy workflow runs inside the api container. migrations/env.py prefers MAINTENANCE_DATABASE_SYNC_URL over DATABASE_SYNC_URL for exactly this reason. Leave it unset and the flip itself succeeds, but the next deploy dies on the first migration that issues DDL — the failure lands one deploy after its cause.

Restart api, worker and beat so they pick it up, and confirm it landed in all three:

docker compose -f docker-compose.prod.yml --env-file .env.prod up -d api worker beat
for svc in api worker beat; do
  docker compose -f docker-compose.prod.yml --env-file .env.prod exec -T "$svc" \
    printenv MAINTENANCE_DATABASE_SYNC_URL
done

Then prove migrations still resolve to the owner — before the app DSN moves, so a wrong answer here costs nothing:

docker compose -f docker-compose.prod.yml --env-file .env.prod exec -T api \
  alembic current      # must print the current revision and `(head)`

3. Flip the app DSN — add APP_DATABASE_URL / APP_DATABASE_SYNC_URL to .env.prod. This is the whole flip; docker-compose.prod.yml needs no edit, and neither does anything else in the repo:

APP_DATABASE_URL=postgresql+asyncpg://climate_lama_app:<APP_PW>@postgres:5432/<POSTGRES_DB>
APP_DATABASE_SYNC_URL=postgresql+psycopg2://climate_lama_app:<APP_PW>@postgres:5432/<POSTGRES_DB>

Leave POSTGRES_USER / POSTGRES_PASSWORD alone — they stay the owner, because they also initialize the postgres container and drive the backup scripts. Do not bother editing the legacy DATABASE_URL / DATABASE_SYNC_URL lines: as explained above they are inert under this compose file.

Both variables must be set together, and both must name the same role — the async DSN serves the api, the sync DSN serves the worker, and a half-flip leaves the two runtimes disagreeing about which role they connect as.

Verify the render before restarting anything. config resolves the file exactly the way up will, at zero risk:

docker compose -f docker-compose.prod.yml -f docker-compose.hetzner.yml \
  --env-file .env.prod config | grep -E 'DATABASE_(SYNC_)?URL'
# every api/worker/beat DATABASE_URL and DATABASE_SYNC_URL must now read
# climate_lama_app, and MAINTENANCE_DATABASE_SYNC_URL must still read the owner

Also confirm CL_SEED_DEMO is unset or false in .env.prod. If it is left true, startup now logs an error and skips the seed rather than half-seeding — correct, but it means the demo data must be seeded out of band with the privileged DSN. The seeder's CLI refuses under the app role too (exit code 2), so docker compose exec api python scripts/seed_demo.py post-flip is a loud refusal rather than a half-seed — see Demo data seeder.

4. Restart only the app services — the database is untouched:

docker compose -f docker-compose.prod.yml up -d api worker beat
curl -fsS https://api.climate-lama.online/health/ready     # exercises a real DB round-trip

5. Verify. Four checks, in this order:

# a) the role really is unprivileged, and the app connects as it
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname LIKE 'climate_lama%';"
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT usename, count(*) FROM pg_stat_activity WHERE datname = current_database()
     AND usename IS NOT NULL GROUP BY 1;"
# api/worker connections must now show up under climate_lama_app

b) one ingest end to end. Upload a small hazard through POST /v1/hazards/ingest and poll GET /v1/hazards/ingest-jobs/{id} to succeeded. This is the chord's whole session surface (stage → validate → plan → write → aggregate → commit) in one shot; a missed GUC shows up as a job that never leaves its state or a dataset committed with zero centroids.
c) one matrix run. build_reference_surfaces with a 2×2 matrix — the task that exposes the "GUC dies at COMMIT" failure, since only cells after the first would misbehave.
d) the nightly tasks. Trigger them by hand rather than waiting for beat, and read the logs:

docker compose -f docker-compose.prod.yml exec -T worker \
  python -c "from climate_lama.worker.tasks import refresh_usage_view, purge_orphan_cache_entries; \
             print(refresh_usage_view.apply().get(), purge_orphan_cache_entries.apply().get())"

purge_orphan_cache_entries returning {"skipped": true} means the guard fired — the maintenance DSN is not privileged. Fix step 2 before proceeding. Then confirm GET /v1/admin/usage still returns non-empty aggregates, which is the check that the matview refresh saw rows. - e) migrations still reach the owner. The one check whose failure would otherwise surface at the next deploy rather than now:

docker compose -f docker-compose.prod.yml exec -T api alembic current
# must print the revision and `(head)` — a permission error here means
# MAINTENANCE_DATABASE_SYNC_URL is missing from the api service (step 2)
docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT usename, application_name FROM pg_stat_activity
     WHERE datname = current_database() AND usename IS NOT NULL;"

alembic current is read-only, so it passes under either role — it proves connectivity, not privilege. For the real proof, run the next deploy's alembic upgrade head and watch it succeed, or confirm the api container's MAINTENANCE_DATABASE_SYNC_URL names the owner (step 2's printenv loop).

Rollback is the inverse and takes one restart: comment out (or blank) the APP_DATABASE_URL / APP_DATABASE_SYNC_URL lines in .env.prod — the :- defaults then resolve back to the POSTGRES_* owner DSN — and up -d api worker beat again. MAINTENANCE_DATABASE_SYNC_URL can stay — it then simply names the same role as the app DSN. The climate_lama_app role can be left in place; it is inert when nothing connects as it.

Pre-flip blockers — cleared¶

The blocker list audited on 2026-08-03 has been implemented; each item below is now the shipped behaviour rather than a warning. Every one of them was a silent behaviour change under the new role, not a permission error, which is why they gated the flip.

Was	Now
The ingest chord opened ~20 sessions with no GUC across `worker/ingest/{pipeline,stage_source,validate_source,plan_chunks,write_chunk,aggregate_and_commit}.py`, all touching FORCE-RLS `ingest_jobs` / `ingest_chunks`	Every one of those sessions stamps the org GUC on its transaction. `write_chunk`'s chunk helpers now take `org_id` explicitly — `IngestChunkRepository` keys on `ingest_job_id` alone, so nothing else carried the org
`worker/ingest/watchdog.py` (`_find_wedged_jobs`, `_find_wedged_writing_jobs`) scanned cross-org with no GUC → permanent no-op	Both candidate scans iterate `organizations` and run one org-scoped transaction per org
`build_reference_surfaces` set the GUC once, then COMMITted inside its loop → every cell after the first ran GUC-less	The GUC is re-set at the top of every matrix-cell iteration
Unfiltered "peek" reads on `hazard_datasets` in `convert_to_cog` / `build_dataset_cog`	`db/rls.py::resolve_owning_org_sync` locates the owning org by probing each org's own scope; the follow-up row read is then org-scoped
`REFRESH MATERIALIZED VIEW CONCURRENTLY usage_by_period` needs ownership, and a non-`BYPASSRLS` owner would refresh to zero rows	Moved to the maintenance lane (`MAINTENANCE_DATABASE_SYNC_URL`)
`purge_orphan_cache_entries` would empty `result_cache` wholesale	Moved to the maintenance lane, keeping the #577 emptiness guard
`CL_SEED_DEMO=true` ran the cross-org seeder inside the API process as the app role	Startup checks `pg_roles` for the connected role and, if it is neither superuser nor `BYPASSRLS`, logs a clear error and skips the seed rather than half-seeding

Found beyond that list and fixed here:

download_to_spaces had the same unfiltered peek on downloads (FORCE-RLS since migration 0041) that convert_to_cog had; the download would have looked deleted. Same resolve_owning_org_sync treatment.
The climate_lama_surface_* ORG-plane gauges were computed with one unscoped rollup at the end of the retention sweep, which under the new role would have reported only the last org visited. refresh_surface_gauges now takes the sweep's org list and sums the ORG plane one RLS scope at a time; the org-less REFERENCE plane is still read once.

Everything else audited clean: login/refresh/API-key paths, /health*, core/audit.py, core/notifications.py, api/v1/admin/*, worker/dataset_polling.py and worker/dataset_ingest.py (they touch pending_ingests, which carries no RLS), and db/repositories/org_download_allowlist_repository.py (which carries its own explicit org_id predicate as belt-and-braces).

Verifying isolation¶

tests/db/test_app_role_rls_isolation.py is the load-bearing proof: it creates its own throwaway NOSUPERUSER / NOBYPASSRLS role, seeds two orgs, and shows that unfiltered SELECTs against hazard_datasets / hazard_centroids return nothing without the GUC and only the caller's own rows with it. It skips when no PostgreSQL is reachable and runs for real in CI.

tests/test_worker/test_rls_lanes.py is its DB-free companion: it pins lane selection — which session factory each task opens and whether it stamps the GUC — which is the property that regresses when someone adds a new session to the worker. Add a case there when you add a session.

Demo data seeder¶

scripts/seed_demo.py (also exposed as make seed-demo) populates a fresh database with a fully worked Greek-flood scenario: org acme-demo, user demo@example.com, one exposure dataset, one hazard dataset, the JRC flood impact function, and a computed result. The UI onboarding banner and guided tour depend on this data.

It also seeds the rollup ladder's subjects — demo assets, a demo portfolio, the GADM Greece admin-0 and Attica admin-1 boundaries, and the reference cells the scores band from — so GET /v1/risk/{assets,portfolios,admin-units}/{id} return scored profiles rather than 404s.

Both boundaries are seeded on purpose (issue #721). Writing any admin_boundaries row arms the region vocabulary (core.region_vocabulary, issue #666): from the first row on, an ingest whose region matches no loaded boundary is rejected with a 422. The demo ingests a national river-flood dataset under region Greece, so the admin-0 country row has to be seeded alongside the admin-1 one — otherwise the seeder arms a guard that rejects its own next call and scripts/demo.py --seed-only exits 1 on a fresh stack. A stack seeded before this change grows the missing admin-0 row on the next seeder run (the same #467 backfill path).

The seeder is idempotent — if acme-demo already exists with at least one scenario it skips the expensive ingest/compute phases and exits 0, so it is safe to run on an existing database. It still ensures the rollup-ladder subjects on that path, so a database seeded before those existed grows them on the next run (issue #467); a fully seeded database is left unchanged.

Refuses a retired org (issue #858). If acme-demo exists but has been soft-deleted (organizations.deleted_at set — e.g. after an org retirement via scripts/migrate_org_data.py), the seeder raises immediately with a RuntimeError naming the org, instead of treating its now-empty scenario list as "needs seeding" and repopulating a decommissioned tenant. A deployment whose acme-demo has been retired needs a different org to seed against — this script only ever targets acme-demo.

Running the seeder¶

# From the host (venv activated, services up):
make seed-demo

# Or inside the API container:
docker compose exec api python scripts/seed_demo.py --verbose

Expected last log line: Demo seed complete for org acme-demo (result_id=…)

After the RLS role flip, neither invocation works from the api container. The seeder creates an organization and then writes rows belonging to it, before any app.current_org_id could name it — cross-org work the climate_lama_app role cannot do, and under FORCE-RLS those writes match zero rows silently. Both entry points check the connected role first and refuse with exit code 2 and a Refusing to seed: … error rather than half-seeding. To seed post-flip, run it with a privileged DSN:

docker compose -f docker-compose.prod.yml exec -T \
  -e DATABASE_URL="postgresql+asyncpg://<POSTGRES_USER>:<POSTGRES_PASSWORD>@postgres:5432/<POSTGRES_DB>" \
  api python scripts/seed_demo.py --verbose

Both docker/core.Dockerfile (api) and docker/worker.Dockerfile bake demo_data/ into the image at the same path, so the default invocation above needs nothing extra. If the fixtures instead live on a mounted volume (e.g. the worker's /data mount) rather than the image, point the seeder at it with --data-dir (same convention as scripts/ingest_scenario_hazards.py ingest --data-dir):

docker compose exec api python scripts/seed_demo.py --data-dir /data/demo_data --verbose

The seeder fails fast with a clear error if the exposure CSV or hazard GeoTIFF is missing at the resolved directory, instead of dispatching an unstageable path to the worker.

Setting a usable password for `demo@example.com`¶

The seeder creates the demo user with a random unguessable password hash. To log in as that user (e.g. for a browser smoke test), reset the password via psql:

HASH=$(docker compose exec -T api python -c \
  "from climate_lama.core.security import hash_password; print(hash_password('demo1234'))")
docker compose exec postgres psql -U climate_lama -d climate_lama \
  -c "UPDATE users SET password_hash = '$HASH' WHERE email = 'demo@example.com';"

Onboarding smoke check¶

After seeding and setting the password:

Open the UI and log in as demo@example.com / demo1234 — select Acme Demo org.
The first-run banner appears on the dashboard → click "Try the demo →" → map and result populate.
Advance through all 5 tour steps → click "Don't show again" → reload → banner must not reappear.

Wiping and re-seeding from scratch¶

docker compose down -v   # drops the postgres_data volume
docker compose up -d     # re-creates containers and runs migrations
make seed-demo           # seeds fresh demo data

Update Flow¶

Deploying a new release is steps 3–6 of Release → Production: cut a tag (CI publishes the image), bump CLIMATE_LAMA_TAG (or UI_TAG) in .env.prod, pull, up -d, then alembic upgrade head if the schema moved. No git pull or rebuild on the host — the image is already built by CI. Rolling restart is not configured yet, so there is a brief downtime during up -d.

The primary path is the deploy workflow below — it performs exactly those steps over SSH. The by-hand commands stay documented as the fallback for when the workflow is unavailable (missing secrets, GitHub outage, host firewall).

Automated deploy (GitHub Actions)¶

.github/workflows/deploy-prod.yml applies a release to the live host. It does not build images — release.yml (backbone) and the UI repo's publish.yml already did that; this only points .env.prod at an already-published tag and rolls the containers.

Trigger: workflow_dispatch only. Deploying to prod is an explicit, auditable click — never a side effect of a push, tag, or release. Do not add another trigger.

gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0                    # backbone only
gh workflow run deploy-prod.yml -f ui_tag=v0.2.0                          # UI only
gh workflow run deploy-prod.yml -f backbone_tag=v0.5.0 -f ui_tag=v0.2.0   # coordinated

Input	Effect
`backbone_tag`	bump `CLIMATE_LAMA_TAG` → `pull api worker` → `up -d --remove-orphans` → `alembic upgrade head` → assert `alembic current` is at head
`ui_tag`	bump `UI_TAG` → `pull ui` → `up -d --no-deps ui`. No migrations.

Both are optional but at least one is required — an empty dispatch fails immediately with a clear message. The two deployables are versioned separately, so a UI-only deploy must not disturb the backbone: the workflow passes --no-deps and then asserts the api/worker container IDs are byte-identical either side of the roll.

What it guarantees, in order:

Tag inputs are charset-restricted ([A-Za-z0-9][A-Za-z0-9._-]{0,63}) before they reach a remote shell — they are interpolated into SSH commands.
GHCR preflight runs before any .env.prod edit. docker manifest inspect (on the host, no layers downloaded) proves every requested image exists. The ordering is the point: a half-applied env bump pointing at an image that was never published leaves prod pinned to something unpullable.
.env.prod is backed up to .env.prod.bak.<UTC timestamp> before every edit (last 10 kept), and the bump is read back and verified.
concurrency: deploy-prod (queue, never cancel) serialises dispatches so two runs cannot interleave mid-edit on the host.
Orphaned containers are reconciled — the backbone roll is up -d --remove-orphans, so a service dropped from the compose file also disappears from the host. See Orphaned containers below for why this is not optional.
Health gate — backbone: /health/live must report healthy; UI: the apex must return HTTP 200. Both retry for up to 3 minutes.
Deploy identity (backbone) — the git_sha baked into the running image is compared with the tag's commit. A healthy old container passes /health/live, so this is what actually proves the tag landed (see Deploy identity). A definite mismatch fails the run; an image that predates the GIT_SHA build-arg only warns.
Superseded images are pruned (issue #579) once the gates above have passed — see Image retention below.
On failure, docker compose ps and the last 200 log lines for api/worker/ui are printed into the job output.

Orphaned containers¶

Removing a service from docker-compose.prod.yml does not remove its container from the host. Compose only ever acts on services it can see in the merged files, so a deleted service becomes an orphan: still running, still restarting on boot (restart: unless-stopped), still holding whatever connections it held — and now invisible to every pull, up, and down you run, because none of them enumerate it.

This is not hypothetical. martin was retired in #562 and its tables dropped in migration 0067; the container went on running for seven weeks. It surfaced only after the #556 RLS flip, when a connection census showed the retired service holding 17 connections as the cluster owner while api, worker and beat held 2 as the unprivileged role — a service with no consumer had become the single largest holder of BYPASSRLS connections to the production database, and the only application process RLS did not constrain (#634).

The deploy workflow now passes --remove-orphans on the backbone roll, which closes this going forward. The flag is scoped to the compose project label (com.docker.compose.project=climate-lama), so a co-tenant stack on the same host is out of reach by construction — the shared Caddy runs as project caddy and is not a candidate. The UI-only path deliberately keeps up -d --no-deps ui with no sweep, so a UI deploy still cannot disturb anything else.

To audit the host by hand — every container carrying the project label, against the services the compose files actually declare:

cd /opt/climate-lama
for c in $(docker ps -a --format '{{.Names}}'); do
  printf '%-28s %s\n' "$c" \
    "$(docker inspect -f '{{index .Config.Labels "com.docker.compose.project"}}' "$c")"
done
./dc config --services | sort

Anything labelled climate-lama that is missing from the service list is an orphan. Remove it targeted (docker stop <name> && docker rm <name>) rather than with up -d --remove-orphans when you are doing this outside a deploy: the sweep form also reconciles the seven live services, and recreating postgres for the sake of deleting an unrelated container is a stateful restart you did not need. Drop the image afterwards too — the post-deploy prune only ever enumerates this project's own three repositories, so a retired third-party image is never a prune candidate.

Rollback is a re-dispatch with the previous tag. If the schema moved, roll it back by hand first (docker compose $P exec api alembic downgrade <revision>) — the workflow only ever upgrades. See Rollback Procedure.

Image retention on the host¶

Every deploy leaves the image it superseded on disk, and nothing used to remove it: the 38 GB host volume reached 93% on 2026-08-02 with tags accumulated since June. The final step of a successful deploy keeps the tag just deployed plus the one before it for climate-lama, climate-lama-worker and climate-lama-ui, and removes older tags of those three repositories.

Keeping n-1 is the point: it is the rollback reserve, which is exactly what a blanket docker image prune -a would destroy (the caveat raised in #539). A rollback re-dispatch to the previous tag therefore needs no re-pull.

The step is written to under-delete rather than over-delete:

Only this project's three repositories are ever listed. Infra images (postgres, redis, caddy, titiler, martin, …) are not enumerated at all, so they cannot be removed by construction.
Scoped to what was deployed. A UI-only deploy prunes only UI tags; a backbone-only deploy only backbone/worker tags.
Matched by image ID, not by tag string. Every alias of a kept image (e.g. :latest) survives, and an alias can never be mistaken for the previous release. Anything a container references — running or stopped — is skipped, and docker image rm is called without -f, so the daemon refuses in-use images as a second line of defence.
If the deployed tag is not on the host, nothing is pruned for that repository — the step warns rather than guessing which tags are stale.
It cannot fail the deploy. The step is continue-on-error and its script always exits 0: reclaiming disk must never turn a good deploy red. Free space before and after is printed in the job log.

Re-dispatching the same tag is a near-no-op up -d, so a retry after a transient failure is safe.

Required secrets¶

These live in GitHub Actions, never in the repo. They belong to the prod-host environment (Settings → Environments → New environment → prod-host → Add secret) — a dedicated environment, not the Cloudflare-Pages climate-lama (Production) one, so host access can carry its own protection rules (e.g. required reviewers on a deploy).

Secret	Value
`DEPLOY_SSH_HOST`	Hostname or IP of the prod box (e.g. `159.69.211.124`)
`DEPLOY_SSH_USER`	SSH user owning `/opt/climate-lama` and the host's GHCR login (today `root`)
`DEPLOY_SSH_KEY`	Private half of a dedicated deploy keypair, OpenSSH format, trailing newline included. Least privilege — generate a fresh key for this workflow, never reuse a personal one.
`DEPLOY_SSH_KNOWN_HOSTS`	Output of `ssh-keyscan -H <host>`; pins the host key so the connection cannot be silently MITM'd

Optional variables (same environment, or repo-level) override the defaults /opt/climate-lama, 22, https://api.climate-lama.online, https://climate-lama.online: DEPLOY_PATH, DEPLOY_SSH_PORT, PROD_API_BASE_URL, PROD_UI_BASE_URL.

No GHCR credential is added. Every pull — and the preflight existence check — runs on the host and reuses its existing docker login ghcr.io for all three images (climate-lama, climate-lama-worker, climate-lama-ui), so the PAT in Registry Authentication stays the single copy. When that PAT expires the preflight is what fails, before anything is touched.

One-time host setup:

ssh-keygen -t ed25519 -f gha-deploy -C "gha-deploy@climate-lama" -N ""
ssh-copy-id -i gha-deploy.pub root@159.69.211.124   # or append to authorized_keys
ssh-keyscan -H 159.69.211.124                       # -> DEPLOY_SSH_KNOWN_HOSTS
# gha-deploy (private) -> DEPLOY_SSH_KEY, then delete the local copy

The host's firewall must accept SSH from GitHub-hosted runners (their egress IPs are dynamic — see GitHub's meta API). A self-hosted runner or a fixed bastion is the alternative if the box is IP-restricted.

In-flight Celery messages: deploy workers before producers¶

Celery serialises an entire chain into its first message, so an ingest chord dispatched before the deploy keeps executing the old task signatures until it drains — the payload on the broker is frozen at dispatch time and is not re-serialised when a link runs. Two rules follow, and neither is optional when a release changes a task's parameters:

Deploy the worker before the producer (API). A new API dispatching a message with parameters an old worker does not accept fails that task with a TypeError — the chord dies mid-flight and nothing notices until a watchdog sweep (the 6h chord timeout for a job stuck in chunking; the shorter wedged- writing TTL if the skew hits write_chunk instead). This direction cannot be fixed in code — the receiver is the old build — so ordering is the mitigation. Draining the ingest queue first — no ingest_jobs row in pending/running/chunking/writing/aggregating/committed (committed is still in flight: aggregate_and_commit walks through it to succeeded inside one task) — removes the window entirely, and is worth doing for a release that changes an ingest task.
New parameters must be optional, and their defaults must not carry meaning. The reverse direction (an old, short message on a new worker) is a plain argument default, so it is fixable in code and is fixed: plan_chunks resolves its bbox from ingest_jobs.bbox (#453) and its legacy job_id from the jobs.params->>'ingest_job_id' back-link (#454), treating both arguments as pure overrides. A default that instead means something — bbox=None once meaning "plan against the raw file" — turns a rolling deploy into a silent behaviour change: in that case a spurious E_INGEST_PLAN_REJECTED on a job whose clipped extent was comfortably under the cell ceiling.

Hetzner shared-host deployment (behind shared Caddy)¶

The live production box (api.climate-lama.online) is a Hetzner host that already runs a shared Caddy owning :80/:443 and fronting the Climate-Lama api/ui vhosts (the box is single-tenant as of 2026-08-03 — see the decommission note in the host file layout above — but the Caddy-in-front-of-compose pattern is designed to front additional apps if the host ever gains one), with object storage on Hetzner Object Storage. That topology is captured in a version-controlled overlay, docker-compose.hetzner.yml, layered on top of the base prod compose:

docker compose -f docker-compose.prod.yml -f docker-compose.hetzner.yml \
  --env-file .env.prod up -d

What the overlay does (and deliberately nothing more):

api joins the external caddy_net network with aliases climate-lama-api (what Caddy reverse-proxies for the API host) and api (what the UI container's nginx targets via proxy_pass http://api:8000).
ui runs the published climate-lama-ui image (pinned with UI_TAG), also on caddy_net, served by Caddy at the apex/www.
the bundled nginx TLS terminator is parked behind a donotstart profile so it never contends for :80/:443 with the shared Caddy.

Object storage and healthchecks are not in the overlay — they are handled by the base docker-compose.prod.yml plus .env.prod: MINIO_ENDPOINT=nbg1.your-objectstorage.com selects Hetzner Object Storage, and the base file ships no embedded object store to park. The caddy_net network must already exist on the host (it is created by the shared Caddy stack; external: true).

DNS records (Namecheap)¶

All records are A records added by hand in Namecheap → Domain List → Manage climate-lama.online → Advanced DNS → Host Records. Namecheap is the authoritative DNS for this domain — there is no Hetzner or Cloudflare DNS zone for it.

Host	Type	Value	Serves
`api`	A	`159.69.211.124`	the backbone API
`@`	A	`159.69.211.124`	the UI (apex)
`www`	A	`159.69.211.124`	the UI (www)

A new record resolves within minutes (set TTL to 5 min while iterating). Caddy can only issue a certificate once the name resolves to the box — add the DNS record first, confirm it resolves (nslookup <host> 1.1.1.1), then add the Caddy block.

Caddy wiring (exposing a host)¶

The shared Caddy routes by container name on caddy_net. To expose a new host, add a block to /opt/caddy/Caddyfile and reload — Caddy auto-issues the Let's Encrypt cert (and auto-renews it):

api.climate-lama.online {
    reverse_proxy climate-lama-api:8000
}

climate-lama.online, www.climate-lama.online {
    reverse_proxy climate-lama-ui:80
}

cp /opt/caddy/Caddyfile /opt/caddy/Caddyfile.bak.$(date +%s)          # back up first
docker exec caddy caddy validate --adapter caddyfile --config /etc/caddy/Caddyfile
docker exec caddy caddy reload   --adapter caddyfile --config /etc/caddy/Caddyfile  # graceful; other vhosts stay up
docker logs --since 1m caddy | grep -i certificate                    # watch issuance

Provisioning an org + user¶

scripts/create_user.py provisions a user in an existing org (it does not create orgs):

docker compose $P exec api python scripts/create_user.py \
  --email you@example.com --role org_admin --org-slug <slug>
# prompts for a password — policy: >= 12 chars, >= 1 uppercase, >= 1 digit

Bootstrapping the first platform admin (#828)¶

GET /v1/admin/* (e.g. the external-dataset review queue, pending-ingests) is gated on the users.is_platform_admin flag, not an org role — a fresh deployment starts with zero platform admins, and that endpoint family is unreachable until one exists. Migration 0014 always seeds a default organization, so this works on a genuinely fresh deployment with no manual SQL:

docker compose $P exec api python scripts/create_user.py \
  --email you@example.com --role org_admin --platform-admin --org-slug default

--platform-admin is idempotent and repeatable: if --email already has an account in the org given by --org-slug (e.g. it was provisioned earlier without the flag), re-running the same command promotes it in place rather than no-op'ing — this is the supported alternative to hand-writing an UPDATE users SET is_platform_admin = true WHERE email = ... against the database. Re-running without --platform-admin remains a plain no-op, as before.

Email is only unique per org, not globally — if the account you mean to promote lives in a different org than --org-slug selects, the script refuses (rather than silently minting a second identity for the same address) and tells you which --org-slug to use instead. Get the org right: e.g. scripts/seed_demo.py's demo user lives in org acme-demo, not default.

To create the org first (and, on images that predate create_user.py, the user too) use the repositories directly:

docker compose $P exec -T api python - <<'PY'
import asyncio
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession
from climate_lama.config import get_settings
from climate_lama.db.repositories.organization_repository import OrganizationRepository

async def main():
    eng = create_async_engine(get_settings().database_url)
    async with async_sessionmaker(eng, class_=AsyncSession)() as s, s.begin():
        repo = OrganizationRepository(s)
        org = await repo.get_by_slug("northlane") or await repo.create(name="Northlane", slug="northlane")
        print("org", org.id)
    await eng.dispose()
asyncio.run(main())
PY

Login is multi-tenant — clients send org_slug alongside email/password.

Optional: UI as a compose service (dev / smoke-test)¶

For a one-command "full stack up" experience — for demos or smoke-tests, not production — the dev docker-compose.yml defines a ui service behind the fullstack profile. The service pulls the published image from GHCR and does not build from source.

# Start the backend only (default):
docker compose up

# Start backend + UI (pulls ghcr.io/cortomaltese3/climate-lama-ui):
docker compose --profile fullstack up

# Pin a specific UI version (default is :latest):
UI_TAG=v0.1.0 docker compose --profile fullstack up

The UI is reachable at http://localhost:3000 and proxies /v1/ to the API container. For production deployments the UI is deployed from its own repo; this profile is not used by docker-compose.prod.yml.

Docs Site (Cloudflare Pages)¶

The MkDocs Material docs site is built by GitHub Actions and deployed to Cloudflare Pages. The live site is at https://climate-lama.pages.dev.

How it works¶

On every PR that touches docs/** or mkdocs.yml: the site is built (mkdocs build --strict) and a preview URL is posted as a PR comment.
On every push to main that touches docs/** or mkdocs.yml: the site is built and deployed to production (climate-lama.pages.dev). The live site always reflects the current state of main.

The workflow lives at .github/workflows/docs.yml.

One-time setup (already done)¶

These steps were completed when the pipeline was first wired up. Documented here so they can be reproduced if credentials need to be rotated.

1. Create a Cloudflare account

Free at cloudflare.com. No credit card required.

2. Create a Pages project

Cloudflare Dashboard → Workers & Pages → Create → Pages → Connect to Git → authorise GitHub → select the climate-lama repo → set project name to climate-lama. Skip the build configuration (GitHub Actions handles the build).

3. Get credentials

Account ID: visible in the right sidebar of any Workers & Pages page, or in the dashboard URL (https://dash.cloudflare.com/<account-id>).
API Token: My Profile → API Tokens → Create Token → Create Custom Token. Add one permission row: Account / Cloudflare Pages / Edit. Leave all other settings at defaults.

4. Add secrets to the GitHub repo

Repository Settings → Secrets and variables → Actions → New repository secret:

Secret name	Value
`CLOUDFLARE_API_TOKEN`	The token created in step 3
`CLOUDFLARE_ACCOUNT_ID`	The account ID from step 3

Rotating credentials¶

If the API token is revoked or expires:

Create a new token in the Cloudflare dashboard (same permissions as above).
Update the CLOUDFLARE_API_TOKEN secret in GitHub repo settings.
The next push to main will use the new token automatically.

Triggering a deploy manually¶

Push any v* tag pointing to the commit you want deployed:

git tag v0.4.0 && git push origin v0.4.0

To test without a real release, use a pre-release tag and delete it afterwards:

git tag v0.4.0-rc1 && git push origin v0.4.0-rc1
# verify the deploy, then clean up
git tag -d v0.4.0-rc1 && git push origin :refs/tags/v0.4.0-rc1

Custom domain (optional)¶

To serve the site from a custom domain (e.g. docs.climate-lama.io):

Cloudflare Dashboard → climate-lama Pages project → Custom domains → Set up a custom domain → enter the domain.
If the domain is managed in Cloudflare DNS, the CNAME is added automatically. Otherwise, add a CNAME record pointing to climate-lama.pages.dev at your DNS provider.
Update site_url in mkdocs.yml to the new domain.

Backup Strategy¶

scripts/backup.sh and scripts/restore.sh (issue #388) replace the old manual pg_dump snippet with a scripted, scheduled backup and a documented, exercised restore. Both are plain bash, no extra host dependencies beyond docker + docker compose — the Postgres dump runs via docker compose exec, and the object-store step runs mc in a throwaway minio/mc container, so it works the same way against either object-store backend the app supports (both speak the S3 API behind the same MINIO_* env vars):

Hetzner Object Storage (prod default) — MINIO_ENDPOINT is a public FQDN.
Embedded/local MinIO (dev docker-compose.yml) — MINIO_ENDPOINT is a bare compose service name (e.g. minio:9000); the script joins the stack's docker network to resolve it.

What is (and isn't) covered¶

Each run writes a timestamped directory under --backup-root (default ./backups):

backups/20260727_030000/
  db.dump         # pg_dump -Fc (Postgres custom format, restorable with pg_restore)
  objects/        # mc mirror of the configured MINIO_BUCKET
  MANIFEST.txt    # what was backed up, from where, at what git sha

Covered: the Postgres database, and the configured object-store bucket. Not covered — nothing is backed up silently:

.env.prod (DB password, API keys, object-store credentials) — never copied into a backup directory. Reprovision it from your secrets manager on restore.
nginx/Caddy TLS config, host firewall/DNS setup.
Redis (queue/cache state) — disposable; in-flight jobs are re-run, not restored.

MANIFEST.txt restates this coverage list on every run so it travels with the backup itself, not just this doc.

Running a backup¶

scripts/backup.sh                                    # defaults: docker-compose.prod.yml, .env.prod, ./backups, 14-day retention, 1024 MB free-space floor
scripts/backup.sh --retention-days 30                 # keep a month
scripts/backup.sh --compose-file docker-compose.yml --env-file .env   # local/dev stack
scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024   # DB-only, the scheduled-run flags (see below)
scripts/backup.sh --skip-objects --retention-days 7 --push-remote        # ...and ship the dump off-host

The dump is verified before the run is called a success¶

pg_dump exiting 0 is not proof of a restorable archive — a dump truncated by a full disk, a killed container or a broken pipe still leaves a plausible-looking file behind. Every run therefore reads its own dump back with pg_restore --list (run inside the postgres container, so no host-side Postgres client is needed) and fails unless the archive's table of contents enumerates at least --min-table-data TABLE DATA entries — default 10; production measures ~50 against 42 declared models. A dump that fails either check is renamed to db.dump.FAILED-VERIFICATION, so scripts/restore.sh cannot mistake it for a usable backup, and the script exits non-zero the night it happens rather than the day someone needs it.

Note the bound: this reads the archive's table of contents. It catches an unreadable, non-pg_dump or implausibly small archive; it does not prove every row inside a well-formed archive restores. A restore into a scratch database is the stronger check and is still a manual drill (see below).

Off-host copy (`--push-remote`)¶

By default a backup lands only on the host it protects, which covers accidental data loss but not loss of the host. --push-remote also uploads db.dump and MANIFEST.txt to <--remote-prefix>/<timestamp>/ in the same S3-compatible bucket the app already uses — same MINIO_* credentials and the same throwaway-mc-container plumbing as the mirror step, so it needs no new machinery or secrets. --remote-prefix defaults to backups.

It is opt-in: without the flag the script behaves exactly as before.
MINIO_* becomes required as soon as --push-remote is passed, even alongside --skip-objects.
The objects/ mirror is never uploaded — it is a copy of that same bucket.
A failed upload fails the run (exit 1) but never discards the verified local dump.
Remote copies are not pruned by --retention-days. Expire them with a bucket lifecycle rule.

This matters more than the object-store side suggests: exposure_datasets has no path column, so the 717k exposure rows are PostGIS geometry in Postgres and exist in no bucket. Losing the bucket costs rasters that can be re-ingested from upstream; losing the host costs data that exists nowhere else.

Retention¶

Backups older than --retention-days (default 14) are pruned at the end of a successful run. A directory under --backup-root is prunable when it is recognisably a backup: its name matches the script's own YYYYMMDD_HHMMSS pattern, or it contains a db.dump or a MANIFEST.txt. The second clause is what hand-taken pre-deploy dumps (predeploy-v0.5.0, pre-v0.6.0, pre-org-migration-…) look like; under the old name-only rule they were never pruned and had accumulated 1.2 GB past the window in production (#771). Anything else parked under --backup-root is still never touched, and the directory the current run just wrote is excluded unconditionally. Pass --prune-timestamped-only to restore the legacy name-only rule.

--skip-objects dumps the database only and skips mirroring the object-store bucket (mirrors the flag scripts/restore.sh already has). Before doing any docker/postgres work, the script also refuses to start with less than --min-free-mb (default 1024) free at --backup-root, so an unattended run on a disk shared with another app fails loudly with the shortfall named, instead of silently filling the disk.

Scheduling (prod host)¶

docker compose has no built-in scheduler, so this runs at the host level. The systemd timer below is the mechanism actually installed on the prod host — DB-only (--skip-objects), 7-day retention. A cron entry works identically and is documented as a fallback, but it is not what is running; if you switch to it, disable the timer first so a backup doesn't run twice.

Sizing, measured on the host on 2026-08-02 right after the timer's first run: 13.4 GB free of a 38 GB volume, and a DB-only dump is ~257 MB, so 7 days of retention is ~1.8 GB — comfortable headroom above the 1024 MB free-space floor backup.sh checks before every run. Re-check this arithmetic if the database grows sharply or the box gains a co-tenant again: the floor makes an over-full disk fail loudly instead of silently, but it does not create space. (That 2026-08-02 figure is already stale in the reassuring direction — the host was re-measured at ~18 GB free of the same volume, 52% used, on 2026-08-03 after lama-chat's removal, so headroom is more comfortable than this section states, not less.)

systemd timer (installed) — /etc/systemd/system/climate-lama-backup.service:

[Unit]
Description=Climate-Lama backup (DB only, object store skipped)

[Service]
Type=oneshot
WorkingDirectory=/opt/climate-lama
ExecStart=/opt/climate-lama/scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024

/etc/systemd/system/climate-lama-backup.timer:

[Unit]
Description=Run Climate-Lama backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl enable --now climate-lama-backup.timer

A failed run is visible via the unit's own exit status — systemctl status climate-lama-backup.service and journalctl -u climate-lama-backup.service show the script's FAIL line (stderr is captured by journald automatically for a systemd-managed service), so a human checking either surfaces it even without dedicated alerting.

cron (documented fallback, not installed) — as the deploy user, crontab -e:

# Climate-Lama backup, daily at 03:00
0 3 * * * cd /opt/climate-lama && ./scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024 >> /var/log/climate-lama-backup.log 2>&1

Object-store mirroring is skipped on the scheduled run (the bucket lives on Hetzner Object Storage with its own durability); run scripts/backup.sh without --skip-objects by hand for an occasional full mirror.

Host-side step, not done by merging code (#771): the installed unit above does not yet pass --push-remote, so the scheduled dump still lands only on the host it protects. Appending --push-remote to the unit's ExecStart (then systemctl daemon-reload) is what closes the host-loss gap; until that is done on the box, the capability exists in the script and is unused in production.

Still out of scope (tracked as follow-ups, not built here): alerting on backup failure, bucket versioning (#857), and an automated restore-into-a-scratch-database drill.

Status (#533): the flags and guard above are implemented and tested in this repo. Installing the timer on the production host — and confirming it produces a dated backup directory without being run by hand — is a separate, host-side step tracked on that issue; it is not done by merging this doc change.

Restore procedure¶

scripts/restore.sh <backup-dir> [--compose-file <path>] [--env-file <path>] [--yes]

<backup-dir> is one of the timestamped directories backup.sh produced (e.g. ./backups/20260727_030000). The script is destructive by design — it prompts for confirmation (--yes to skip, required when not run at a tty) before:

pg_restore --clean --if-exists — drops and recreates database objects from db.dump.
mc mirror — overwrites matching keys in the target bucket from objects/.

Use --skip-db / --skip-objects to restore only one half.

After any DB restore, re-run scripts/create_app_role.py --apply if the RLS role flip is live. pg_restore --clean drops and recreates every object as POSTGRES_USER, and the recreated objects carry no grants to climate_lama_app — ALTER DEFAULT PRIVILEGES applies to objects created after it was set, not retroactively to a restore. The api and worker would come back up throwing permission denied for table … on everything. The role itself survives the restore (roles are cluster-level, not in the dump); only its grants are lost.

backup.sh and restore.sh themselves are unaffected by the flip: they exec into the postgres container and authenticate as POSTGRES_USER over the container-local socket, never through DATABASE_URL.

Exercised restore drill¶

Run once against a scratch stack (local docker-compose.yml, isolated project/network, torn down afterward — no prod data involved) to prove the procedure actually works, not just that it parses. Transcript, lightly trimmed:

$ docker compose -f docker-compose.yml up -d postgres minio minio-init
 Container climate_lama_postgres  Started
 Container climate_lama_minio     Healthy
 Container climate_lama_minio_init Started

# seed data to lose
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama \
    -c "CREATE TABLE backup_drill (id serial PRIMARY KEY, note text); \
        INSERT INTO backup_drill (note) VALUES ('issue-388-restore-drill');"
CREATE TABLE
INSERT 0 1
$ mc cp drill-object.txt local/climate-lama/drill-object.txt
`drill-object.txt` -> `local/climate-lama/drill-object.txt`

# back up
$ scripts/backup.sh --compose-file docker-compose.yml --env-file .env --backup-root ./backups
-> dumping database "climate_lama" (pg_dump -Fc)...
✓ database dump: ./backups/20260727_055915/db.dump (8.0K)
-> mirroring object store bucket "climate-lama" (minio:9000)...
✓ object store mirror: ./backups/20260727_055915/objects (1 object(s))
✓ manifest: ./backups/20260727_055915/MANIFEST.txt
✓ backup complete: ./backups/20260727_055915

# simulate data loss
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "DROP TABLE backup_drill;"
DROP TABLE
$ mc rm local/climate-lama/drill-object.txt
Removed `local/climate-lama/drill-object.txt`.
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "SELECT * FROM backup_drill;"
ERROR:  relation "backup_drill" does not exist

# restore
$ scripts/restore.sh ./backups/20260727_055915 --compose-file docker-compose.yml --env-file .env --yes
This will overwrite data using ./backups/20260727_055915:
  - DROP and recreate objects in database "climate_lama" (pg_restore --clean)
  - overwrite matching keys in bucket "climate-lama" (minio:9000)
-> restoring database "climate_lama" from ./backups/20260727_055915/db.dump...
✓ database restored from ./backups/20260727_055915/db.dump
-> restoring object store bucket "climate-lama" from ./backups/20260727_055915/objects...
✓ object store restored from ./backups/20260727_055915/objects
✓ restore complete from ./backups/20260727_055915

# verify
$ docker exec climate_lama_postgres psql -U climate_lama -d climate_lama -c "SELECT * FROM backup_drill;"
 id |          note
----+-------------------------
  1 | issue-388-restore-drill
(1 row)

$ mc cat local/climate-lama/drill-object.txt
issue-388 backup/restore drill object

Both the dropped table (and its row) and the deleted object came back exactly as backed up. Scratch stack torn down with docker compose -f docker-compose.yml down -v afterward — nothing above touched prod data or credentials.

Hetzner Object Storage (production object storage)¶

Production deployments use Hetzner Object Storage as the S3-compatible backend for the same MINIO_* env vars the backbone uses locally. The canonical bucket layout is defined by ADR-029 (written for DO Spaces, but S3-generic — it applies to any backend unchanged).

Current bucket: climate-lama-storage in location nbg1 (endpoint https://nbg1.your-objectstorage.com).

Migration note (2026-07): the previous backend, DigitalOcean Spaces (climate-lama-space @ fra1), is decommissioned. A mirror of its last contents sits in the new bucket under the legacy climate-lama-space/ prefix (a handful of June hazard HDF5s with no DB rows referencing them) — safe to delete once confirmed orphaned.

Credentials¶

Hetzner S3 credentials are scoped to the Hetzner Cloud project, not the bucket — one access/secret key pair reaches every bucket in the project. Generate or revoke them in the Hetzner Cloud console under the project's Object Storage section, and put them in .env.prod only (mode 600 on the host; never committed).

One-time setup (done 2026-07-28)¶

Documented here so the bucket can be reprovisioned or a second location added. The bucket itself is created in the Hetzner Cloud console. Everything else is plain S3 API — the commands below use mc (MinIO client) in a throwaway container so the host needs nothing beyond docker; run them on the prod host so the credentials are read from .env.prod instead of appearing in a shell history:

alias hzmc='docker run --rm --env-file /opt/climate-lama/.env.prod \
  --entrypoint sh minio/mc:latest -c'

1. Create the canonical prefix markers

S3 has no real folders — empty .keep objects make the ADR-029 prefix hierarchy (raw/, processed/, processed/public/, tiles/, reports/) visible in the console and give lifecycle rules a stable target:

hzmc 'mc alias set hz "https://$MINIO_ENDPOINT" "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" &&
  for p in raw processed processed/public tiles reports; do
    printf "" | mc pipe hz/$MINIO_BUCKET/$p/.keep
  done && mc ls hz/$MINIO_BUCKET'

2. Lifecycle rule for tiles/ — pending

Tiles are regenerable (ADR-029 invariant 4) — expire them after 30 days to keep storage bounded. Hetzner implements the S3 lifecycle API (verified: GetBucketLifecycleConfiguration answers NoSuchLifecycleConfiguration rather than rejecting the call), but the rule has not been applied yet:

hzmc 'mc alias set hz "https://$MINIO_ENDPOINT" "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" &&
  mc ilm rule add --expire-days 30 --prefix tiles/ hz/$MINIO_BUCKET &&
  mc ilm rule ls hz/$MINIO_BUCKET'

3. CORS — none configured, none currently needed

The UI never fetches the bucket directly: tiles and reports are served through the backbone's read-through proxy (/v1/tiles/...), so browser CORS rules on the bucket are unnecessary today. Hetzner answers the bucket-CORS S3 API (verified via mc cors get), so if a browser-direct path is ever introduced (e.g. presigned-URL fetches from the SPA), add rules with mc cors set for https://climate-lama.online + www at that point.

4. Configure environment variables

Set in .env.prod (and .env locally when testing against the live bucket):

MINIO_ENDPOINT=nbg1.your-objectstorage.com
MINIO_ACCESS_KEY=<hetzner-s3-access-key>
MINIO_SECRET_KEY=<hetzner-s3-secret-key>
MINIO_BUCKET=climate-lama-storage
MINIO_SECURE=true
TITILER_S3_PREFIX=s3://climate-lama-storage

The backbone reads these from src/climate_lama/config.py. Hetzner Object Storage is S3-compatible, so no code change is needed to switch from MinIO. The setting that toggles HTTPS is MINIO_SECURE (the minio Python client's secure= flag) — set it true here. There is no MINIO_USE_SSL variable; because Settings ignores unknown env keys, a misnamed one is silently dropped and the client falls back to plain HTTP.

docker-compose.prod.yml no longer hardcodes MINIO_ENDPOINT; it reads the whole MINIO_* block from .env.prod, so pointing at the external bucket is purely a config change. The prod compose ships no embedded object store — the external bucket is required; the embedded MinIO exists only in the dev docker-compose.yml.

Geocoding in production¶

Production deliberately runs without the opt-in geocoding compose profile. nominatim (docker-compose.prod.yml) is never started on the live host, and that is correct — not an oversight to fix. GET /v1/geocode returning 503 E_GEOCODE_UNAVAILABLE is the intended production behaviour: it is the caller's explicit signal to fall back to manual pin-drop (ADR-044). See issue #534 for the decision record.

GEOCODER_ENABLED stays "true" on the api service regardless — flipping it to false would change nothing observable (no endpoint or /v1/info field advertises geocoder state to a client) and would erase a diagnostic signal: with it true, the 503's internal reason distinguishes a genuine Nominatim outage ("connection") from a deliberate disable ("disabled").

Why the 503 is safe to ship¶

src/climate_lama/core/geocoding.py:137-138 — when Settings.geocoder_enabled is False (the library default — config.py:387), geocode_address/reverse_geocode raise GeocodeUnavailableError("disabled") before any network call.
src/climate_lama/core/geocoding.py:229-230 — when the provider can't be reached (a compose service that isn't running fails DNS resolution immediately), the adapter catches httpx.HTTPError — the base class of ConnectError, ConnectTimeout, and ReadTimeout — and raises GeocodeUnavailableError("connection", ...). Every call is additionally bounded by geocoder_timeout_seconds (config.py:389, default 5.0), so a hung provider can't hang the request either.
Both paths map to the same 503 E_GEOCODE_UNAVAILABLE at src/climate_lama/main.py:323. A client that handles the disabled case already handles the not-yet-provisioned case — there is no separate failure mode to build for.

What standing Nominatim up would actually cost¶

mediagis/nominatim imports an OSM extract into its own PostgreSQL instance on first boot — not a config toggle. Even a Greece-only extract is a multi-GB, long-running, memory-hungry import. The production host had 13.4 GB free of a 38 GB volume when measured on 2026-08-02, and that one box runs the entire climate-lama stack plus Caddy, so an unattended multi-GB import competes with the live service for both disk and memory. Standing the profile up for real needs a disk assessment (likely a dedicated volume) first, or routing through a hosted/third-party geocoder instead — the adapter is provider-agnostic by design (ADR-044), so either is a legitimate answer without reversing the fail-closed default.

Update (2026-08-03): the co-tenant/disk-contention framing above is now void — lama-chat has been decommissioned (see the note in the host inventory near the top of this doc), the host is single-tenant, and free disk has grown to ~18 GB of the same volume (52% used), up from the 13.4 GB measured on 2026-08-02. This retires the rationale above, not the decision: geocoding stays disabled-by-fallback in production until the owner explicitly revisits it — a memory-hungry OSM import competing with the live service is still a real cost worth a deliberate decision, just no longer one made scarcer by a co-tenant.

Remaining host-side step¶

.env.prod has no NOMINATIM_PASSWORD, so every docker compose invocation on the host — even though the nominatim service is never started — prints:

level=warning msg="The \"NOMINATIM_PASSWORD\" variable is not set. Defaulting to a blank string."

Compose interpolates variables for every service defined in the file at parse time, regardless of which profiles are active, so the warning fires unconditionally. Setting NOMINATIM_PASSWORD in .env.prod silences it. This is tracked as the one remaining host-side step on issue #534, to be done in the same SSH session as the backup timer install; that issue stays open until it lands.

Rollback Procedure¶

Identify the last good release tag (e.g. v0.3.0).
Point .env.prod at it: set CLIMATE_LAMA_TAG=v0.3.0.
Pull and restart — no rebuild, the image already exists in GHCR ($P is the both-files shorthand from Release → Production):
```
docker compose $P pull api worker
docker compose $P up -d
```
A one-step rollback needs no network: the deploy workflow deliberately keeps the previous release's images on the host (see Image retention), so the pull is a no-op. Going back two or more releases does re-pull from GHCR.

Roll back migrations if the schema changed:

docker compose $P exec api alembic downgrade -1

Repeat -1 for each migration to undo, or target a specific revision:

docker compose $P exec api alembic downgrade <revision>

Verify the rollback landed — same identity check, with the previous tag:
```
./scripts/verify-deploy.sh v0.3.0
```
A rollback that silently kept serving the bad image is the failure mode this catches; /health/live cannot.

Monitoring (optional)¶

A Grafana + Prometheus overlay is available for local observability. It layers on top of docker-compose.yml without modifying it.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

Prometheus UI: http://localhost:9090 — scrapes api:8000/metrics every 15s and loads the JobFailureRateHigh alert rule.
Grafana: http://localhost:3000 — login admin / admin. The preconfigured Climate-Lama dashboard is auto-provisioned with request rate, 5xx error rate, job completions, and p95 job duration panels.

Logs¶

On the live host use the overlay shorthand $P (from Release → Production) so the ui service is included:

# All services
docker compose $P logs -f

# Single service
docker compose $P logs -f api