Secret rotation runbook — the `.env.prod` + prod-host plane¶

Status: Active Owner: backbone maintainers (per ADR-034 §Rotation story) Tracking issue: #640

Scope — read this first¶

This runbook covers the secrets plane that exists today: values held in .env.prod on the single-tenant Hetzner host (mode 600 — docs/DEPLOYMENT.md:17, :43), the host's own docker login ghcr.io credential, and the GitHub Actions secrets that reach that host (prod-host environment) or drive the release gate (SDK_SMOKE_* repo secrets).

It is superseded the moment a hosted secrets backend is adopted. ADR-034 deliberately deferred that pick, and it is tracked in #285 with the working document at secrets-backend-followup.md. When a backend (Doppler / 1Password Connect / AWS Secrets Manager / Hetzner vault) lands, every "edit .env.prod and restart" step below is replaced by that backend's rotation path — see step 5 of that follow-up doc. Until then, this file is the procedure.

This runbook does not cover: Cloudflare Pages docs credentials (rotation is already documented at docs/DEPLOYMENT.md), TLS certificates (Caddy auto-renews), or the seeded-login password (held out-of-band; reset procedure in Demo data seeder).

Inventory¶

#	Secret	Lives in	Consumed by	Rotate when
1	GHCR PAT `ghcr-pull-hetzner`	host `~/.docker/config.json` (of `DEPLOY_SSH_USER`)	every `docker pull` / `docker manifest inspect` on the host, including the deploy workflow's GHCR preflight	before its expiry, or on suspected exposure
2	`DEPLOY_SSH_KEY` + `DEPLOY_SSH_KNOWN_HOSTS`	GitHub `prod-host` environment secrets	`deploy-prod.yml` `Configure SSH` step	≥ every 90 days; on offboarding; on host rebuild (known-hosts also on host-key change)
3a	`POSTGRES_PASSWORD` (owner role `climate_lama`)	`.env.prod`	`postgres` container init, `scripts/backup.sh` / `scripts/restore.sh`, the default owner DSN, `MAINTENANCE_DATABASE_SYNC_URL`	≥ every 90 days
3b	`climate_lama_app` role password	`.env.prod` (`APP_DATABASE_URL` / `APP_DATABASE_SYNC_URL`)	`api`, `worker`, `beat` runtime DSNs	≥ every 90 days
3c	`MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY`	`.env.prod`	`api`, `worker`, `beat` (via `env_file`) and `titiler` (as `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`)	≥ every 90 days
3d	`APP_SECRET_KEY` (JWT signing key)	`.env.prod`	`api` — HS256 sign/verify of access tokens	≥ every 90 days; immediately on suspected exposure
3e	`API_KEY_PEPPER`	`.env.prod`	`api` — peppered hashes for API keys, refresh tokens, invites	≥ every 90 days; see the blast-radius warning in §3e before touching it
4	`SDK_SMOKE_TOKEN` / `SDK_SMOKE_EMAIL` / `SDK_SMOKE_PASSWORD` / `SDK_SMOKE_ORG_SLUG`	GitHub repo secrets (no environment)	`sdk-smoke.yml` `Detect smoke credentials` + `End-to-end smoke`	≥ every 90 days; on suspected exposure

Cadence and ownership come from ADR-034 §Rotation story: long-lived shared credentials rotate at least every 90 days and immediately after suspected exposure or contributor offboarding; the owning team per CODEOWNERS rotates.

Known expiry — the one date this runbook cannot supply. ghcr-pull-hetzner is a GitHub PAT (classic) created with an operator-chosen expiry (docs/DEPLOYMENT.md:268-273). The date exists only in GitHub's PAT settings page and the owner's password manager — nothing in this repository records it. [ASSUMPTION: owner to fill] — record the expiry date here, and set a calendar reminder for two weeks before it. Do not guess: an expired PAT fails the deploy workflow's GHCR preflight, which is the first thing a deploy does, on a host you rarely touch.

Rules that apply to every `.env.prod` rotation¶

These are not optional; each one has already produced an incident or is enforced by the tooling.

Back up the file before editing. The deploy workflow does this on every run (.env.prod.bak.<UTC timestamp>, last 10 kept — docs/DEPLOYMENT.md:937-938); a hand edit must do the same:
```
cd /opt/climate-lama
cp .env.prod ".env.prod.bak.$(date -u +%Y%m%dT%H%M%SZ)"
```
Keep mode 600 after any edit: chmod 600 .env.prod.
Always pass --env-file. env_file: feeds container environments; it does not feed Compose's own ${...} interpolation, and only --env-file does. Use the installed helper instead of remembering:
```
cd /opt/climate-lama && ./dc ps          # scripts/dc — supplies all three flags
```
See docs/DEPLOYMENT.md:89-107.
Render before restarting. config resolves the file exactly as up will, at zero risk:
```
./dc config | grep -E '<the var you changed>'
```
environment: beats env_file:. DATABASE_URL / DATABASE_SYNC_URL lines in .env.prod are inert — docker-compose.prod.yml:50-51 (and :114-115, :162-163) declare them in environment:. The live knobs are APP_DATABASE_URL / APP_DATABASE_SYNC_URL. Editing the inert lines changes nothing, silently.
Placeholder values will not boot in prod. Settings refuses to start with a _PLACEHOLDER_SECRETS value for app_secret_key / api_key_pepper (src/climate_lama/config.py:529-532). Generate real values with python -c "import secrets; print(secrets.token_hex(32))".
Verify the new value works before revoking the old one. Every section below is ordered rotate → verify → revoke for that reason. A revocation done first turns a rotation into an outage.

The single command that proves the running stack can still reach PostgreSQL, Redis and the object-storage bucket (the MinIO probe requires the configured bucket to exist, not merely that the endpoint answers — src/climate_lama/main.py:276-284):

curl -fsS https://api.climate-lama.online/health/ready | python -m json.tool
# every check must read {"status": "ok"}; any failure returns 503 with details

1. GHCR PAT — `ghcr-pull-hetzner`¶

Where it lives. On the host only, in ~/.docker/config.json of the user that runs docker compose (today root — deploy-prod.yml:22-23). It is stored base64, not encrypted (docs/DEPLOYMENT.md:279), so the file is mode 600.

Who consumes it. Every docker pull and docker manifest inspect for ghcr.io/cortomaltese3/climate-lama{,-worker,-ui} — including the deploy workflow's GHCR preflight, which runs on the host and deliberately adds no GHCR credential of its own, so this PAT is the single copy (docs/DEPLOYMENT.md:1063-1067).

Blast radius of rotation. None while the old token is still valid — Docker overwrites the cached credential on a successful docker login. No container restarts. The stack keeps running on already-pulled images regardless; only future pulls depend on this.

Rotate¶

Mint a replacement PAT (classic) with the single read:packages scope: GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) → Generate new token (classic). Name it distinguishably from the outgoing one (e.g. ghcr-pull-hetzner-<YYYY-MM>), set an expiry, check only read:packages, copy the ghp_… value. Same procedure as first issuance — docs/DEPLOYMENT.md:268-273.
Store it in the password manager before using it. Record the new expiry date at the top of this file.

Log in on the host, as the same user that runs docker compose:

ssh root@159.69.211.124
echo "ghp_xxxxxxxxxxxx" | docker login ghcr.io -u CortoMaltese3 --password-stdin
chmod 600 ~/.docker/config.json

Verify — proves the NEW token works¶

docker login reporting success only proves the token authenticated to the registry, not that it can read this project's private packages. Prove the authorisation too, for all three repositories. docker manifest inspect downloads no layers, so it is safe and cheap to repeat (docs/DEPLOYMENT.md:120-129):

ssh root@159.69.211.124 'for r in climate-lama climate-lama-worker climate-lama-ui; do
  docker manifest inspect "ghcr.io/cortomaltese3/$r:latest" >/dev/null \
    && echo "OK   $r" || echo "FAIL $r"
done'

→ verify: three OK lines. A unauthorized / denied means the new token lacks read:packages or was minted on the wrong account.

Then re-run the deploy preflight end to end, which is the real consumer:

HOST=root@159.69.211.124
ssh -o BatchMode=yes "$HOST" true
ssh "$HOST" 'docker manifest inspect ghcr.io/cortomaltese3/climate-lama:latest >/dev/null'

→ verify: both exit 0.

Revoke the old token¶

Only after the verification above passes:

GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) → the outgoing token → Delete. Deleting is immediate and irreversible; there is no grace period, which is why it comes last.

→ verify the revocation actually took effect: re-run the three-repository docker manifest inspect loop above — it must still print three OK lines (the host is now using the new token). If it fails after the delete, the host was still holding the old credential: re-run the docker login in step 3.

2. `prod-host` deploy keypair — `DEPLOY_SSH_KEY` / `DEPLOY_SSH_KNOWN_HOSTS`¶

Where it lives. GitHub → Settings → Environments → prod-host → Secrets. A dedicated environment, deliberately not the Cloudflare-Pages climate-lama (Production) one, so host access can carry its own protection rules (docs/DEPLOYMENT.md:1043-1049). The public half lives in the host's ~/.ssh/authorized_keys.

Who consumes it. deploy-prod.yml only. Its Configure SSH step (:145-176) writes the private key to ~/.ssh/deploy_key (mode 600) and the known-hosts to ~/.ssh/known_hosts, then builds a Host deploy-target block with IdentitiesOnly yes and StrictHostKeyChecking yes. A separate guard (:129-139) fails the run with a named list if any of the four secrets is empty.

Blast radius of rotation. Deploys only. Nothing in the running stack authenticates with this key. concurrency: deploy-prod (queue, never cancel) means a rotation cannot interleave with an in-flight deploy — but do not rotate while a deploy is queued.

Rotate¶

Generate a fresh, dedicated keypair locally (never reuse a personal key — docs/DEPLOYMENT.md:1055):
```
ssh-keygen -t ed25519 -f gha-deploy-$(date -u +%Y%m) -C "gha-deploy@climate-lama" -N ""
```

Append the new public half to the host — append, do not replace, so the outgoing key keeps working until the new one is proven:

ssh-copy-id -i gha-deploy-<YYYYMM>.pub root@159.69.211.124
# or, by hand:
# cat gha-deploy-<YYYYMM>.pub | ssh root@159.69.211.124 'cat >> ~/.ssh/authorized_keys'

Refresh the host-key pin (do this whenever the host is rebuilt; harmless otherwise):
```
ssh-keyscan -H 159.69.211.124
```

Update both environment secrets — the private key must be OpenSSH format including the trailing newline:

gh secret set DEPLOY_SSH_KEY --env prod-host < gha-deploy-<YYYYMM>
ssh-keyscan -H 159.69.211.124 | gh secret set DEPLOY_SSH_KNOWN_HOSTS --env prod-host

Delete the local private copy: shred -u gha-deploy-<YYYYMM> (or rm -P / secure-delete equivalent).

Verify — proves the NEW key works¶

The workflow's own Configure SSH step ends with ssh deploy-target true before anything mutates the host (deploy-prod.yml:174-176), so a dispatch is a real end-to-end test of the key and the host-key pin. Use a UI-only dispatch pinned to the tag already deployed — re-dispatching the same tag is a near-no-op up -d (docs/DEPLOYMENT.md:1040-1041) and a UI-only path never touches api/worker and never runs migrations (docs/DEPLOYMENT.md:920-927):

# read the tag the host is CURRENTLY serving — do not invent one
ssh root@159.69.211.124 'grep "^UI_TAG=" /opt/climate-lama/.env.prod'
gh workflow run deploy-prod.yml -f ui_tag=<that exact value>
gh run watch "$(gh run list --workflow deploy-prod.yml -L1 --json databaseId --jq '.[0].databaseId')"

→ verify: the run is green, and its Configure SSH step logged SSH to the deploy target OK. A failure there — and not later — is the key or the known-hosts pin.

This dispatch spends GitHub Actions minutes. It is the only mechanism that exercises the secret exactly as the consumer does; a local ssh -i test proves the keypair but not that the secret was stored correctly (trailing newline, OpenSSH format).

Revoke the old key¶

Only after the dispatch above is green:

ssh root@159.69.211.124
cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak.$(date -u +%Y%m%dT%H%M%SZ)
# remove the line whose comment/fingerprint matches the SUPERSEDED key
ssh-keygen -lf ~/.ssh/authorized_keys      # list fingerprints to identify it
vi ~/.ssh/authorized_keys                  # delete that one line

→ verify the revocation: re-run the same UI-only dispatch. It must still be green (new key still works), and an ssh -i gha-deploy-<old> attempt with the old key must now be refused. Keep the authorized_keys.bak.* copy until the next successful deploy.

Do not remove your own interactive/admin key while doing this. If the file is mangled and the session is closed, the box is only reachable through the Hetzner Cloud console. Keep the SSH session open until the verification dispatch is green.

3. Application secrets in `.env.prod`¶

All of §3 shares the rules above: back up, chmod 600, render with ./dc config, restart, verify, then revoke.

3a. `POSTGRES_PASSWORD` — the owner role `climate_lama`¶

Who consumes it. Three distinct consumers, which is what makes this the most dangerous entry in the file (.env.example.prod, "Database" block):

the postgres container itself (docker-compose.prod.yml:6-8);
scripts/backup.sh (:158, :187) and scripts/restore.sh (:129, :167);
the default owner DSN that api/worker/beat fall back to when APP_DATABASE_URL / APP_DATABASE_SYNC_URL are unset (docker-compose.prod.yml:50-51), and MAINTENANCE_DATABASE_SYNC_URL if it spells the password out (it must — env_file values are literal strings and ${...} inside them is not expanded; docs/DEPLOYMENT.md:616-621).

POSTGRES_PASSWORD in .env.prod does not change an existing cluster's password. It initializes the container (.env.example.prod: "These three initialize the postgres container itself"), so on an already-provisioned postgres_data volume the file and the actual role password can silently diverge. The rotation is therefore an ALTER ROLE plus a file edit, in that order — never the file alone.

Rotation steps (production is single-tenant with brief-downtime deploys already, so a short app restart is acceptable):

Generate a strong value and store it in the password manager.

Change it in the database, as the owner:

cd /opt/climate-lama
./dc exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  -c "ALTER ROLE climate_lama WITH PASSWORD '<new>';"

Update POSTGRES_PASSWORD in .env.prod, and the password component of MAINTENANCE_DATABASE_SYNC_URL if that line is present.

Render, then restart the consumers:

./dc config | grep -E 'MAINTENANCE_DATABASE_SYNC_URL|POSTGRES_'
./dc up -d api worker beat

postgres itself needs no restart — ALTER ROLE is live.

Verify — proves the NEW value works:

# a) the new password actually authenticates
./dc exec -T postgres env PGPASSWORD='<new>' \
  psql -U climate_lama -d climate_lama -c 'SELECT 1;'
# b) the app can still reach the DB (real round-trip, not a liveness ping)
curl -fsS https://api.climate-lama.online/health/ready | python -m json.tool
# c) the maintenance lane still resolves to the owner — this is the check whose
#    failure would otherwise surface at the NEXT deploy, not now
./dc exec -T api alembic current          # must print the revision and (head)
# d) the backup path still authenticates — it reads POSTGRES_* independently
#    (same flags the scheduled run uses — docs/DEPLOYMENT.md:1436)
cd /opt/climate-lama && ./scripts/backup.sh --skip-objects --retention-days 7 --min-free-mb 1024

→ verify: (a) prints 1, (b) every check ok, (c) prints (head), (d) exits 0 and writes a db.dump under ./backups/<timestamp>/. See Backup Strategy.

Revoke the old value. There is nothing to revoke server-side — ALTER ROLE replaces the password atomically, so the old one is dead the moment step 2 lands. The revocation work is hygiene: delete the old value from the password manager, and remove the pre-edit .env.prod.bak.* files that still contain it once (c) and (d) pass:

ls -la /opt/climate-lama/.env.prod.bak.*    # these contain the OLD secret
shred -u /opt/climate-lama/.env.prod.bak.<timestamps older than this rotation>

3b. The `climate_lama_app` role password (#556)¶

Where it lives. .env.prod, inside APP_DATABASE_URL and APP_DATABASE_SYNC_URL (both, same role, same password — a half-flip leaves api and worker disagreeing about which role they connect as; docs/DEPLOYMENT.md:654-670).

Who consumes it. The api, worker and beat runtime DSNs — the unprivileged, RLS-enforced lane. Applied on production 2026-08-04 (docs/DEPLOYMENT.md:387-392).

Blast radius. The three app services restart. The maintenance lane (MAINTENANCE_DATABASE_SYNC_URL) is not affected — it names the owner and must keep doing so, or the next alembic upgrade head dies.

Rotation steps. scripts/create_app_role.py is idempotent and rotates the password on re-run, re-asserting the role attributes and grants (docs/DEPLOYMENT.md:545-569). Run it as the privileged role — i.e. before the app DSN change, from a container whose DSN is still the owner's, or with the owner DSN passed explicitly:

cd /opt/climate-lama
read -rs -p "new app-role password: " APP_PW; echo
./dc exec -T -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" \
  -e DATABASE_SYNC_URL="postgresql+psycopg2://<POSTGRES_USER>:<POSTGRES_PASSWORD>@postgres:5432/<POSTGRES_DB>" \
  api python scripts/create_app_role.py            # dry run — prints statements

./dc exec -T -e CLIMATE_LAMA_APP_DB_PASSWORD="$APP_PW" \
  -e DATABASE_SYNC_URL="postgresql+psycopg2://<POSTGRES_USER>:<POSTGRES_PASSWORD>@postgres:5432/<POSTGRES_DB>" \
  api python scripts/create_app_role.py --apply

Then update both APP_DATABASE_URL and APP_DATABASE_SYNC_URL in .env.prod with the new password, render, and restart:

./dc config | grep -E 'DATABASE_(SYNC_)?URL'
# every api/worker/beat DATABASE_URL and DATABASE_SYNC_URL must read
# climate_lama_app; MAINTENANCE_DATABASE_SYNC_URL must still read the owner
./dc up -d api worker beat

Verify — proves the NEW value works:

# a) real DB round-trip through the app's own DSN
curl -fsS https://api.climate-lama.online/health/ready | python -m json.tool
# b) the app is connecting AS climate_lama_app, and the role is still unprivileged
./dc exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname LIKE 'climate_lama%';"
./dc exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT usename, count(*) FROM pg_stat_activity
     WHERE datname = current_database() AND usename IS NOT NULL GROUP BY 1;"
# c) the maintenance lane was NOT collaterally damaged
for svc in api worker beat; do ./dc exec -T "$svc" printenv MAINTENANCE_DATABASE_SYNC_URL; done
./dc exec -T api alembic current

→ verify: (a) all ok; (b) climate_lama_app shows f | f and holds the api/worker/beat connections; (c) the owner DSN is present in all three services and alembic current prints (head).

Revoke the old value. ALTER ROLE … PASSWORD (what the script issues) replaces it atomically — the old password is dead immediately. Hygiene: purge it from the password manager and shred the pre-edit .env.prod.bak.* copies as in §3a.

Rollback if verification fails: blank the two APP_* lines and ./dc up -d api worker beat — the :- defaults resolve back to the POSTGRES_* owner DSN (docs/DEPLOYMENT.md:749-754). RLS is then inert, so treat it as a short window, not a resting state.

3c. `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` — Hetzner Object Storage¶

Where it lives. .env.prod. The values are Hetzner Object Storage S3 credentials for bucket climate-lama-storage, location nbg1 (docs/DEPLOYMENT.md:31).

Who consumes it — four services, not three. api, worker and beat read MINIO_* from env_file (docker-compose.prod.yml:55-57, :118, :166), and titiler consumes the same two values re-exported as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY via Compose interpolation (docker-compose.prod.yml:185-187). A rotation that restarts only the app trio leaves tile serving on the revoked key.

Blast radius. All dataset, tile and report I/O. Restart api, worker, beat and titiler.

Rotation steps:

Create a second, additional S3 credential pair in the Hetzner Cloud console for the same project/bucket — additive, so the old pair keeps working while you switch over. [ASSUMPTION: owner to fill] — the exact console navigation path, and whether Hetzner Object Storage permits two concurrent credential pairs on one project. Nothing in this repository documents the provider console; if only one pair is permitted, this becomes a cut-over with a short window in which object I/O fails, and the restart in step 3 must follow the key change immediately.
Update MINIO_ACCESS_KEY and MINIO_SECRET_KEY in .env.prod; back up and chmod 600 first.

Render, then restart all four consumers:

cd /opt/climate-lama
./dc config | grep -E 'MINIO_|AWS_(ACCESS|SECRET)'
./dc up -d api worker beat titiler

Verify — proves the NEW value works:

# a) the readiness probe's MinIO check requires the configured BUCKET to exist,
#    not merely that the endpoint answers (src/climate_lama/main.py:276-284)
curl -fsS https://api.climate-lama.online/health/ready | python -m json.tool
# b) titiler is healthy on its own credentials
./dc ps titiler                                   # State must be healthy
./dc exec -T titiler curl -fsS http://127.0.0.1/healthz
# c) a real signed read through the new key — fetch a tile for an existing COG
#    dataset through the API and confirm 200, not 403

→ verify: (a) minio: ok; (b) healthy and healthz 200; (c) a tile request returns image bytes. A stale key surfaces as 403/SignatureDoesNotMatch inside the worker logs rather than as a failed health check, so do not stop at (a):

./dc logs --since 10m worker | grep -iE 'signature|accessdenied|403' || echo "clean"

Revoke the old credential pair. In the Hetzner Cloud console, delete the superseded S3 credential after the checks above pass. [ASSUMPTION: owner to fill] — the console path for deleting an S3 credential. Then re-run (a)–(c): they must still pass, which is what proves the running stack is genuinely on the new pair and not still holding the old one in a cached client.

3d. `APP_SECRET_KEY` — the JWT signing key¶

Where it lives. .env.prod. Who consumes it. The api service only: create_access_token signs with it and decode_access_token verifies with it, HS256 (src/climate_lama/core/security.py:140, :162).

Blast radius — every access token in circulation is invalidated. There is no key-id/rollover mechanism: one key signs and verifies. On restart, every logged-in user's access token fails verification and they must log in again. Refresh tokens are not JWTs (they are peppered hashes — see §3e), so a client that holds a valid refresh token can re-obtain access without a full re-authentication; browser sessions in flight will still see 401s.

Rotation steps:

python -c "import secrets; print(secrets.token_hex(32))" → password manager.
Back up .env.prod, set APP_SECRET_KEY=<new>, chmod 600.
./dc config | grep APP_SECRET_KEY then ./dc up -d api. (worker and beat do not sign or verify tokens; restarting them is harmless but unnecessary.)

Verify — proves the NEW value works, and that the OLD one is dead:

API=https://api.climate-lama.online

# capture a token minted under the OLD key BEFORE the restart:
OLD=$(curl -fsS -X POST "$API/v1/auth/login" -H 'content-type: application/json' \
      -d '{"email":"<user>","password":"<pw>","org_slug":"<slug>"}' | python -c 'import json,sys;print(json.load(sys.stdin)["access_token"])')

# ...rotate and restart, then:

# a) the OLD token must now be rejected — this is the proof of rotation
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/auth/me" -H "authorization: Bearer $OLD"
# expect 401

# b) a FRESH login must succeed and its token must be accepted
NEW=$(curl -fsS -X POST "$API/v1/auth/login" -H 'content-type: application/json' \
      -d '{"email":"<user>","password":"<pw>","org_slug":"<slug>"}' | python -c 'import json,sys;print(json.load(sys.stdin)["access_token"])')
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/auth/me" -H "authorization: Bearer $NEW"
# expect 200

→ verify: (a) 401, (b) 200. (a) alone is the rotation proof — a 200 there means the api did not pick up the new key (check ./dc config and that up -d api actually recreated the container).

Login is rate-limited (login_rate_limit) and repeated failures lock the account, so do not loop these calls.

Revoke the old value. Nothing external to revoke; the key is dead as soon as the api restarts. Purge it from the password manager and shred the pre-edit .env.prod.bak.* copies as in §3a.

3e. `API_KEY_PEPPER` — read the warning before rotating¶

Who consumes it. The api service, as the shared pepper for three hash families: API keys, refresh tokens, and invites (src/climate_lama/core/security.py:227, :258, :289).

Blast radius: this is not a like-for-like rotation. The pepper is baked into stored hashes. Changing it does not re-key them — it makes every issued API key, every outstanding refresh token, and every pending invite unverifiable, permanently. There is no migration path in the codebase that re-peppers existing rows. Treat a pepper rotation as a deliberate mass revocation event, scheduled and announced, not as routine hygiene.

Prefer the per-key path for routine work. The API exposes POST /v1/auth/api-keys/{key_id}/rotate (src/climate_lama/api/v1/auth.py:661) and DELETE /v1/auth/api-keys/{key_id} (:702) — rotating an individual compromised key needs no .env.prod change and no restart.

If a full pepper rotation is genuinely required (pepper exposure):

Announce it — every integration holding an API key must be re-issued one.
python -c "import secrets; print(secrets.token_hex(32))" → password manager.
Back up .env.prod, set API_KEY_PEPPER=<new>, chmod 600, ./dc config | grep API_KEY_PEPPER, ./dc up -d api.
Re-issue API keys for every consumer via POST /v1/auth/api-keys (src/climate_lama/api/v1/auth.py:615).

Verify — proves the NEW value works:

API=https://api.climate-lama.online
# a) an API key issued BEFORE the rotation must now be rejected
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/auth/me" -H "authorization: Bearer clk_<old key>"
# expect 401
# b) a key issued AFTER the rotation must be accepted
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/auth/me" -H "authorization: Bearer clk_<new key>"
# expect 200
# c) refresh-token issuance still works end to end: log in, then exchange the
#    refresh token — it must succeed, proving new-pepper hashes verify

→ verify: (a) 401, (b) 200, (c) the refresh exchange returns a new access token.

Revoke the old value. The old pepper is dead on restart, and the old hashes it protected are already unusable. Purge it from the password manager and shred the pre-edit .env.prod.bak.* copies.

4. `SDK_SMOKE_*` repo secrets¶

Where they live. GitHub → Settings → Secrets and variables → Actions → repository secrets (not an environment — sdk-smoke.yml's jobs declare no environment:).

Who consumes them. sdk-smoke.yml only, in three places: the Detect smoke credentials job (:38-60), the Provision smoke user step that seeds the analyst user against the ephemeral CI stack (:127-148), and the smoke run itself (:151-164, :215-233).

Two mutually sufficient shapes, per the detection logic (:54-55):

SDK_SMOKE_TOKEN alone — an API key (clk_…) or a JWT; or
SDK_SMOKE_EMAIL + SDK_SMOKE_PASSWORD + SDK_SMOKE_ORG_SLUG, from which the smoke mints a JWT.

Which shape is currently configured: [ASSUMPTION: owner to fill] — repository secret values and their presence are not readable from the repository, and this runbook does not inspect them.

Blast radius — this is a release gate, not a runtime credential. The email/password/org-slug triple is seeded into the CI stack by the workflow itself, so it authenticates against an ephemeral database, not production. But it fails closed: if the secrets are absent or malformed, Detect smoke credentials emits configured=false, the End-to-end smoke job is skipped rather than vacuously green, and scripts/verify_ci_green.sh therefore fails the release gate (#470, ADR-046). A botched rotation blocks every publish (v* → GHCR, sdk-v* → PyPI) until it is fixed.

If SDK_SMOKE_TOKEN is in use it is a real API key against whichever stack issued it — rotate it as an API key (§3e's per-key path) as well as updating the secret.

Rotate¶

# triple form — the values are arbitrary; the workflow seeds this user itself
gh secret set SDK_SMOKE_EMAIL      --body '<new email>'
gh secret set SDK_SMOKE_PASSWORD   --body '<new password>'   # policy: >=12 chars, >=1 upper, >=1 digit
gh secret set SDK_SMOKE_ORG_SLUG   --body '<new slug>'

# token form
gh secret set SDK_SMOKE_TOKEN      --body '<new clk_… or JWT>'

Set all three of the triple together — the detection requires all three to be non-empty (sdk-smoke.yml:54-55), and the seed step hard-fails if SDK_SMOKE_EMAIL or SDK_SMOKE_PASSWORD is empty (:140-142).

Verify — proves the NEW values work¶

gh workflow run sdk-smoke.yml --ref main
gh run watch "$(gh run list --workflow sdk-smoke.yml -L1 --json databaseId --jq '.[0].databaseId')"

→ verify: the Detect smoke credentials job is green and the End-to-end smoke job concluded success — not skipped. A skipped smoke means the new values did not satisfy the detection, and the release gate is now closed.

Confirm the gate itself, at zero Actions minutes:

scripts/verify_ci_green.sh --sha "$(git rev-parse origin/main)"

→ verify: exit 0. This is the same check release.yml / sdk-publish-*.yml run before publishing, and it requires the named End-to-end smoke job to have concluded success.

sdk-smoke.yml is a release gate, not a merge gate — per the project's CI model it is dispatched only when cutting a tag or on explicit request. A rotation is an explicit request; one dispatch is the correct cost. Do not add the ci label to a PR to test it — that fires both ci.yml and sdk-smoke.yml on that PR.

Revoke the old values¶

gh secret set overwrites — the old value is gone from GitHub the moment the new one is stored; there is nothing further to delete there. What remains:

if the old SDK_SMOKE_TOKEN was a real API key, revoke it at the issuing stack: DELETE /v1/auth/api-keys/{key_id} (src/climate_lama/api/v1/auth.py:702) — otherwise a leaked-and-replaced secret is still a live key;
purge the old values from the password manager.

→ verify the revocation: the deleted key must be rejected — curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/auth/me" -H "authorization: Bearer clk_<old>" returns 401.

Unverified values — the register¶

Everything in this runbook is derived from files in this repository. The items below could not be, and are marked [ASSUMPTION: owner to fill] at their point of use as well as here:

#	What	Why it is not in the repo	Fill it in by
1	`ghcr-pull-hetzner` expiry date (§Inventory)	The expiry is chosen at PAT creation and stored only in GitHub's PAT settings and the password manager	Reading GitHub → Settings → Developer settings → Personal access tokens (classic), then recording it in §Inventory and setting a T-14-day reminder
2	Hetzner Object Storage console path for creating an S3 credential (§3c)	The repo names the provider and bucket but documents no console navigation	Doing it once and pasting the path
3	Whether Hetzner Object Storage allows two concurrent credential pairs (§3c)	Provider capability, not a repo fact — decides whether §3c is overlap-then-revoke or a cut-over	Checking the provider console/docs
4	Hetzner Object Storage console path for deleting a superseded credential (§3c)	Same as #2	Same as #2
5	Which `SDK_SMOKE_` shape is currently configured* — token, or the email/password/org triple (§4)	Repository secret values and presence are not readable from a checkout, and this runbook does not inspect them	Reading GitHub → Settings → Secrets and variables → Actions (names are listed even though values are not)

Rotation log¶

Append one row per rotation. This is the audit trail the .env.prod plane does not otherwise have — GitHub Actions secrets carry their own audit log, a file on a VPS does not.

Date (UTC)	Secret	Rotated by	Verification that passed	Old value revoked

Secret rotation runbook — the .env.prod + prod-host plane¶

Scope — read this first¶

Inventory¶

Rules that apply to every .env.prod rotation¶

1. GHCR PAT — ghcr-pull-hetzner¶

Rotate¶

Verify — proves the NEW token works¶

Revoke the old token¶

2. prod-host deploy keypair — DEPLOY_SSH_KEY / DEPLOY_SSH_KNOWN_HOSTS¶

Rotate¶

Verify — proves the NEW key works¶

Revoke the old key¶

3. Application secrets in .env.prod¶

3a. POSTGRES_PASSWORD — the owner role climate_lama¶

3b. The climate_lama_app role password (#556)¶

3c. MINIO_ACCESS_KEY / MINIO_SECRET_KEY — Hetzner Object Storage¶

3d. APP_SECRET_KEY — the JWT signing key¶

3e. API_KEY_PEPPER — read the warning before rotating¶

4. SDK_SMOKE_* repo secrets¶

Rotate¶

Verify — proves the NEW values work¶

Revoke the old values¶

Unverified values — the register¶

Rotation log¶

Secret rotation runbook — the `.env.prod` + prod-host plane¶

Rules that apply to every `.env.prod` rotation¶

1. GHCR PAT — `ghcr-pull-hetzner`¶

2. `prod-host` deploy keypair — `DEPLOY_SSH_KEY` / `DEPLOY_SSH_KNOWN_HOSTS`¶

3. Application secrets in `.env.prod`¶

3a. `POSTGRES_PASSWORD` — the owner role `climate_lama`¶

3b. The `climate_lama_app` role password (#556)¶

3c. `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` — Hetzner Object Storage¶

3d. `APP_SECRET_KEY` — the JWT signing key¶

3e. `API_KEY_PEPPER` — read the warning before rotating¶

4. `SDK_SMOKE_*` repo secrets¶