Self-hosted CI runner — implementation guide

Status: Proposed — awaiting go/no-go. On adoption, record the decision as an ADR (next free number at the time of writing: ADR-048 — re-check docs/DECISIONS.md, ADRs are minted frequently) and work through the post-adoption follow-ups. Date: 2026-08-03 Driver: #516 / PR #517 (Actions minutes diet). 2026-08-01 burned 253 billed minutes in one 30-merge day — 12.7% of the 2,000-min/month free pool. The diet (single-job CI, debounced trunk gate, smoke as release-gate-only) projects ~20–40 min/day, which fits the pool but leaves no headroom for restoring richer verification cadence (per-wave gates, routine smoke, the 30-min slow memory gate). A self-hosted runner removes the ceiling: jobs on self-hosted runners bill zero minutes, on any plan, for private repos. Audience: a human or an agent implementing this end-to-end. Every step has a verification check. Total hands-on time ≈ 30–45 minutes plus one CI validation cycle.

---

1. Decision summary

Choice	Decision	Why
Machine	New dedicated Hetzner Cloud VM	The prod box (2 vCPU / 3.7 GiB, no swap) already OOM-killed workers under load and runs the production compose stack; CI on it would contend with prod and put repo-controlled code next to prod data. A CI box is disposable; the prod box is not.
Architecture	x86-64 (Intel/AMD), not ARM	Bit-for-bit parity with ubuntu-latest history and with prod images (amd64). postgis/postgis and minio/minio are multi-arch, and most Python wheels ship aarch64 manylinux — but "most" is where debugging weekends come from. The ARM saving is ~€1/mo; not worth the variance.
Size	CX32 (4 vCPU / 8 GiB / 80 GB) recommended; CX22 (2 vCPU / 4 GiB / 40 GB) minimum	Checks + service PostGIS fits in 4 GiB, but the SDK smoke boots the full compose stack (api + worker + Postgres + MinIO + Redis) and runs pytest beside it — that is exactly the workload that OOM-killed the 3.7 GiB prod box. ~€3/mo extra buys never thinking about it. Prices drift; check current Hetzner pricing at implementation time (CX32 was ~€6.8/mo as of mid-2026).
OS	Ubuntu 24.04 LTS	Same family as ubuntu-latest; runner and Docker support are first-class.
Runner scope	Repo-level, climate-lama only (phase 1)	Personal GitHub accounts have no org-level runners — registration is per-repo. climate-lama was ~95% of the 2026-08-01 burn. Other repos are phase 2 (§9).
Runner lifecycle	Persistent, as a systemd service (svc.sh)	Ephemeral (--ephemeral) is better hygiene but needs a token-minting loop (registration tokens expire in 1 h). Not worth the machinery for a single-owner private setup; noted as a hardening option in §8.
Wiring	runs-on: ${{ vars.CI_RUNNER \|\| 'ubuntu-latest' }} with repo variable CI_RUNNER=hetzner-ci	Switching runner ⇄ GitHub-hosted is a repo-variable flip — no workflow edit, no redeploy, instant rollback. Unset variable ⇒ empty string ⇒ falls back to ubuntu-latest. The vars context is available in runs-on expressions.
Never	No runner on riskwise-v2 (public repo)	GitHub's own hard warning: on a public repo, a fork PR can execute arbitrary code on your runner. All runner-equipped repos must be private.

Billing facts (verified 2026-07-31, re-verify at implementation): self-hosted runner minutes are free on all plans; GitHub shelved a planned $0.002/min charge (may return); private-repo scheduled workflows never auto-disable.

---

2. Prerequisites

• Hetzner Cloud account with a project (the prod box's project is fine — the server is separate, not the project).
• An SSH public key registered in that Hetzner project. Note: on this workstation, working SSH to Hetzner boxes goes via WSL Ubuntu (wsl -d Ubuntu -- ssh ...) — Windows-side keys have been rejected by the prod box before. Reuse the WSL key pair.
• gh CLI authenticated as CortoMaltese3 with admin on climate-lama (needed to mint the runner registration token and set repo variables). Run gh on the workstation, not on the runner box — the box should never hold gh credentials.
• The repo's CI must be green on GitHub-hosted runners before switching (see §6 step 0) — otherwise a red run on the new box is unattributable.

---

3. Provision the server

Console: Hetzner Cloud → project → Add Server → Ubuntu 24.04, CX32, the SSH key, one location (any EU location; latency to GitHub is irrelevant). Or with the hcloud CLI:

hcloud server create --name ci-runner-1 --type cx32 --image ubuntu-24.04 \
  --ssh-key <key-name> --location nbg1

Attach a Hetzner Cloud Firewall allowing inbound SSH (22/tcp) only (optionally restricted to your IP). The runner needs zero inbound — it long-polls GitHub over outbound HTTPS. Prefer the Hetzner firewall over ufw-only: it survives OS-level misconfiguration.

Verify: wsl -d Ubuntu -- ssh root@<new-ip> 'echo ok && uname -m' → ok + x86_64.

---

4. Bootstrap the box

One idempotent script, run as root. It creates the unprivileged runner user, installs Docker (the services: containers and the smoke's docker compose require it), preinstalls the packages CI jobs would otherwise sudo apt-get install (the runner user gets no sudo — see §8), adds swap (the prod box's OOM history says: never again swapless), and sets up cleanup + unattended security updates.

#!/usr/bin/env bash
# bootstrap-ci-runner.sh — run as root on the fresh Ubuntu 24.04 box.
set -euo pipefail

# --- base + security updates ------------------------------------------------
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y \
  git curl ca-certificates unattended-upgrades \
  libpango-1.0-0 libpangoft2-1.0-0 fonts-dejavu-core   # WeasyPrint libs the Checks job guards on
dpkg-reconfigure -f noninteractive unattended-upgrades

# --- swap (2G) — the 3.7GiB swapless prod box OOM-killed workers; don't repeat it
if ! swapon --show | grep -q /swapfile; then
  fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
  echo '/swapfile none swap sw 0 0' >> /etc/fstab
fi

# --- docker engine + compose + buildx (get.docker.com installs all three) ----
if ! command -v docker >/dev/null; then
  curl -fsSL https://get.docker.com | sh
fi

# --- unprivileged runner user, docker-capable, NO sudo -----------------------
id ghrunner >/dev/null 2>&1 || useradd -m -s /bin/bash ghrunner
usermod -aG docker ghrunner

# --- weekly docker/disk cleanup ----------------------------------------------
cat > /etc/cron.weekly/ci-cleanup <<'EOF'
#!/bin/sh
docker system prune -af --filter "until=168h" >/dev/null 2>&1
docker volume prune -f >/dev/null 2>&1
EOF
chmod +x /etc/cron.weekly/ci-cleanup

echo "bootstrap done"

Verify: sudo -u ghrunner docker run --rm hello-world prints the hello message (proves the docker group membership without sudo), and swapon --show lists /swapfile.

---

5. Register the runner

On the workstation, mint a registration token (valid 1 hour — mint it right before use):

gh api -X POST repos/CortoMaltese3/climate-lama/actions/runners/registration-token --jq .token

On the box, as ghrunner (su - ghrunner):

mkdir -p ~/actions-runner && cd ~/actions-runner
VER=$(curl -fsSL https://api.github.com/repos/actions/runner/releases/latest | grep -oP '"tag_name": "v\K[^"]+')
curl -fsSL -o runner.tar.gz -L \
  "https://github.com/actions/runner/releases/download/v${VER}/actions-runner-linux-x64-${VER}.tar.gz"
tar xzf runner.tar.gz && rm runner.tar.gz
sudo ./bin/installdependencies.sh   # .NET prerequisites — the ONE root step; run it as root instead if ghrunner has no sudo (it shouldn't)
./config.sh --url https://github.com/CortoMaltese3/climate-lama \
  --token <TOKEN-FROM-WORKSTATION> \
  --name hetzner-ci-1 \
  --labels hetzner-ci \
  --work _work \
  --unattended

(installdependencies.sh needs root: run that single line from the root session, then continue as ghrunner.) Then install it as a service so it survives reboots, running as ghrunner:

# as root, in /home/ghrunner/actions-runner
./svc.sh install ghrunner
./svc.sh start
./svc.sh status    # expect: active (running)

The runner self-updates; no version pinning needed. Two lifecycle facts worth knowing: GitHub auto-removes a registered runner that stays offline > 14 days, and a job that finds no online matching runner queues for up to 24 h, then fails (see §7).

Verify (workstation):

gh api repos/CortoMaltese3/climate-lama/actions/runners \
  --jq '.runners[] | {name, status, labels: [.labels[].name]}'
# expect: name hetzner-ci-1, status "online", labels including "hetzner-ci"

---

6. Wire the workflows

Four runs-on sites move; everything else is untouched. Line numbers as of ab03b74 — re-locate by job name, not line number.

.github/workflows/ci.yml — jobs Checks and Slow tests (memory-bound, opt-in); .github/workflows/sdk-smoke.yml — jobs Detect smoke credentials and End-to-end smoke. In all four:

# before
    runs-on: ubuntu-latest
# after
    runs-on: ${{ vars.CI_RUNNER || 'ubuntu-latest' }}

Plus one mandatory fix in the slow job (ci.yml, "Provision MinIO bucket and fixture" step): it writes mc to /usr/local/bin, which is root-owned on a self-hosted box (GitHub-hosted images happen to make it runner-writable). Use the per-job temp dir — works identically on both runner types:

# before
          curl -fsSL -o /usr/local/bin/mc \
            https://dl.min.io/client/mc/release/linux-amd64/mc
          chmod +x /usr/local/bin/mc
          mc alias set ci http://localhost:9000 minioadmin minioadmin
          mc mb --ignore-existing ci/climate-lama
          ...
          mc cp /tmp/rf-5gib.nc ci/climate-lama/fixtures/memory-bound/rf-5gib.nc
# after
          curl -fsSL -o "$RUNNER_TEMP/mc" \
            https://dl.min.io/client/mc/release/linux-amd64/mc
          chmod +x "$RUNNER_TEMP/mc"
          "$RUNNER_TEMP/mc" alias set ci http://localhost:9000 minioadmin minioadmin
          "$RUNNER_TEMP/mc" mb --ignore-existing ci/climate-lama
          ...
          "$RUNNER_TEMP/mc" cp /tmp/rf-5gib.nc ci/climate-lama/fixtures/memory-bound/rf-5gib.nc

Do not touch scripts/verify_ci_green.sh — it keys on job names (Checks, End-to-end smoke), which don't change. Leave docs.yml, release.yml, sdk-publish-*.yml, sdk-regenerate.yml and parity.yml on GitHub-hosted in phase 1: together they are a few minutes per week, and the publish paths benefit from a clean throwaway environment.

Land the workflow edits through the normal flow (they only take effect after merge to main for workflow_dispatch runs). Then flip the switch:

gh variable set CI_RUNNER --repo CortoMaltese3/climate-lama --body hetzner-ci

Rollback at any time (also works while jobs are stuck queued — cancel them after):

gh variable delete CI_RUNNER --repo CortoMaltese3/climate-lama

---

7. Validation runbook

Run in order; each step gates the next.

1. Baseline (before flipping the variable): dispatch ci.yml on main on GitHub-hosted and confirm green — and if smoke will move too, confirm sdk-smoke.yml is green on hosted first (it was red on main 2026-08-01 from a real SDK bug, #505/#510 — a red suite makes runner validation unattributable). → verify: gh run watch <id> --exit-status.
2. Flip: set CI_RUNNER=hetzner-ci (§6). → verify: gh variable list --repo ... shows it.
3. Dispatch: gh workflow run ci.yml --ref main. → verify the job actually ran on the box, not just green:

   RUN=$(gh run list --workflow ci.yml --branch main -L 1 --json databaseId --jq '.[0].databaseId')
   gh run watch "$RUN" --exit-status
   gh api repos/CortoMaltese3/climate-lama/actions/runs/"$RUN"/jobs \
     --jq '.jobs[] | {name, conclusion, runner_name}'
   # expect: runner_name == "hetzner-ci-1" for Checks
   

4. Release gate still passes: bash scripts/verify_ci_green.sh --sha "$(git rev-parse origin/main)" (or with --check "ci.yml=Checks" if smoke hasn't been dispatched for that SHA) → PASS.
5. Service containers: step 2's green already proves PostGIS services: work (the Checks test step needs the DB). If it failed at "Initialize containers", docker/permissions on the box are the suspect — re-run §4's verify.
6. Smoke on the box (only with smoke green on hosted, step 0): gh workflow run sdk-smoke.yml --ref main, same runner_name check. Known wrinkle: the compose build's type=gha BuildKit layer cache (docker-compose.ci.yml) is expected to work from self-hosted (the runtime token is job-scoped, and docker/setup-buildx-action runs a container-driver builder), but if the build fails at cache import/export, drop to docker compose -f docker-compose.yml up -d --wait --build without the ci overlay — on a persistent box, the local Docker layer cache makes the GHA cache redundant anyway.
7. Slow job once (optional but recommended — it's free now): gh workflow run ci.yml --ref main -f run_slow=true → both jobs green, mc fix from §6 proven.
8. Rollback drill (do it now, not during an incident): delete the variable, dispatch, confirm runner_name is a GitHub-hosted name (GitHub Actions NN), re-set the variable.
9. Reboot drill: reboot the box; after it returns, gh api .../actions/runners shows online (proves svc.sh autostart), and a dispatch completes.

---

8. Security model

• Blast radius: anything that runs in CI (our own private-repo code + its dependencies) executes on this box as ghrunner and can read every Actions secret exposed to the job, poison local Docker/uv caches, and persist between jobs (the box is not wiped per job, unlike GitHub-hosted). Supply-chain compromise of a dependency is the realistic threat, not fork PRs (private repos; collaborator-only).
• Therefore: the box is single-purpose (CI only), holds no prod SSH keys, no prod secrets, no gh login, and prod keeps no trust of the CI box (nothing on prod authorizes it). Compromise of CI must not be lateral movement into prod.
• ghrunner has no sudo. Every package a workflow may need is preinstalled by §4 (the Checks WeasyPrint step's dpkg -s ... || guard then never reaches its sudo apt-get). If a future workflow adds an apt dependency, add it to the bootstrap script and the box — do not grant NOPASSWD: apt-get (apt hooks make that root-equivalent).
• Membership in the docker group is root-equivalent on this box — accepted for a single-purpose CI machine; it is why the box holds nothing worth taking.
• Inbound: SSH only (Hetzner firewall). The runner is outbound-HTTPS long-polling.
• Hardening later, if wanted: --ephemeral + a re-registration loop (clean slate per job), or rootless Docker. Neither blocks phase 1.

---

9. Failure modes, maintenance, scope extensions

Failure	Symptom	Fix
Box down / runner offline	dispatched runs sit Queued (up to 24 h, then fail)	gh variable delete CI_RUNNER → hosted fallback; cancel queued runs and re-dispatch. Then ./svc.sh status / reboot the box.
Offline > 14 days	GitHub silently deletes the runner registration	Re-register (§5). The 24 h queue symptom appears first.
Disk full	docker build/pull failures	docker system prune -af (cron does weekly); resize is a console click.
Two runners on one box	services: port collisions (both Checks and slow map host 5432)	Don't. One runner per box; a second repo's runner goes on its own box, or accept serial queueing on one registration.
Docker Hub rate limits	image pull errors	Rare on a persistent box — postgis/minio images stay in the local cache between runs (a hosted-runner cost this setup removes). If hit: docker login with a free Hub account.

Standing maintenance ≈ zero: runner self-updates, unattended-upgrades patches the OS, the weekly cron prunes Docker. A monthly gh api .../actions/runners glance (or the optional pre-dispatch check below, wired into the trunk-gate snippet in CLAUDE.md) covers the rest:

# optional debounce-preamble guard: hosted fallback if the box is dark
gh api repos/CortoMaltese3/climate-lama/actions/runners \
  --jq '[.runners[] | select(.status=="online")] | length'   # 0 -> delete CI_RUNNER before dispatching

Phase 2 candidates, each its own decision: climate-lama-engine and climate-lama-ui runners (separate registrations; see the port-collision row before co-hosting), release.yml image builds (needs qemu/binfmt for multi-arch), docs.yml (hardly worth it at ~1 min/weekday). Never riskwise-v2 (public — §1).

---

10. Post-adoption follow-ups

• [ ] Record the decision as an ADR in docs/DECISIONS.md (context: #516's measurements; alternatives: stay on hosted within the diet budget, RAM-upgrade the prod box and co-host — rejected per §1).
• [ ] Update CLAUDE.md "CI model" section: minutes no longer bind, but keep the debounced trunk gate — the constraint becomes wall-clock on a single serial runner, and batching still bounds revert scope; optionally re-enable per-batch smoke now that it's free.
• [ ] Add the runner-offline guard (§9) next to the debounce snippet in CLAUDE.md.
• [ ] Update the gh-actions-minutes-posture memory: runner live, date, box specs/IP, rollback = variable delete.
• [ ] Store the box in the Hetzner project with an obvious name (ci-runner-1) and add it to whatever inventory tracks the prod box.
• [ ] After two quiet weeks: consider restoring richer cadence (per-wave gates, routine run_slow=true) — the diet's austerity was a billing artifact, not a testing ideal.