Engine parity harness¶

The parity harness compares climate-lama-engine output against a CLIMADA reference on identical inputs. It is the scaffold laid down by issue #206; Phase-3 tiers build on top of it to grow hazard coverage and metric depth.

Why¶

Our default compute path (climate-lama-engine) is a trimmed re-implementation of CLIMADA's impact calculation. Once we diverge from CLIMADA, we need a mechanical way to prove that:

a given fixture → engine → impact matrix, and
the same fixture → CLIMADA → impact matrix,

agree within a documented tolerance. Without this, engine drift is invisible until someone audits a specific number by hand.

ADR-024 boundary — why CLIMADA is subprocess-only¶

ADR-024 forbids the backbone from importing CLIMADA. The parity harness must therefore never run import climada inside the backbone process. Instead:

 backbone process                            sidecar container
 ┌────────────────────────┐      JSON       ┌────────────────────────┐
 │ tests/parity/harness.py│ ─── stdin  ───▶ │ sidecar_runner.py      │
 │  - builds fixtures     │                 │  - import climada      │
 │  - runs EngineAdapter  │ ◀─── stdout ─── │  - ImpactCalc(...)     │
 │  - diffs the outputs   │      JSON       │  - writes result       │
 └────────────────────────┘                 └────────────────────────┘

tests/parity/harness.py is pure backbone code — it only imports climate_lama_engine (via the adapter) and the fixture generator.
docker/climada-sidecar/server.py is the only module in the repository permitted to import climada. It is the production compute module, shared by this image and the runtime HTTP sidecar since issue #1104, so a parity number describes the code path a real request takes.
tests/parity/sidecar_runner.py is a thin entrypoint over it — it adapts the stdin/stdout protocol onto server.compute_impact and contains no CLIMADA code of its own. Both files are only ever executed inside the sidecar image, never as part of the backbone's own test run. (Its load_production_server helper is the one exception: it imports no CLIMADA, and the backbone's test suite uses it to exercise server.py's protocol layer off-container.)
Communication is a single JSON object on stdin and a single JSON object on stdout. Any diagnostics go to stderr so stdout stays clean.

This keeps ADR-024's "no CLIMADA in the backbone" guarantee watertight: the sidecar image has its own Python, its own dependencies, and dies at the end of the test run.

What ships today¶

Piece	Path	Notes
Harness	`tests/parity/harness.py`	`build_inputs`, `run_engine`, `run_climada_sidecar`, `compare`, `ParityResult.assert_within_tolerance`
Sidecar entrypoint	`tests/parity/sidecar_runner.py`	stdin/stdout shim over `server.compute_impact`; executes inside the sidecar container only
Compute module	`docker/climada-sidecar/server.py`	Shared with the runtime HTTP sidecar (#1104) — the only file that imports `climada`
Sidecar image	`tests/parity/Dockerfile.climada-sidecar`	`python:3.11-slim` + pinned `climada`; built from the repo root context
Protocol tests	`tests/test_worker/test_climada_sidecar_server.py`	Runs `server.py`'s CSR/insured/HTTP paths in-process, CLIMADA stubbed; no container
PoC test	`tests/parity/test_parity_river_flood.py`	Skipped unless sidecar is configured; asserts river-flood EAD within 1%
Gated workflow	`.github/workflows/parity.yml`	`workflow_dispatch` only

Default CI (ci.yml) does not build the sidecar or run these tests — the CLIMADA install alone takes several minutes and pulls in a large chain of scientific dependencies. Running the harness is a deliberate, on-demand action.

Running it locally¶

Two ways to point the harness at a CLIMADA runtime:

1. Docker sidecar (matches CI)

docker build \
  -f tests/parity/Dockerfile.climada-sidecar \
  -t climate-lama-climada-sidecar \
  .

CLIMADA_SIDECAR_IMAGE=climate-lama-climada-sidecar \
  uv run pytest tests/parity/ -v

2. Explicit command (local CLIMADA venv)

If you already have CLIMADA installed in a separate virtualenv, skip Docker:

CLIMADA_SIDECAR_CMD="/path/to/climada-venv/bin/python tests/parity/sidecar_runner.py" \
  uv run pytest tests/parity/ -v

With neither env var set — i.e. the default developer state — the parity tests collect but are skipped. This keeps uv run pytest fast and keeps CLIMADA off the dependency graph for contributors who don't need it.

Running it in CI¶

Trigger the Engine parity (CLIMADA) workflow from the GitHub Actions UI (workflow_dispatch). Optionally pass a pytest -k filter to narrow the run. The workflow:

Installs the backbone's dev,worker extras.
Builds the sidecar image from tests/parity/Dockerfile.climada-sidecar.
Runs pytest tests/parity/ with CLIMADA_SIDECAR_IMAGE pointed at the freshly built image.

Tolerance policy¶

The scaffold's one shipped test asserts EAD agreement within 1% on a small river-flood fixture. Per-event impact arrays are checked at the same rtol. These bounds are strict on purpose — the harness exists to catch divergence, not paper over it. When a legitimate algorithmic difference appears (e.g. the engine chooses a different summation order), the resolution is a dedicated follow-up issue, not a looser tolerance.

Future work¶

Phase-3 tiers layered on top of this scaffold extend coverage:

Tropical cyclone, wildfire, and storm-Europe parity fixtures.
Cost-benefit (calc_cost_benefit) parity with a small measure set.
A nightly scheduled variant of the parity workflow once runtime is acceptable.

All of those live under tests/parity/ and reuse the same harness + sidecar pair — no second CLIMADA import path.