Engine parity harness¶

The parity harness compares climate-lama-engine output against a CLIMADA reference on identical inputs. It is the scaffold laid down by issue #206; Phase-3 tiers build on top of it to grow hazard coverage and metric depth.

Why¶

Our default compute path (climate-lama-engine) is a trimmed re-implementation of CLIMADA's impact calculation. Once we diverge from CLIMADA, we need a mechanical way to prove that:

a given fixture → engine → impact matrix, and
the same fixture → CLIMADA → impact matrix,

agree within a documented tolerance. Without this, engine drift is invisible until someone audits a specific number by hand.

ADR-024 boundary — why CLIMADA is subprocess-only¶

ADR-024 forbids the backbone from importing CLIMADA. The parity harness must therefore never run import climada inside the backbone process. Instead:

 backbone process                            sidecar container
 ┌────────────────────────┐      JSON       ┌────────────────────────┐
 │ tests/parity/harness.py│ ─── stdin  ───▶ │ sidecar_runner.py      │
 │  - builds fixtures     │                 │  - import climada      │
 │  - runs EngineAdapter  │ ◀─── stdout ─── │  - ImpactCalc(...)     │
 │  - diffs the outputs   │      JSON       │  - writes result       │
 └────────────────────────┘                 └────────────────────────┘

tests/parity/harness.py is pure backbone code — it only imports climate_lama_engine (via the adapter) and the fixture generator.
tests/parity/sidecar_runner.py is the only module in the repository permitted to import climada, and it is only ever executed inside the sidecar image, never as part of the backbone's own test run.
Communication is a single JSON object on stdin and a single JSON object on stdout. Any diagnostics go to stderr so stdout stays clean.

This keeps ADR-024's "no CLIMADA in the backbone" guarantee watertight: the sidecar image has its own Python, its own dependencies, and dies at the end of the test run.

What ships today¶

Piece	Path	Notes
Harness	`tests/parity/harness.py`	`build_inputs`, `run_engine`, `run_climada_sidecar`, `compare`, `ParityResult.assert_within_tolerance`
Sidecar entrypoint	`tests/parity/sidecar_runner.py`	Executes inside the sidecar container only
Sidecar image	`tests/parity/Dockerfile.climada-sidecar`	`python:3.11-slim` + pinned `climada`
PoC test	`tests/parity/test_parity_river_flood.py`	Skipped unless sidecar is configured; asserts river-flood EAD within 1%
Gated workflow	`.github/workflows/parity.yml`	`workflow_dispatch` only

Default CI (ci.yml) does not build the sidecar or run these tests — the CLIMADA install alone takes several minutes and pulls in a large chain of scientific dependencies. Running the harness is a deliberate, on-demand action.

Running it locally¶

Two ways to point the harness at a CLIMADA runtime:

1. Docker sidecar (matches CI)

docker build \
  -f tests/parity/Dockerfile.climada-sidecar \
  -t climate-lama-climada-sidecar \
  tests/parity

CLIMADA_SIDECAR_IMAGE=climate-lama-climada-sidecar \
  uv run pytest tests/parity/ -v

2. Explicit command (local CLIMADA venv)

If you already have CLIMADA installed in a separate virtualenv, skip Docker:

CLIMADA_SIDECAR_CMD="/path/to/climada-venv/bin/python tests/parity/sidecar_runner.py" \
  uv run pytest tests/parity/ -v

With neither env var set — i.e. the default developer state — the parity tests collect but are skipped. This keeps uv run pytest fast and keeps CLIMADA off the dependency graph for contributors who don't need it.

Running it in CI¶

Trigger the Engine parity (CLIMADA) workflow from the GitHub Actions UI (workflow_dispatch). Optionally pass a pytest -k filter to narrow the run. The workflow:

Installs the backbone's dev,worker extras.
Builds the sidecar image from tests/parity/Dockerfile.climada-sidecar.
Runs pytest tests/parity/ with CLIMADA_SIDECAR_IMAGE pointed at the freshly built image.

Tolerance policy¶

The scaffold's one shipped test asserts EAD agreement within 1% on a small river-flood fixture. Per-event impact arrays are checked at the same rtol. These bounds are strict on purpose — the harness exists to catch divergence, not paper over it. When a legitimate algorithmic difference appears (e.g. the engine chooses a different summation order), the resolution is a dedicated follow-up issue, not a looser tolerance.

Future work¶

Phase-3 tiers layered on top of this scaffold extend coverage:

Tropical cyclone, wildfire, and storm-Europe parity fixtures.
Cost-benefit (calc_cost_benefit) parity with a small measure set.
A nightly scheduled variant of the parity workflow once runtime is acceptable.

All of those live under tests/parity/ and reuse the same harness + sidecar pair — no second CLIMADA import path.