Skip to content

Observability

Climate-Lama ships an opt-in observability stack built on Prometheus (metrics), Grafana (dashboards), and Loki + Promtail (log aggregation). The stack runs as a Docker Compose profile — there are no always-on sidecars in the base deployment.

The full rationale and alternatives are recorded in ADR-030.

What's instrumented

Surface Mechanism Metrics
FastAPI HTTP traffic prometheus-fastapi-instrumentator auto-wired in main.py http_requests_total, http_request_duration_seconds_*
Celery tasks (all) task_prerun / task_postrun / task_failure signals in worker/celery_app.py climate_lama_celery_task_runs_total{task_name, status}, climate_lama_celery_task_duration_seconds_*, climate_lama_celery_task_failures_total{task_name}
Compute jobs (hazard-scoped) In-task emission in worker/tasks.py climate_lama_job_completions_total{hazard_type, status}, climate_lama_job_duration_seconds_*
PostgreSQL postgres_exporter sidecar scraped by Prometheus pg_stat_activity_count, connection-pool and query metrics
Redis / Celery queue depth redis_exporter sidecar redis_key_size{key="compute"}
Container logs Promtail tails Docker containers labelled logging=promtail → Loki structured log lines keyed by container

Endpoints

  • /metrics on the API pod — Prometheus text format, unauthenticated. The rationale and threat model are in ADR-030; operators who need to gate it should restrict the port at the ingress/network level or run a reverse-proxy with auth in front of the API.
  • Grafana UI — http://localhost:3001 when running the compose profile (admin / admin by default; change via GF_SECURITY_ADMIN_PASSWORD).
  • Prometheus UI — http://localhost:9090.
  • Loki API — http://localhost:3100.

Running the stack locally

# Base services only (default):
docker compose up

# Base services + Prometheus + Grafana + Loki + Promtail + exporters:
docker compose --profile monitoring up

The bundled Grafana dashboard (infra/monitoring/grafana/provisioning/dashboards/climate-lama.json) is auto-provisioned on Grafana startup and covers:

  • Request rate and 5xx error ratio per endpoint
  • Request latency p50 / p95 / p99 per endpoint
  • Celery task rate and duration p95 per task name
  • Celery queue depth (compute queue)
  • Postgres connection count by state
  • Job completions by hazard type and status (legacy panel, retained for continuity)

OpenTelemetry tracing (opt-in)

FastAPI → Celery span propagation is wired via OpenTelemetry but is off by default and not installed by default. Activating it is a two-step opt-in:

  1. Install the optional extra in the deployment environment:
pip install -e .[otel]
  1. Set the environment variable and point to an OTLP collector:
export OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

The API and worker both read OTEL_ENABLED and install FastAPIInstrumentor / CeleryInstrumentor at startup. When OTEL_ENABLED is unset or the otel extra is missing, the code path is a no-op — no collector is required for normal operation.

Alerts

A minimal ruleset lives at infra/monitoring/prometheus_alerts.yml and currently covers the job-failure rate. Add new rules alongside it; Prometheus picks them up without a restart via the /-/reload lifecycle endpoint.