Data pipeline

Open Canadian data, staged so each step is inspectable.

data/external/  Stage 0  pristine downloads + manifest.json (sha256, retrieved, licence)
data/interim/   Stage 1  cleaned, Canada-filtered, standard units (PJ/GW/$2020)
data/mapped/    Stage 2  re-keyed to OSeCAN codes (province→grid, sector, fuel)
data/params/    Stage 3  otoole parameters, harmonized against the SETS

Retrieve (Stage 0)

uv run osecan retrieve      # reads config/data_sources.yaml

Downloads every source with a URL, writing sha256 + timestamp + licence to data/external/manifest.json — the reproducibility record.

Clean & map (Stage 1→3)

Worked example — CER end-use demand → SpecifiedAnnualDemand:

uv run osecan clean-demand "Current Measures"

Filters to the scenario, aggregates NB/NS/PE→T and YT/NT/NU→N, maps sectors/fuels to OSeCAN codes, and emits otoole rows keyed <sector><fuel><grid>. Validated: national electricity 2025 ≈ 2,040 PJ (QC>ON>AB>BC), matching known provincial demand.

Harmonize (gate)

uv run osecan harmonize     # fails if any param references a code/year not in the SETS

Sources

Anchor: CER Energy Futures 2026 (OGL-Canada). Supporting: StatCan RESD, ECCC NIR, federal OBPS, CODERS, RESource. Internal soft-link: WS3 (biomass), FABLE (agriculture). Full table in data/SOURCES.md.