RESource Step Cache#

RESource.step_cache implements a lightweight, config-hash-based cache that prevents pipeline steps from re-running when their configuration has not changed. It is not a full data version control system — it is deliberately minimal: hash a config subset, check for data in the store, skip or run.

The Problem It Solves#

The RESource pipeline has eight sequential steps. Each step is computationally expensive. On a re-run — whether to change an economic parameter, update turbine specs, or test a new clustering tolerance — every step was previously re-executed from scratch regardless of what actually changed.

The step cache breaks that coupling. Only steps whose governing configuration has changed (or whose output is missing) will execute. Steps whose inputs are unchanged and whose output is already in the HDF5 store are skipped.

Mechanism#

Before each pipeline step, StepCache.is_current(step, store_key) checks two conditions:

Config hash match — SHA-256 of the config sections relevant to that step equals the hash stored from the previous run.
Data present — the HDF5 store contains the expected output dataset.

Both conditions must hold to skip the step. If either fails, the step runs and its hash is updated on completion.

Hashes are persisted to a JSON sidecar file next to the HDF5 store:

data/store/<Country>/<RUN_ID>/resources_<Country>_<Region>_<RUN_ID>.h5
data/store/<Country>/<RUN_ID>/resources_<Country>_<Region>_<RUN_ID>.checksums.json

Wind and solar results share the same HDF5 file but are namespaced independently in the sidecar (wind::grid_cells vs. solar::grid_cells), so a turbine config change only invalidates wind steps — solar cache entries are unaffected.

Config Keys Per Step#

Each step monitors a specific subset of config keys. Changing any key in a step's subset invalidates that step and all downstream steps whose inputs depend on it.

Step	Config keys monitored
`grid_cells`	`grid_cell_resolution`, `region_mapping`, `GADM`, `default_CRS`
`grid_nodes`	`OSM_data`, `region_mapping`, `capacity_disaggregation`
`cell_capacity`	`custom_land_layers`, `CORINE`, `GAEZ`, `capacity_disaggregation`, `region_mapping`
`weather_data`	`cutout`, `weather_year`, `region_mapping`
`gwa_scaling`	`GWA`, `region_mapping`
`cf_timeseries`	`cutout`, `capacity_disaggregation`, `weather_year`
`scoring`	`economic_parameters`, `NREL`, `capacity_disaggregation`
`clustering`	`capacity_disaggregation`, `region_mapping`

Note: Downstream invalidation is not automatic. If you change a grid parameter and want scoring to reflect the new geometry, either pass clean_store=True to build() or call cache.invalidate_all() before the run. The cache guards individual steps, not dependency chains.

Usage#

Normal pipeline run (automatic)#

StepCache is instantiated inside RESources_builder.build() and requires no direct interaction from the caller. Pass clean_store=True to force a full re-run:

builder = RESources_builder(
    config_file_path="config/config_CAN_baseline.yaml",
    region_short_code="BC",
    resource_type="wind",
    weather_year=2024,
)

# Skips unchanged steps automatically
builder.build(select_top_sites=True, use_grid_lines=True)

# Forces all steps to re-run
builder.build(select_top_sites=True, use_grid_lines=True, clean_store=True)

Manual cache control#

from RESource.step_cache import StepCache

cache = StepCache(
    store_path=builder.store,
    config=builder.config,
    resource_type="wind",
)

# Check whether a step would be skipped
cache.is_current("scoring", "cells")   # True → skip; False → run

# Invalidate a single step (forces it to re-run next time)
cache.invalidate("scoring")

# Invalidate all steps (equivalent to a fresh start)
cache.invalidate_all()

# Manually mark a step as done after running it outside build()
scored_cells = builder.score_cells()
cache.mark_done("scoring")

Worked Examples#

Scenario 1: Change discount rate only#

Only the scoring step sees a changed hash. Steps 1–4 are skipped; step 5 (scoring), step 6 (clustering), and step 7 (site selection) re-run.

Scenario 2: Change turbine specification#

capacity_disaggregation is monitored by cell_capacity, cf_timeseries, scoring, and clustering. All four re-run. Steps 1–2 (grid_cells, grid_nodes) are skipped because grid_cell_resolution, GADM, and OSM_data are unchanged.

Scenario 3: Change weather year#

weather_year appears in weather_data and cf_timeseries. Both re-run, along with all downstream steps (scoring, clustering). Grid cells and capacity estimates are unaffected and are skipped.

Scenario 4: Re-run with identical config#

All hashes match and all HDF5 keys exist. Every step is skipped. The run completes in seconds — only the export and metadata steps execute.

Adding a New Step#

To guard a new pipeline step, add its name and governing config keys to STEP_CONFIG_KEYS in step_cache.py:

STEP_CONFIG_KEYS: dict[str, list[str]] = {
    ...
    "my_new_step": [
        "my_config_section",
        "region_mapping",
    ],
}

Then wrap the step in build():

if cache.is_current("my_new_step", "my_hdf5_key"):
    utils.print_update(level=2, message="my_new_step config unchanged — skipping.")
else:
    self.my_new_step_method()
    cache.mark_done("my_new_step")

No other changes are required.

Limitations#

No automatic downstream propagation. If you change a grid parameter, the cache does not know that scoring depends on grid geometry. Either use clean_store=True for full re-runs or manually call cache.invalidate() on each affected step.
Config hash only. The cache detects changes to config values but not to changes in raw input data files (e.g., a new GWA raster or a corrected GADM boundary). If input files change without a config change, call cache.invalidate("gwa_scaling") or the relevant step before the run.
Single-process only. The JSON sidecar is not safe for concurrent writes. Do not run two build() calls for the same region and resource type simultaneously.

RESource Step Cache

Contents