Reproducibility

Recommended evidence bundle

Do not quote a benchmark value without preserving these files from the same run:

• metrics.csv;
• summary_overall.csv;
• summary_winners_volume.csv;
• run_metadata.json; and
• generated report/figures, if used.

run_metadata.json records the UTC creation time, command arguments, current Git commit when available, effective configuration and its SHA-256 digest, panel name and size, effective model labels, Python/platform details, and relevant package versions.

Dependency-light verification

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
ruff check src scripts app tests
pytest -q
python -m scripts.run_benchmark --classical-only --source synthetic --max-series 20 --horizon 14

The classical run is the best smoke test because it avoids neural-library nondeterminism and external model downloads.

Neural-model caveats

The configured seed is applied by the project, but exact neural results can still differ by device, low-level library version, kernel choice, and pretrained checkpoint revision. A serious comparison should pin dependencies and checkpoint revisions, repeat runs across seeds, and report dispersion.

M5 caveats

M5 files are user-supplied and ignored by Git. Record their source and checksums outside this repository if exact data provenance matters. This harness does not currently calculate official hierarchical, sales-weighted M5 leaderboard metrics.

Result publication checklist

• State whether the source is synthetic or m5.
• Include the effective model label, especially any fallback suffix.
• Link the metadata and metric artifacts.
• State series count, horizon, quantiles, seed, and commit.
• Distinguish a single run from repeated-seed evidence.
• Avoid generalizing synthetic-panel rankings to real retail data.