Methodology
Research question
How do classical intermittent-demand methods, a compact supervised Transformer, and a zero-shot foundation model behave across different retail-demand regimes under one evaluation interface?
The repository supplies the machinery to investigate that question. It does not assert a universal winner: any conclusion must be tied to a committed code revision, dataset, effective configuration, and generated artifacts.
Evaluation protocol
- Holdout: the last
horizonobservations of each eligible series form one test window. - Training boundary: global supervised models train on histories with that test window removed. Local and zero-shot models receive only the history at forecast time.
- Outputs: every adapter returns a point forecast and requested quantiles.
- Scoring: MASE, RMSSE, the repository’s per-series WQL variant, and coverage are computed for each series/model pair.
- Stratification: rows are grouped by demand-volume bucket and Syntetos-Boylan class.
This is a single-holdout experiment. It does not currently include rolling-origin cross-validation, uncertainty intervals over repeated runs, or statistical significance tests.
Demand regimes
The Syntetos-Boylan classification uses the average inter-demand interval (ADI) and squared coefficient of variation of non-zero demand (CV-squared):
| CV-squared < 0.49 | CV-squared >= 0.49 | |
|---|---|---|
| ADI < 1.32 | smooth | erratic |
| ADI >= 1.32 | intermittent | lumpy |
Volume buckets are configurable and should be interpreted relative to the selected dataset.
Dataset choices
data.source: synthetic is the default. The generator is seeded and covers dense, erratic, intermittent, and lumpy profiles. It is useful for software verification and controlled experiments, but its construction can favor or disadvantage particular methods. Results on it are not empirical evidence about Walmart demand.
data.source: m5 loads bottom-level M5 sales series from local Kaggle files. The repository does not redistribute M5 and does not implement all official M5 hierarchy and weighting rules.
Reproducing a run
python -m scripts.run_benchmark --classical-only --source synthetic --max-series 50
python -m scripts.generate_report --results results/metrics.csv
Keep metrics.csv, the summaries, and run_metadata.json together. Seeded execution improves repeatability, but exact numerical identity can still depend on hardware and dependency implementations, especially for neural models. See REPRODUCIBILITY.md.