Metrics

Metrics are implemented in src/evaluation/metrics.py. They are computed per bottom-level series and then summarized with an unweighted mean unless the caller performs a different aggregation.

MASE

Mean Absolute Scaled Error divides forecast MAE by the in-sample seasonal-naive MAE:

MASE = mean(abs(y - y_hat)) / mean(abs(y_t - y_(t-m)))

m defaults to seven for daily data. The code protects degenerate denominators with a small numerical floor.

RMSSE

Root Mean Squared Scaled Error is the squared-error analogue of MASE. M5 also uses scaled squared error, but this repository’s reported mean RMSSE is not the official M5 WRMSSE: it omits the official sales weights and hierarchical aggregation levels.

Pinball loss

For quantile q and forecast y_hat_q:

L_q = max(q * (y - y_hat_q), (q - 1) * (y - y_hat_q))

Pinball loss is asymmetric and evaluates a forecast quantile rather than only a point estimate.

Repository WQL variant

The pipeline totals pinball loss across requested quantiles and horizon steps and normalizes it using in-sample seasonal-naive MAE times the horizon:

WQL = 2 * sum_q sum_h L_(q,h) / scale

The in-sample denominator avoids division by an empty test-window demand total for intermittent series. This is a deliberate project-specific choice and should not be conflated with every library’s WQL definition or the official M5 uncertainty metric. Compare values only when the definition, quantile grid, horizon, and data are held constant.

Coverage

Coverage is the fraction of observations at or below each predicted quantile. It is a calibration diagnostic: over many comparable observations, a calibrated q quantile should have coverage near q.