Model card — Forecasting and forecast evaluation¶

Family: backtest, theta_forecast, accuracy, dm_test, cw_test, gw_test

Producing a forecast is easy; knowing whether it is any good is the hard part. This family covers a strong, hard-to-beat benchmark (theta_forecast), an honest pseudo-out-of-sample evaluation engine (backtest), the standard accuracy measures (accuracy), and the three formal tests for comparing two forecasters (dm_test, cw_test, gw_test). The discipline they enforce: never report an accuracy number from a single split, and never claim one model beats another without a test that accounts for dependent forecast errors.

Function	Role
`theta_forecast`	The Theta method — a benchmark that wins forecasting competitions
`backtest`	Rolling / expanding pseudo-out-of-sample evaluation
`accuracy`	ME / RMSE / MAE / MAPE / sMAPE / MASE / RMSSE
`dm_test`	Diebold-Mariano test of equal predictive accuracy (any two forecasters)
`cw_test`	Clark-West test for nested models
`gw_test`	Giacomini-White unconditional test of equal predictive ability

What it estimates¶

theta_forecast(y, steps) — the Theta method (Assimakopoulos-Nikolopoulos 2000): deseasonalize, decompose into two "theta lines," extrapolate, and recombine. It is equivalent to simple exponential smoothing with drift and is a notoriously strong benchmark.
backtest(y, ...) — walks an origin forward through the series, re-fits a chosen forecaster on each training window, forecasts horizon steps ahead, and tabulates accuracy by horizon. This is the correct way to estimate out-of-sample error; a single train/test split is not.
accuracy(actual, forecast) — the standard scale-dependent (RMSE, MAE), percentage (MAPE, sMAPE), and scaled (MASE, RMSSE) error measures. The scaled measures divide by a naïve in-sample benchmark, so a value below 1 means you beat that benchmark.
dm_test / cw_test / gw_test — turn "model A looks better" into a hypothesis test on the loss differential, with variance estimates that account for the autocorrelation multi-step forecast errors always carry.

Assumptions¶

Diebold-Mariano tests equal unconditional expected loss. Its variance is a HAC estimate to lag h−1 (multi-step errors are MA(h−1)-correlated); the reported hln_stat further applies the Harvey-Leybourne-Newbold (1997) small-sample correction and a t-reference. DM is designed for non-nested models — applied to nested ones it is undersized and loses power.
Clark-West is the nested-model fix: under the null the larger model equals the smaller, but its extra parameters add estimation noise that inflates its MSE; CW adjusts the loss differential for exactly that noise. Use cw_test when one model is a special case of the other.
Giacomini-White (unconditional here) tests equal finite-sample predictive ability of forecasting methods, and is agnostic about nesting; it operates on two loss series you supply.
MASE/RMSSE require an in-sample series to scale against; without insample they cannot be computed. The scaling benchmark is the seasonal naïve at frequency period.
All of these assume the forecast errors were generated by a genuine out-of-sample procedure. Feeding in in-sample residuals invalidates every number here.

When to use¶

theta_forecast as the benchmark every other model must beat, and as a fast production forecaster for seasonal series.
backtest whenever you report an accuracy figure. Expanding window for a stable process, rolling window when you suspect the data-generating process drifts.
accuracy to summarize a set of forecasts; prefer MASE/RMSSE for cross-series comparison (they are unit-free and benchmark-relative) over MAPE (undefined near zero, asymmetric).
dm_test to compare two distinct models (e.g. ARIMA vs Theta).
cw_test to compare a restricted model against the model that nests it (e.g. "add this predictor or not").
gw_test to compare two forecasting procedures including estimation uncertainty, or when you simply have two loss series.

Key arguments and defaults¶

Call	Argument	Default	Notes
`theta_forecast`	`steps`	— (required)	forecast horizon
	`period`	`1`	seasonal period; set to 12/4 for monthly/quarterly
`backtest`	`window`	`"expanding"`	or `"rolling"`
	`train`	`20`	initial (or fixed, if rolling) training length
	`horizon`	`1`	steps forecast at each origin
	`refit_every`	`1`	re-fit cadence (speed vs freshness)
	`forecaster`	`"naive"`	`naive`, `drift`, `mean`, `seasonal_naive`, `theta`
	`period`	`1`	seasonal period for `seasonal_naive` / `theta`
`accuracy`	`insample`	`None`	required for MASE / RMSSE
	`period`	`1`	seasonal frequency of the scaling benchmark
`dm_test`	`h`	`1`	forecast horizon; sets the HAC lag to h−1
	`loss`	`"squared"`	or `"absolute"`
`cw_test`	`lrv_lags`	`0`	long-run-variance lags (0 = no HAC correction)
`gw_test`	`lrv_lags`	`0`	long-run-variance lags

How to read the output¶

theta_forecast → a bare array of length steps.
backtest → {"origins", "n_origins", "horizon", "forecasts", "targets", "accuracy"}. forecasts and targets are horizon × n_origins; accuracy is a list of dicts, one per horizon, each with name ("h=1", …) and the full measure set (rmse, mae, mase, rmsse, …). Read bt["accuracy"][0]["rmse"] for one-step RMSE.
accuracy → {"me", "rmse", "mae", "mape", "smape", "mase", "rmsse"}. me is mean error (bias); MASE/RMSSE below 1 beat the naïve benchmark.
dm_test → {"dm_stat", "hln_stat", "p_value", "mean_loss_diff"}. Report hln_stat and p_value. The sign of mean_loss_diff (loss₁ − loss₂) says which model won: negative ⇒ the first forecaster had lower loss.
cw_test → {"cw_stat", "p_value", "mean_adj_diff"}. A large positive cw_stat favors the larger model; the test is one-sided.
gw_test → {"gw_stat", "p_value", "df"}. gw_stat is χ²(df); small p_value ⇒ the two methods differ.

Failure modes¶

Single-split accuracy. One train/test cut gives a noisy, often optimistic error estimate. Use backtest and read accuracy by horizon.
DM on nested models. The classic error: comparing "model" vs "model + extra regressor" with dm_test. It is undersized there — use cw_test.
MAPE on near-zero data. MAPE explodes and becomes meaningless when actuals approach zero, and it penalizes over- and under-prediction asymmetrically. Prefer MASE / RMSSE.
Forgetting insample. accuracy cannot return MASE/RMSSE without it.
Wrong h. dm_test sets its HAC lag to h−1; passing h=1 for a 12-step forecast under-corrects the variance and overstates significance.
In-sample errors. Every test here assumes genuine out-of-sample errors; in-sample residuals make everything look significant.

Validated against¶

theta_forecast — statsmodels ThetaModel (deseasonalize=True), matched numerically.
dm_test — the Harvey-Leybourne-Newbold (1997) small-sample-corrected statistic, computed from the documented formula and pinned as a golden.
cw_test, gw_test — documented-formula goldens from Clark-West (2007) and Giacomini-White (2006), cross-checked against an independent NumPy reference.
accuracy — the documented error-measure definitions.

Fixtures: fixtures/forecast.json and fixtures/forecast_eval2.json.

References¶

Assimakopoulos, V. & Nikolopoulos, K. (2000). "The theta model." Int. J. Forecasting 16.
Diebold, F. & Mariano, R. (1995). "Comparing Predictive Accuracy." JBES 13.
Harvey, D., Leybourne, S. & Newbold, P. (1997). "Testing the equality of prediction mean squared errors." Int. J. Forecasting 13.
Clark, T. & West, K. (2007). "Approximately normal tests for equal predictive accuracy in nested models." J. Econometrics 138.
Giacomini, R. & White, H. (2006). "Tests of Conditional Predictive Ability." Econometrica 74.
Hyndman, R. & Koehler, A. (2006). "Another look at measures of forecast accuracy." Int. J. Forecasting 22.

See the guide: Forecasting: Practice and Evaluation.

Runnable example¶

import numpy as np
import tsecon

rng = np.random.default_rng(1)
# monthly series: random-walk trend + a 12-period seasonal
y = 100 + np.cumsum(rng.standard_normal(120)) + 5 * np.sin(np.arange(120) * 2 * np.pi / 12)

# 1. A point forecast from the Theta method (a hard-to-beat benchmark).
fc = tsecon.theta_forecast(y[:-12], steps=12, period=12)

# 2. Score it against the held-out tail. MASE/RMSSE are scaled against an
#    in-sample seasonal-naive benchmark: < 1 beats it, > 1 loses to it.
acc = tsecon.accuracy(y[-12:], fc, insample=y[:-12], period=12)
print("RMSE:", round(acc["rmse"], 3), " MASE:", round(acc["mase"], 3))

# 3. An honest pseudo-out-of-sample backtest instead of a single split.
bt = tsecon.backtest(y, window="expanding", train=60, horizon=6,
                     forecaster="theta", period=12)
print("origins:", bt["n_origins"], " h=1 RMSE:", round(bt["accuracy"][0]["rmse"], 3))

# 4. Diebold-Mariano: do two forecasters' errors differ significantly?
e_theta = y[-12:] - fc
e_naive = y[-12:] - y[-13]                       # last training value carried forward
dm = tsecon.dm_test(e_theta, e_naive, h=1, loss="squared")
print("DM (HLN) stat:", round(dm["hln_stat"], 3), " p:", round(dm["p_value"], 3))

# 5. Giacomini-White unconditional test on any two loss series.
gw = tsecon.gw_test(e_theta ** 2, e_naive ** 2)
print("GW stat:", round(gw["gw_stat"], 3), " p:", round(gw["p_value"], 3))

# 6. Clark-West for NESTED models. Build a genuine nested pair by expanding-window
#    pseudo-OOS: small = random walk (last value), large = drift (RW + mean step).
h = 1
origin0 = 60
tgt, ys, yl = [], [], []
for t in range(origin0, len(y) - h):
    train = y[: t + 1]
    f_small = train[-1]                          # random walk
    f_large = train[-1] + np.mean(np.diff(train))  # random walk + drift (nests RW)
    tgt.append(y[t + h]); ys.append(f_small); yl.append(f_large)
tgt, ys, yl = map(np.asarray, (tgt, ys, yl))
cw = tsecon.cw_test(tgt - ys, tgt - yl, ys, yl)
print("CW stat:", round(cw["cw_stat"], 3), " p:", round(cw["p_value"], 3))

Expected output:

RMSE: 2.131  MASE: 1.268
origins: 55  h=1 RMSE: 0.936
DM (HLN) stat: -2.932  p: 0.014
GW stat: 9.377  p: 0.002
CW stat: -0.726  p: 0.766