Model card — Diagnostics and the stationarity workflow¶
Family: acf, pacf, ljung_box, jarque_bera, arch_lm, adf, kpss,
check_stationarity — plus the seasonal workflow: stl, seasonal_strength,
nsdiffs (see the dedicated section
at the end of this card) and the trend-cycle workflow: hamilton_filter's
inference surface, bn_decomposition, bn_filter (see the
dedicated section)
The first hour with any series. Before you fit a model you need to know how persistent the data are, what lag structure they carry, and whether they must be differenced. After you fit one, the same battery tells you whether the residuals are the white noise the model assumed. These are cheap, standard, and the mistakes people make with them are equally standard — this card is about avoiding those.
| Function | What it answers |
|---|---|
acf / pacf |
How persistent is the series, and what AR/MA order does it suggest? |
ljung_box |
Is there any linear autocorrelation left, jointly across lags? |
jarque_bera |
Are the (residual) innovations Gaussian? |
arch_lm |
Is there conditional heteroskedasticity (volatility clustering)? |
adf / kpss |
Is there a unit root? (opposite nulls — read them together) |
check_stationarity |
The ADF + KPSS confirmatory quadrant, with a recommendation |
What it estimates¶
acf(y)— the autocorrelation function ρ(k) = Corr(yₜ, yₜ₋ₖ) for k = 0…nlags, with Bartlett standard errors for the "is this spike real?" bands.pacf(y)— the partial autocorrelations, the correlation at lag k after projecting out lags 1…k−1 (Yule-Walker or OLS).ljung_box(y)— the portmanteau statistic Q = n(n+2)Σρ̂(k)²/(n−k), which is χ²(nlags) under the white-noise null, plus the Box-Pierce variant.jarque_bera(x)— a χ²(2) test built from sample skewness and excess kurtosis; the null is normality.arch_lm(resid)— Engle's LM test: regress squared residuals on their own lags and test joint significance (null: no ARCH).adf(y)— the Augmented Dickey-Fuller t-statistic for a unit root (null: unit root), with MacKinnon response-surface p-values.kpss(y)— the KPSS statistic (null: stationary), the deliberate mirror of ADF.check_stationarity(y)— runs both, places the series in the ADF×KPSS confirmatory quadrant, and returns a plain-language recommendation.
Assumptions¶
- ACF/PACF and Ljung-Box describe linear dependence. A series can be
serially dependent through its variance (GARCH) or nonlinearly while showing
a flat ACF — a clean Ljung-Box is not a clean bill of health; pair it with
arch_lm. - Ljung-Box on model residuals should have its degrees of freedom reduced by the number of estimated ARMA parameters. This function returns the raw χ²(lag) p-values; for an ARMA(p,q) fit, compare against χ² with lag−p−q df.
- Jarque-Bera is asymptotic and over-rejects in small samples; with a few hundred observations a "significant" p-value often just means fat tails, not a broken model.
- ADF assumes the only nonstationarity is a unit root — a deterministic
trend must be modeled through
regression="ct", or ADF will confound trend with a root. KPSS assumes the alternative is a unit root. - Both unit-root tests have low power near the boundary (φ close to 1): a near-unit-root stationary series and a true random walk look alike in samples of a few hundred. This is why you run both.
When to use¶
- Always, first. ACF/PACF and
check_stationarityare the opening move on any univariate series — they tell you whether to difference and roughly what order to fit. - ACF geometric decay + PACF cutoff at lag p → an AR(p); the mirror image (PACF decay, ACF cutoff) → an MA(q). This is Box-Jenkins identification.
- Ljung-Box / Jarque-Bera / ARCH-LM after fitting — the residual battery. A surviving Ljung-Box rejection means the mean model is under-specified; a surviving ARCH-LM rejection means you need a volatility model (see the realized-vol and GARCH cards).
- Use
check_stationarityrather than ADF alone — running one test and ignoring its complement is the single most common unit-root mistake.
Key arguments and defaults¶
| Call | Argument | Default | Notes |
|---|---|---|---|
acf |
nlags |
20 |
number of lags returned (plus lag 0) |
adjusted |
False |
True uses the n−k divisor (less biased, higher variance) |
|
pacf |
nlags |
20 |
|
method |
"yw" |
Yule-Walker; "ols" for the regression estimator |
|
ljung_box |
nlags |
10 |
statistic reported for each lag 1…nlags |
arch_lm |
nlags |
4 |
number of squared-residual lags |
adf |
regression |
"c" |
"c" constant, "ct" constant+trend, "n" none |
autolag |
"aic" |
lag selection; or pass maxlag directly |
|
kpss |
regression |
"c" |
"c" level-stationary, "ct" trend-stationary |
nlags |
None |
None → automatic (Hobijn-Franses-Ooms) bandwidth |
|
check_stationarity |
alpha |
0.05 |
significance level for both underlying tests |
How to read the output¶
acf→{"acf", "bartlett_se"}, both lengthnlags+1(index 0 is the trivial ρ(0)=1). A spike outside ±1.96·bartlett_se[k]is significant at 5%.pacfreturns a bare array of the same length.ljung_box→{"lags", "lb_stat", "lb_pvalue", "bp_stat", "bp_pvalue"}, one entry per lag. Smalllb_pvalue⇒ reject white noise. Prefer the Ljung-Box (lb_*) columns; Box-Pierce is the older, less accurate variant.jarque_bera→{"statistic", "p_value", "skewness", "kurtosis", "n"}. Notekurtosisis the raw (not excess) value — 3 is Gaussian.arch_lm→{"statistic", "p_value", "df", "nobs"}. Smallp_value⇒ volatility clustering.adf→{"statistic", "p_value", "used_lag", "nobs", "crit"}, wherecritis a dict of the 1/5/10% critical values. Smallp_value⇒ reject the unit root (series looks stationary).kpss→{"statistic", "p_value", "lags"}.p_valueis clipped to the tabulated[0.01, 0.10]range; smallp_value⇒ reject stationarity.check_stationarity→quadrant∈ {Stationary,UnitRoot,Conflict,Inconclusive}, arecommendation(Proceed/Difference/Detrend), a plain-languageinterpretation, and the raw test statistics/p-values.
Failure modes¶
- Reading ADF alone. A failure to reject a unit root is not evidence of
one — it may just be low power.
check_stationarityexists to force the confirmatory reading; act on thequadrant, not a single p-value. - Trend mistaken for a root. A trend-stationary series fed to
adfwith the defaultregression="c"will look like a unit root. Use"ct"when a deterministic trend is plausible, andkpss(..., regression="ct")to match. - Clean Ljung-Box, dirty variance. Linear-autocorrelation tests miss ARCH.
Always run
arch_lmon residuals before declaring them white noise. - Ljung-Box df on residuals. These functions do not subtract estimated parameters from the degrees of freedom; over-optimistic p-values result if you read them naïvely on ARMA residuals (see Assumptions).
- Jarque-Bera in large samples rejects on economically trivial fat tails — inspect skewness and kurtosis, do not stop at the p-value.
Validated against¶
statsmodels to tight tolerance: acf/pacf (acf, pacf), Ljung-Box and
Box-Pierce (acorr_ljungbox), ARCH-LM (het_arch), Jarque-Bera, and the ADF
and KPSS statistics with MacKinnon (2010) p-value response surfaces and the
Hobijn-Franses-Ooms automatic KPSS bandwidth; scipy.stats for the
distributional pieces. The golden values are pinned in
fixtures/diagnostics.json and
fixtures/unitroot.json.
References¶
- Ljung, G. & Box, G. (1978). "On a Measure of Lack of Fit in Time Series Models." Biometrika 65.
- Jarque, C. & Bera, A. (1980). "Efficient tests for normality, homoscedasticity and serial independence." Economics Letters 6.
- Engle, R. (1982). "Autoregressive Conditional Heteroscedasticity." Econometrica 50.
- Dickey, D. & Fuller, W. (1979). "Distribution of the Estimators for Autoregressive Time Series with a Unit Root." JASA 74.
- Kwiatkowski, Phillips, Schmidt & Shin (1992). "Testing the null hypothesis of stationarity against the alternative of a unit root." J. Econometrics 54.
- MacKinnon, J. (2010). "Critical Values for Cointegration Tests." Queen's Economics Department WP 1227.
See the guide: Exploring and Diagnosing a Series.
Runnable example¶
import numpy as np
import tsecon
rng = np.random.default_rng(0)
walk = np.cumsum(rng.standard_normal(300)) # a random walk (unit root)
# 1. Is it white noise? Ljung-Box portmanteau on the levels.
lb = tsecon.ljung_box(walk, nlags=10)
print("Ljung-Box p at lag 10:", round(lb["lb_pvalue"][-1], 4)) # ~0 -> not white noise
# 2. ACF and PACF shape (Box-Jenkins identification).
r = tsecon.acf(walk, nlags=10) # dict: acf, bartlett_se
p = tsecon.pacf(walk, nlags=10) # array; method "yw" (default) or "ols"
print("acf(1):", round(r["acf"][1], 3), " pacf(1):", round(p[1], 3))
# 3. The confirmatory stationarity workflow: ADF (H0: unit root) + KPSS (H0: stationary).
rep = tsecon.check_stationarity(walk)
print(rep["quadrant"], "->", rep["recommendation"]) # UnitRoot -> Difference
# 4. Re-run on the differences; they should now look stationary.
print("after differencing:", tsecon.check_stationarity(np.diff(walk))["recommendation"])
# 5. Post-fit residual checks: normality and conditional heteroskedasticity.
resid = rng.standard_normal(300)
print("Jarque-Bera p:", round(tsecon.jarque_bera(resid)["p_value"], 3))
print("ARCH-LM p:", round(tsecon.arch_lm(resid, nlags=5)["p_value"], 3))
# The individual unit-root tests are available directly with their p-values.
print("ADF p:", round(tsecon.adf(walk)["p_value"], 3),
" KPSS p:", round(tsecon.kpss(walk)["p_value"], 3))
Expected output:
Ljung-Box p at lag 10: 0.0
acf(1): 0.971 pacf(1): 0.971
UnitRoot -> Difference
after differencing: Proceed
Jarque-Bera p: 0.001
ARCH-LM p: 0.175
ADF p: 0.841 KPSS p: 0.01
STL decomposition and the seasonal workflow¶
Family: stl, mstl, seasonal_strength, nsdiffs
Season-Trend decomposition using LOESS (Cleveland, Cleveland, McRae &
Terpenning 1990) — the workhorse for exploratory seasonal adjustment outside
official statistics — plus the two advisors built on it: the
Wang-Smith-Hyndman strength-of-seasonality measures and the
Hyndman-Khandakar nsdiffs seasonal-differencing rule.
What it estimates¶
stl(y, period, ...)— the additive decompositiony = seasonal + trend + resid. The inner loop LOESS-smooths each cycle-subseries (all Januaries, all Februaries, …), low-passes the result so the seasonal averages ~0 over every cycle, and LOESS-smooths the deseasonalized series into the trend; the optional outer loop downweights outliers with bisquare robustness weights on the remainder. This is the netlib Fortranstl.fsemantics, matched tostatsmodels.tsa.seasonal.STLelementwise.seasonal_strength(y, period)— from a default STL fit:strength_seasonal = max(0, 1 − var(resid)/var(seasonal + resid))and the analogousstrength_trend(sample variances). Near 1: the component dominates; near 0: absent.nsdiffs(y, period, alpha=0.05, max_d=1)— the number of seasonal differencesD: difference at the seasonal lag whileseasonal_strength >= 0.64, capped atmax_d(theforecast::nsdiffs(test="seas")rule).alphais validated but unused — the rule is threshold-based, not a hypothesis test (forecast ignores it for this test too).
Assumptions and when to use¶
- Additive components. STL decomposes additively; for multiplicative
seasonality (amplitude growing with the level) log-transform first —
box_cox_lambdatells you whether the log is defensible. - One fixed integer period (12 monthly, 4 quarterly) per component.
Multiple seasonalities (hourly data with daily and weekly cycles) are
mstl's job — see its section below; non-integer seasonalities need STR (roadmap). - Use
stlbefore a SARIMA fit (does the seasonal look stable? how big is it relative to the noise?), to seasonally adjust for eyeballing turning points, or to feedresidto outlier screens.robust=Truewhen the series has suspected outliers — the weights returned tell you which points the fit ignored. - Use
nsdiffs+ndiffs, in that order (seasonal difference first, then the regular difference on the seasonally-differenced series) to pick SARIMA's(d, D)the way auto-arima procedures do.
Key arguments and defaults (mirror statsmodels exactly)¶
| Argument | Default | Notes |
|---|---|---|
seasonal |
7 |
seasonal LOESS window; odd, ≥ 3; larger → smoother, more nearly periodic seasonal |
trend |
None |
odd, > period; None → smallest odd ≥ 1.5·period/(1−1.5/seasonal) |
low_pass |
None |
odd, > period; None → smallest odd > period |
seasonal_deg/trend_deg/low_pass_deg |
1 |
LOESS degree, 0 or 1 |
robust |
False |
True runs bisquare outer iterations |
*_jump |
1 |
evaluate the LOESS every jump-th point, interpolate between (speedup) |
inner_iter/outer_iter |
None |
None → 2/15 if robust else 5/0 (Cleveland et al. §3.3) |
Requires n >= 2·period (R's stl() bound; statsmodels silently misbehaves
below it) and period >= 2.
How to read the output¶
stl→{"seasonal", "trend", "resid", "weights", "period", "config"}. The identityy = seasonal + trend + residholds exactly;weightsare all 1 unless the outer loop ran (0 = ignored as an outlier);configreports every resolved window/degree/jump and the iteration counts actually used.seasonal_strength→{"seasonal_strength", "trend_strength", "period"}.nsdiffs→{"d", "stop", "steps", ...}in thendiffshouse style: per-order evidence insteps, andstopsays why the sequence ended (WeakSeasonalityis the intended exit;MaxD/TooShortmeandis a floor, not a verdict;Constantmeans the seasonality was deterministic and is gone).
Failure modes¶
- Seasonal window too small.
seasonal=7lets the seasonal pattern evolve; if you believe the pattern is fixed, use a large odd window (e.g. 51+ — "periodic-ish") or the seasonal barely smooths and noise leaks into it. - Trend window ≤ period is rejected — it would let the trend absorb the seasonal. The default rule exists precisely to prevent that leakage.
- Robust weights all ≈ 1 under
robust=Truejust means no outliers — not a failure. Conversely a weight of 0 on a real event (a strike, a recession trough) means the decomposition is describing the series without that event; checkweightsbefore interpretingresid. nsdiffson seasonally adjusted data (e.g. most US macro releases) correctly returnsD=0; running it is still worthwhile as a check that the adjustment did its job.- Fewer than ~4 full cycles saturates the strength rule (audit round 6,
measured on pure white noise, period 12): with only 2 cycles (n = 24) the
STL cycle-subseries interpolates noise straight into the seasonal component
and
seasonal_strengthis 1.000 on every draw —nsdiffsflags D = 1 on 100% of white-noise series at n ≤ 28, 38% at n = 48 (four cycles), 2% at n = 72, 0% by n = 120. R'sforecast::nsdiffsbehaves identically (the rule is matched, not mis-implemented), and thestop="TooShort"marker warns in the other direction (d as a floor). With under ~4–6 cycles, treatD = 1as "not enough data to tell", not as evidence of seasonality. - A constant series raises from
seasonal_strength(the variance-ratio measure is undefined there — the ratio of the decomposition's float-noise variances is implementation noise; audit round 6 measured ≈ 0.61–0.67 on flat lines before the guard).nsdiffsandcheck_seriesalready special-case constants themselves.
Validated against¶
statsmodels.tsa.seasonal.STL 0.14.6 elementwise (seasonal, trend, resid,
robustness weights) on CO2 monthly, 100·log US real GDP quarterly, and a
seeded synthetic monthly series, across defaults / robust=True / a large
seasonal window / seasonal_deg=0 / non-unit jumps / explicit inner-outer
counts, at 1e-8 tolerance (observed agreement ~1e-12; the algorithm is a
deterministic port of the same Fortran). The strength measures and the 0.64
rule have no reference implementation in the test environment (R-only), so
they are graded honestly as documented-formula/rule transcriptions computed
from statsmodels components — see the header of
fixtures/generate_stl_fixtures.py.
Pinned in fixtures/stl.json.
References¶
- Cleveland, R. B., Cleveland, W. S., McRae, J. E. & Terpenning, I. (1990). "STL: A Seasonal-Trend Decomposition Procedure Based on Loess." Journal of Official Statistics 6, 3–73.
- Wang, X., Smith, K. & Hyndman, R. (2006). "Characteristic-based clustering for time series data." Data Mining and Knowledge Discovery 13, 335–364.
- Hyndman, R. & Khandakar, Y. (2008). "Automatic time series forecasting: the forecast package for R." JSS 27(3).
- Hyndman, R. & Athanasopoulos, G. Forecasting: Principles and Practice
(3rd ed.), §3.6 (STL), §4.3 (strength), §9.9 (
nsdiffs).
Runnable example¶
import numpy as np
import tsecon
t = np.arange(240, dtype=float)
y = 10 + 0.05*t + 3*np.sin(2*np.pi*t/12) + 0.4*np.sin(t*0.7134)
r = tsecon.stl(y, 12, robust=True) # statsmodels-exact STL
print("trend window:", r["config"]["trend"], # default rule -> 23
"inner/outer:", r["config"]["inner_iter"], r["config"]["outer_iter"])
s = tsecon.seasonal_strength(y, 12)
print("seasonal strength:", round(s["seasonal_strength"], 3))
d = tsecon.nsdiffs(y, 12) # SARIMA's D
print("nsdiffs D =", d["d"], "| stop:", d["stop"])
Expected output:
MSTL: decomposition with multiple seasonal cycles¶
Family: mstl
Multiple Seasonal-Trend decomposition using LOESS (Bandara, Hyndman &
Bergmeir 2021) — STL iterated over several seasonal periods, for series
whose seasonality has more than one layer: hourly load with a daily and a
weekly cycle (periods=[24, 168]), daily sales with weekly and annual
cycles, and so on. Matches statsmodels.tsa.seasonal.MSTL elementwise.
What it estimates¶
The additive decomposition
y = seasonal_1 + … + seasonal_K + trend + resid, one seasonal component
per period. Periods are sorted ascending and any period ≥ n/2 is dropped
(statsmodels warns; here the drop is reported in dropped_periods). Each
of iterate rounds (default 2; forced to 1 for a single period) cycles
over the periods, re-running STL at each period on the series
deseasonalized of all the other components — so each seasonal is
re-extracted with the competing cycles removed, which is what lets nested
cycles (24 inside 168) separate cleanly. trend and the robustness
weights come from the final STL pass; resid is what's left.
Assumptions and when to use¶
- Additive components, as with
stl: log-transform first for multiplicative seasonality. statsmodels'lmbda/Box-Cox option is deliberately not implemented — pre-transformyyourself (box_cox_lambdaadvises on the exponent). - Distinct integer periods, each < n/2. A single period degenerates to
plain
stlwith seasonal window 11 (bit-for-bit — tested), somstlis a safe default entry point for seasonal decomposition generally. - Use it before modeling multi-seasonal data (which cycle dominates?
is the weekly pattern stable?), to seasonally adjust at several
frequencies at once, or to feed per-cycle strengths into a seasonality
triage. For one ordinary monthly/quarterly cycle,
stlgives you finer control (itsseasonalwindow default 7 vs MSTL's 11).
Key arguments and defaults (mirror statsmodels exactly)¶
| Argument | Default | Notes |
|---|---|---|
periods |
required | sequence of observations-per-cycle, e.g. [24, 168]; a single period is [12]; sorted ascending internally |
windows |
None |
per-period seasonal LOESS window (odd, ≥ 3), paired with the same-index period; None → the paper's rule 7 + 4·k over the sorted periods: 11, 15, 19, … |
iterate |
2 |
refinement rounds over all periods; 1 is faster and usually close; forced to 1 when only one period survives |
| STL kwargs | as in stl |
trend, low_pass, degrees, robust, jumps, inner_iter/outer_iter are forwarded unchanged to every per-period STL pass |
Deliberate safe-side refusals where statsmodels crashes or degrades
silently: empty periods, duplicate periods, iterate=0, and "every
period was dropped" are teaching ValueErrors.
How to read the output¶
{"seasonal", "trend", "resid", "weights", "periods", "windows", "iterate", "dropped_periods", "seasonal_strength"}.seasonalis a dict keyed"seasonal_<period>"in ascending-period order; the components plustrendplusresidreconstructyexactly.periods/windowsare the resolved values (sorted, post-drop) — checkdropped_periodswhenever n is short relative to the longest cycle: a silently absent component changes the meaning oftrend.seasonal_strengthgives the Wang-Smith-Hyndman strength of each component against the shared remainder (same guarded formula asseasonal_strength); it isNonefor a constant input series, where the variance ratio would be float noise.weightsare the final pass's bisquare robustness weights (all 1 unlessrobust=True/outer_iter>0).
Failure modes¶
- A period ≥ n/2 vanishes by design (with
dropped_periodssaying so): fewer than two full cycles cannot be told from trend. If the long cycle is the one you care about, you need more data, not different windows. - Close periods compete. Periods like 28 and 30 have nearly identical frequencies at short n; MSTL will split their energy arbitrarily. Merge them or fix one of them by prior knowledge.
- Leakage between cycles at iterate=1. With strongly nested cycles
the first pass extracts the short cycle from a series still carrying
the long one; the second round (the default) cleans this up. If
components look contaminated, raise
iterate, not the windows. - The trend window binds across all periods. A forwarded
trendwindow must exceed the longest period (each pass validates it); the default rule re-resolves per pass, which is almost always what you want. - Duplicate periods are refused rather than silently producing two components of the same period (which statsmodels does).
Validated against¶
statsmodels.tsa.seasonal.MSTL 0.14.6 elementwise (trend, every
per-period seasonal, resid, robustness weights) on a seeded two-seasonal
hourly-like series (24/168), a seeded three-seasonal awkward-period series
(5/12/31), the degenerate single-period case, and a dropped-period case —
across default and explicit (unsorted) windows, robust, forwarded
stl_kwargs including inner_iter/outer_iter, and iterate 1–4, at
1e-8 tolerance (observed ≤ ~5e-11 on components; the algorithm drives the
same netlib STL core our stl pins). Grade: strong third-party golden
(statsmodels MSTL, elementwise). The single-period case is additionally
required to reproduce tsecon's own stl bitwise — internal
consistency, graded separately. Pinned in
fixtures/mstl.json; provenance in
fixtures/generate_mstl_fixtures.py.
References¶
- Bandara, K., Hyndman, R. J. & Bergmeir, C. (2021). "MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns." arXiv:2107.13462.
- Cleveland, R. B., Cleveland, W. S., McRae, J. E. & Terpenning, I. (1990). "STL: A Seasonal-Trend Decomposition Procedure Based on Loess." Journal of Official Statistics 6, 3–73.
- Hyndman, R. & Athanasopoulos, G. Forecasting: Principles and Practice (3rd ed.), §12.1 (complex seasonality).
Runnable example¶
import numpy as np
import tsecon
t = np.arange(24 * 7 * 6, dtype=float) # 6 weeks hourly
y = (20 + 0.01*t + 3*np.sin(2*np.pi*t/24) # daily cycle
+ 5*np.sin(2*np.pi*t/168) + 0.3*np.sin(t*0.91)) # weekly cycle + wobble
r = tsecon.mstl(y, [168, 24]) # order doesn't matter
print("periods:", r["periods"], "| windows:", r["windows"],
"| dropped:", r["dropped_periods"])
for k, v in r["seasonal_strength"].items():
print(k, "strength:", round(v, 3))
recon = sum(np.asarray(s) for s in r["seasonal"].values()) \
+ r["trend"] + r["resid"]
print("reconstructs y:", bool(np.allclose(recon, y)))
Expected output:
periods: [24, 168] | windows: [11, 15] | dropped: []
seasonal_24 strength: 0.991
seasonal_168 strength: 0.997
reconstructs y: True
Trend-cycle decomposition: the Hamilton filter's inference and the Beveridge-Nelson family¶
Family: hamilton_filter (extended: method, se), bn_decomposition,
bn_filter
Three ways to split a drifting macro series into trend and cycle, each with
a different discipline on the trend. The Hamilton (2018) regression filter
is the recommended replacement for HP filtering; the classic
Beveridge-Nelson (1981) decomposition defines the trend as the long-horizon
conditional expectation of an estimated ARIMA; the Kamber-Morley-Wong
(2018) BN filter keeps the BN definition but pins the signal-to-noise
ratio, which is what turns the classic BN's famously tiny cycle into an
intuitive output gap. When these fixtures were generated statsmodels shipped
none of these — which is why the hamilton_filter and bn_decomposition
goldens below are formula transcriptions rather than reference runs — and the
absence is pinned by a canary in fixtures/bn_filters.json. That canary has
since fired on one half: statsmodels 0.15.0 added
tsa.filters.api.hamilton_filter (the decomposition only — no standard
errors, no method="random_walk"), so the canary test now runs a live,
version-gated cross-check of our full cycle/trend decomposition against it —
measured max abs 4.2e-14 on first contact, asserted at 1e-10.
statsmodels still ships no BN decomposition in any form (re-verified live
by the same test), which shapes how the BN pair is validated below.
What they estimate¶
hamilton_filter(y, h, p, method="regression")— OLS ofy_ton[1, y_{t-h}, …, y_{t-h-p+1}];cycle= residual,trend= fitted value.method="random_walk"is the short-sample variant Hamilton recommends when the regression sample is thin:cycle = y_t − y_{t−h}(the population regression under a random-walk null; no coefficients). Frequency-aware defaults (the horizon spans two years, the lags one):
| frequency | h |
p |
|---|---|---|
| quarterly | 8 | 4 |
| monthly | 24 | 12 |
| annual | 2 | 1 |
hamilton_filter(..., se="hac")— Newey-West standard errors on the regression coefficients, through the library's single HAC engine (tsecon-hac). The residualv_tis anh-step-ahead forecast error observed at overlapping horizons, so it is serially correlated by construction — MA(h−1) under a correctly specified model — and classical OLS standard errors are simply wrong for this regression (Hamilton's own tables use Newey-West). The default bandwidth is the h-overlap rulemaxlags = h: it covers the known MA(h−1) correlation with one lag of slack, where generic plug-in rules (0.75·n^{1/3}≈ 4 at n ≈ 200) can land belowh−1and truncate autocorrelation known to exist.se="nonrobust"is provided as the comparison point.bn_decomposition(y, p, q)— classic BN fromARIMA(p, 1, q)with constant, fit by the library's exact MLE (defaultp=2, q=2, the Morley-Nelson-Zivot 2003 US-GDP spec). The trend is the long-horizon conditional expectation net of deterministic growth — algebraically a random walk with drift in the series' own innovations,Δτ_t = μ + ψ(1)·ε_t, whereψ(1) = θ(1)/φ(1)is the long-run multiplier (the cumulative impulse response — the permanent effect of a unit shock). Cycle:c_t = y_t − τ_t = −e1′F(I−F)⁻¹X_tin the ARMA companion form (Morley 2002), with conditional (zero-presample) innovations. Passingar/ma/driftdecomposes at fixed (e.g. published) coefficients instead of fitting.bn_filter(y, p, delta, demean)— Kamber-Morley-Wong: AR(p) on demeaned growth withΣφfixed atρ = 1 − 1/√δ(a Bayesian ridge on the Dickey-Fuller form with theirN(0, 0.5/j²)shrinkage prior),δselected by the paper's amplitude-to-noise criterion (first local maximum ofvar(cycle)/mean(residual²)on the gridd0=0.01, dt=0.0005) or imposed. Baselinep=12for quarterly data;cycle_seis the reference code's fixed error band (95% bandcycle ± 1.96·cycle_se).
When to use which¶
- Hamilton for a regression-based cycle with no model of the trend at
all — robust to the exact ARIMA form, loses
h+p−1observations, and produces a cycle whose interpretation ("what was not predictable two years out") differs from a band-pass or BN gap. - Classic BN when you want the trend/cycle split implied by the
series' own estimated dynamics. Expect a small, choppy cycle on US-GDP-
like series — that is the honest answer of the freely estimated model
(Stock-Watson 1988; MNZ 2003), not a bug. The
long_run_multiplieris itself the economically interesting number (>1: shocks are amplified into the trend; <1: partly transitory). - KMW
bn_filterwhen you want an output gap — large, persistent, intuitive — while keeping the BN definition of trend. The pinned δ is a judgment (that trend shocks contribute a small share of forecast-error variance); the amplitude-to-noise criterion makes it data-driven but it remains a discipline imposed, not discovered. On the fixture's simulated drifting series the KMW cycle variance is 37.6× the classic BN's — the paper's headline contrast, reproduced and asserted.
Failure modes¶
se=Noneon the Hamilton regression is not neutral — reading the plain OLSbetat-statistics off a hand-rolled covariance understates uncertainty badly ath=8(overlap correlation). Ask forse="hac".bn_decompositionrefuses unit-circle fits. An MA root numerically on the unit circle (the classic boundary pile-up, common whenqis too generous and AR/MA roots nearly cancel — the fixture's simulated series does exactly this at(2,2)) makes the innovation recursion unreliable; the error says to lowerq. Likewise a nonstationary AR (φ(1) ≈ 0after differencing usually means over-differencing).bn_filterneedsp ≥ 2andn ≥ 2p+3, and its automatic δ search errors out (rather than walking forever) if the amplitude-to-noise ratio never peaks — impose a fixeddeltathere.- The KMW cycle depends on the demeaning choice.
demean="sm"(the baseline) attributes the full-sample mean growth to trend; on samples with a structural growth slowdown the authors' later work uses dynamic demeaning (not implemented here — the 2018 baseline is).
Validated against (grades, with measured numbers)¶
bn_filter— grade: reference-run (R). Pinned against actual runs of the authors' own replication code (bnfiltering.com lineage: Ben Wong's MATLAB, R conversion by Luke Hartigan, updated by James Morley — as packaged atgithub.com/kletts/bnfilter@8af7924, sourced at fixture generation, not vendored) at the KMW-2018 baseline options (delta_select=1,ib=FALSE,d0=0.01,dt=0.0005, fixed bands), on 100·log US real GDP and a seeded simulated series, four cases spanning auto/fixed δ, sample-mean/no demeaning,p ∈ {8, 12}. Rust matches the R runs elementwise at ≤ 2.9e-15 (cycle) / ≤ 1.6e-15 (AR), the automatic δ lands on the identical grid point, andcycle_se/ amplitude-to-noise are pinned at 1e-8. The generator additionally re-implements the whole procedure in NumPy and asserts agreement with R at 1e-9 before writing, so the stored numbers are simultaneously a reference run and a two-implementation cross-check. Honest caveats: (a) the packaged code is the authors' current (2022–2025-refined) lineage run at its 2018-baseline settings, not a bit-frozen 2017 snapshot — it includes the shrinkage prior the refined code applies on all paths; (b)kletts/bnfilteris a re-packaging of the bnfiltering.com code, not the authors' own repository. US-GDP auto δ comes out 0.2295 on the macrodata sample (KMW report ≈ 0.24 on theirs).bn_decomposition— grade: documented-formula transcription with a genuine statsmodels pin on ψ(1), plus exact identities. statsmodels has no BN decomposition, so trend/cycle/innovations are pinned against an independent NumPy transcription of the Morley-2002 companion computation (three cases: MNZ ARIMA(2,1,2) coefficients on GDP, fixed ARMA(1,1) and AR(2) on the simulated series) — Rust matches at ≤ 2.4e-16. The number that defines the decomposition, ψ(1), IS third-party checkable: it equals the cumulative sum of statsmodels'arma_impulse_response, asserted at generation (< 1e-8) and re-pinned in the crate and binding tests at 1e-7. The identities are asserted on the library's own output:trend + cyclereconstructsy[1:](≤ 1 ulp),Δtrend = μ + ψ(1)·εat 1e-9, and ARIMA(0,1,1) reproduces the textbookc_t = −θε_t,ψ(1) = 1+θexactly. The fit path (library MLE vs statsmodels MLE of the same spec) lands within 1.4e-4 on ψ(1) and 6.8e-6 on the drift for the GDP ARIMA(2,1,2).hamilton_filter— grade: independent package (statsmodels), now on both legs. Since statsmodels 0.15.0 the decomposition has a third-party reference it could not have had when it shipped: the version-gated canary test pins ourcycle/trendagainsttsa.filters.api.hamilton_filterat ≤ 4.2e-14 max abs (asserted 1e-10). The inference surface still has no counterpart to compare against — statsmodels' filter returns cycle/trend only — but the filter is literally OLS, so its coefficient inference is statsmodels territory anyway:OLS(...).fit(cov_type="HAC", cov_kwds={"maxlags": …, "use_correction": …})on the identical design pinsbse/tvaluesfor nonrobust and three HAC settings (including themaxlags = h = 8default). Measured agreement ≤ 2.9e-8 (bse) / ≤ 6.8e-8 (tvalues), pinned at 1e-6 — the design is raw levels of a trending series, so the two solvers (statsmodels pinv vstsecon-hacrefined Cholesky) agree to ~1e-8 here rather than the engine's 1e-10 on its own calmer goldens. The decomposition andbetaare asserted bit-identical with and withoutse(the defaults-unchanged guarantee), andhamilton_filter(y)still reproduces the originalfixtures/filters.jsongolden.
References¶
- Hamilton, J. D. (2018). "Why You Should Never Use the Hodrick-Prescott Filter." Review of Economics and Statistics 100(5), 831–843.
- Beveridge, S. & Nelson, C. R. (1981). "A New Approach to Decomposition of Economic Time Series into Permanent and Transitory Components…" Journal of Monetary Economics 7(2), 151–174.
- Morley, J. C. (2002). "A state-space approach to calculating the Beveridge-Nelson decomposition." Economics Letters 75(1), 123–127.
- Morley, J. C., Nelson, C. R. & Zivot, E. (2003). "Why Are the Beveridge-Nelson and Unobserved-Components Decompositions of GDP So Different?" Review of Economics and Statistics 85(2), 235–243.
- Kamber, G., Morley, J. & Wong, B. (2018). "Intuitive and Reliable Estimates of the Output Gap from a Beveridge-Nelson Filter." Review of Economics and Statistics 100(3), 550–566. Replication code: bnfiltering.com (R conversion by Luke Hartigan).
- Newey, W. K. & West, K. D. (1987). "A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix." Econometrica 55(3), 703–708.
Runnable example¶
import numpy as np
import tsecon
# A synthetic quarterly log-level series, built as the ARIMA(2,1,2)+drift that
# bn_decomposition fits by default: phi = (0.6, -0.2), theta = (0.3, 0.1), so
# the true long-run multiplier is psi(1) = theta(1)/phi(1) = 1.4/0.6 = 2.33.
n = 240
e = np.random.default_rng(11).standard_normal(n)
dy = np.zeros(n)
for t in range(2, n):
dy[t] = (0.8 + 0.6 * dy[t - 1] - 0.2 * dy[t - 2]
+ e[t] + 0.3 * e[t - 1] + 0.1 * e[t - 2])
y = 700.0 + np.cumsum(dy) # quarterly log-level, 100*log units
ham = tsecon.hamilton_filter(y, se="hac") # h=8, p=4, NW maxlags=8
print("beta_1 t-stat (HAC):", round(ham["tvalues"][1], 2))
classic = tsecon.bn_decomposition(y) # ARIMA(2,1,2)+c, exact MLE
print("psi(1):", round(classic["long_run_multiplier"], 2), " (truth 2.33)",
"| cycle sd:", round(np.std(classic["cycle"]), 2))
gap = tsecon.bn_filter(y) # KMW, p=12, auto delta
print("delta:", round(gap["delta"], 3),
"| gap sd:", round(np.std(gap["cycle"]), 2),
"| band:", round(1.96 * gap["cycle_se"], 2))
Expected output:
beta_1 t-stat (HAC): 4.58
psi(1): 2.38 (truth 2.33) | cycle sd: 1.35
delta: 0.504 | gap sd: 2.24 | band: 2.68
Read the three lines together. The Hamilton regression finds a real y_{t−8}
coefficient, and the HAC bandwidth is what keeps that t-statistic honest under
the overlapping-horizon residual — the same fit reports t = 5.26 with
se="nonrobust", which is the number you should not quote. The classic BN
decomposition recovers the DGP's long-run multiplier (2.38 against a truth of
2.33), and that psi(1) > 1 is exactly why its cycle is the small one: sd 1.35
against the BN filter's 2.24 on the same series, the famous BN "tiny cycle"
reproduced on data whose ψ(1) we know. The KMW filter pins the signal-to-noise
ratio rather than inheriting it (delta = 0.504 from the automatic search) and
returns a gap with a standard error, so the ±2.68 band is the one number the
classic decomposition cannot give you.