Using forecast modelling to evaluate treatment effects in single‐group interrupted time series analysis

Rationale, aims, and objectives Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to “interrupt” the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post‐intervention trend. A treatment effect may be inferred if the actual post‐intervention observations diverge from the forecasts by some specified amount. Method The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series—linear regression (REG), Holt‐Winters (HW) non‐seasonal smoothing, and autoregressive moving average (ARIMA)—and forecasts are generated into the post‐intervention period. The actual observations are then compared with the forecasts to assess intervention effects. Results The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post‐intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect. Conclusions In a single‐group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single‐group ITSA studies.