Premium
Using machine learning to evaluate treatment effects in multiple‐group interrupted time series analysis
Author(s) -
Linden Ariel,
Yarnold Paul R.
Publication year - 2018
Publication title -
journal of evaluation in clinical practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.737
H-Index - 73
eISSN - 1365-2753
pISSN - 1356-1294
DOI - 10.1111/jep.12966
Subject(s) - interrupted time series analysis , generalizability theory , linear discriminant analysis , interrupted time series , medicine , intervention (counseling) , statistics , psychology , computer science , mathematics , psychological intervention , nursing
Rationale, aims, and objectives Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to “interrupt” the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple‐group ITSA. Method We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)—the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)—a commonly used approach for evaluating treatment effects in ITSA studies. Results Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post‐intervention period ( P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave‐one‐out analysis to assess generalizability. Conclusions The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model‐free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance‐ and maximum‐corrected index of classification accuracy, and cross‐validation to assess generalizability.