Premium
Rates of False Flagging Due to Statistical Artifact in CMS Evaluations of Transplant Programs: Results of a Stochastic Simulation
Author(s) -
Massie A. B.,
Segev D. L.
Publication year - 2013
Publication title -
american journal of transplantation
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 2.89
H-Index - 188
eISSN - 1600-6143
pISSN - 1600-6135
DOI - 10.1111/ajt.12325
Subject(s) - flagging , medicine , spurious relationship , statistics , artifact (error) , risk assessment , covariate , demography , computer science , mathematics , artificial intelligence , computer security , geography , cartography , sociology
The recent CMS conditions of participation are based on risk‐adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor‐performing centers is unknown. In this stochastic simulation study, 1‐year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by CMS, evaluating nine overlapping 2.5‐year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false‐positive flagging was lowest in low‐volume centers (2.5%) and highest in high‐volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk (“poor‐performing centers”), only 32% of poor‐performing centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater‐than‐median risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately‐performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk.