Watching the Watchmen: Assessment-Biases in Waiting List Prioritization for the Delivery of Mental Health Services

Purpose: While the demand for mental health services increases, supply often stagnates. Providing treatment to those most in need is an important factor in its efficient distribution. We propose and conduct a statistical procedure for detecting rater-biases in patient prioritization tools. Design / Method / Approach: We gather real-life data from 266 illness severity assessments in an Austrian publicly funded mental health service provider, including a rich set of covariates. To ensure robustness, we merge this data with determinants of mental health and assessment identified by previous research, such as weather or seasonal indicators. Findings: We find statistically significant effects of rater-biases. These effects are robust to a large array of controls. Practical Implications: A back-of-the-envelope calculation reveals that the identified rater effects can translate to large changes in the waiting times for patients. Misspecified treatment allocations may lead to worsened symptoms and potentially fatal outcomes. Originality / Value: Although a growing literature focuses on patient prioritization tools, many articles study these in synthetic contexts using “vignettes”. In comparison, our study adds external validity by considering real-life treatments in the field. Research Limitations / Future Research: This study can be used as a starting point for deeper, causally focused studies. Disclaimer: In accordance with publisher policies and our ethical obligations as researchers, we report that one of the authors is employed at a company that may be affected by the research reported in the enclosed paper. We have disclosed those interests fully. Paper type: Empirical