English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When evaluating the performance of clinical machine learning models, one mustconsider the deployment population. When the population of patients withobserved labels is only a subset of the deployment population (labelselection), standard model performance estimates on the observed population maybe misleading. In this study we describe three classes of label selection andsimulate five causally distinct scenarios to assess how particular selectionmechanisms bias a suite of commonly reported binary machine learning modelperformance metrics. Simulations reveal that when selection is affected byobserved features, naive estimates of model discrimination may be misleading.When selection is affected by labels, naive estimates of calibration fail toreflect reality. We borrow traditional weighting estimators from causalinference literature and find that when selection probabilities are properlyspecified, they recover full population estimates. We then tackle thereal-world task of monitoring the performance of deployed machine learningmodels whose interactions with clinicians feed-back and affect the selectionmechanism of the labels. We train three machine learning models to flaglow-yield laboratory diagnostics, and simulate their intended consequence ofreducing wasteful laboratory utilization. We find that naive estimates of AUROCon the observed population undershoot actual performance by up to 20%. Such adisparity could be large enough to lead to the wrongful termination of asuccessful clinical decision support tool. We propose an altered deploymentprocedure, one that combines injected randomization with traditional weightedestimates, and find it recovers true model performance.

Avoiding Biased Clinical Machine Learning Model Performance Estimates in the Presence of Label Selection