English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As the use of machine learning in high impact domains becomes widespread, theimportance of evaluating safety has increased. An important aspect of this isevaluating how robust a model is to changes in setting or population, whichtypically requires applying the model to multiple, independent datasets. Sincethe cost of collecting such datasets is often prohibitive, in this paper, wepropose a framework for analyzing this type of stability using the availabledata. We use the original evaluation data to determine distributions underwhich the algorithm performs poorly, and estimate the algorithm's performanceon the "worst-case" distribution. We consider shifts in user definedconditional distributions, allowing some distributions to shift while keepingother portions of the data distribution fixed. For example, in a healthcarecontext, this allows us to consider shifts in clinical practice while keepingthe patient population fixed. To address the challenges associated withestimation in complex, high-dimensional distributions, we derive a "debiased"estimator which maintains $\sqrt{N}$-consistency even when machine learningmethods with slower convergence rates are used to estimate the nuisanceparameters. In experiments on a real medical risk prediction task, we show thisestimator can be used to analyze stability and accounts for realistic shiftsthat could not previously be expressed. The proposed framework allowspractitioners to proactively evaluate the safety of their models withoutrequiring additional data collection.

Evaluating Model Robustness and Stability to Dataset Shift