English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

When deployed in the real world, machine learning models inevitably encounterchanges in the data distribution, and certain -- but not all -- distributionshifts could result in significant performance degradation. In practice, it maymake sense to ignore benign shifts, under which the performance of a deployedmodel does not degrade substantially, making interventions by a human expert(or model retraining) unnecessary. While several works have developed tests fordistribution shifts, these typically either use non-sequential methods, ordetect arbitrary shifts (benign or harmful), or both. We argue that a sensiblemethod for firing off a warning has to both (a) detect harmful shifts whileignoring benign ones, and (b) allow continuous monitoring of model performancewithout increasing the false alarm rate. In this work, we design simplesequential tools for testing if the difference between source (training) andtarget (test) distributions leads to a significant increase in a risk functionof interest, like accuracy or calibration. Recent advances in constructingtime-uniform confidence sequences allow efficient aggregation of statisticalevidence accumulated during the tracking process. The designed framework isapplicable in settings where (some) true labels are revealed after theprediction is performed, or when batches of labels become available in adelayed fashion. We demonstrate the efficacy of the proposed framework throughan extensive empirical study on a collection of simulated and real datasets.

Tracking the risk of a deployed model and detecting harmful distribution shifts