English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Generalization error predictors (GEPs) aim to predict model performance onunseen distributions by deriving dataset-level error estimates fromsample-level scores. However, GEPs often utilize disparate mechanisms (e.g.,regressors, thresholding functions, calibration datasets, etc), to derive sucherror estimates, which can obfuscate the benefits of a particular scoringfunction. Therefore, in this work, we rigorously study the effectiveness ofpopular scoring functions (confidence, local manifold smoothness, modelagreement), independent of mechanism choice. We find, absent complexmechanisms, that state-of-the-art confidence- and smoothness- based scores failto outperform simple model-agreement scores when estimating error underdistribution shifts and corruptions. Furthermore, on realistic settings wherethe training data has been compromised (e.g., label noise, measurement noise,undersampling), we find that model-agreement scores continue to perform welland that ensemble diversity is important for improving its performance.Finally, to better understand the limitations of scoring functions, wedemonstrate that simplicity bias, or the propensity of deep neural networks torely upon simple but brittle features, can adversely affect GEP performance.Overall, our work carefully studies the effectiveness of popular scoringfunctions in realistic settings and helps to better understand theirlimitations.

On the Efficacy of Generalization Error Prediction Scoring Functions