English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Deep learning models are known to overfit and memorize spurious features inthe training dataset. While numerous empirical studies have aimed atunderstanding this phenomenon, a rigorous theoretical framework to quantify itis still missing. In this paper, we consider spurious features that areuncorrelated with the learning task, and we provide a precise characterizationof how they are memorized via two separate terms: (i) the stability of themodel with respect to individual training samples, and (ii) the featurealignment between the spurious feature and the full sample. While the firstterm is well established in learning theory and it is connected to thegeneralization error in classical work, the second one is, to the best of ourknowledge, novel. Our key technical result gives a precise characterization ofthe feature alignment for the two prototypical settings of random features (RF)and neural tangent kernel (NTK) regression. We prove that the memorization ofspurious features weakens as the generalization capability increases and,through the analysis of the feature alignment, we unveil the role of the modeland of its activation function. Numerical experiments show the predictive powerof our theory on standard datasets (MNIST, CIFAR-10).

How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features