English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural Collapse (NC) is a well-known phenomenon of deep neural networks inthe terminal phase of training (TPT). It is characterized by the collapse offeatures and classifier into a symmetrical structure, known as simplexequiangular tight frame (ETF). While there have been extensive studies onoptimization characteristics showing the global optimality of neural collapse,little research has been done on the generalization behaviors during theoccurrence of NC. Particularly, the important phenomenon of generalizationimprovement during TPT has been remaining in an empirical observation andlacking rigorous theoretical explanation. In this paper, we establish theconnection between the minimization of CE and a multi-class SVM during TPT, andthen derive a multi-class margin generalization bound, which provides atheoretical explanation for why continuing training can still lead to accuracyimprovement on test set, even after the train accuracy has reached 100%.Additionally, our further theoretical results indicate that different alignmentbetween labels and features in a simplex ETF can result in varying degrees ofgeneralization improvement, despite all models reaching NC and demonstratingsimilar optimization performance on train set. We refer to this newlydiscovered property as "non-conservative generalization". In experiments, wealso provide empirical observations to verify the indications suggested by ourtheoretical results.

Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges