English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We provide the first global optimization landscape analysis of$Neural\;Collapse$ -- an intriguing empirical phenomenon that arises in thelast-layer classifiers and features of neural networks during the terminalphase of training. As recently reported by Papyan et al., this phenomenonimplies that ($i$) the class means and the last-layer classifiers all collapseto the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and($ii$) cross-example within-class variability of last-layer activationscollapses to zero. We study the problem based on a simplified$unconstrained\;feature\;model$, which isolates the topmost layers from theclassifier of the neural network. In this context, we show that the classicalcross-entropy loss with weight decay has a benign global landscape, in thesense that the only global minimizers are the Simplex ETFs while all othercritical points are strict saddles whose Hessian exhibit negative curvaturedirections. In contrast to existing landscape analysis for deep neural networkswhich is often disconnected from practice, our analysis of the simplified modelnot only does it explain what kind of features are learned in the last layer,but it also shows why they can be efficiently optimized in the simplifiedsettings, matching the empirical observations in practical deep networkarchitectures. These findings could have profound implications foroptimization, generalization, and robustness of broad interests. For example,our experiments demonstrate that one may set the feature dimension equal to thenumber of classes and fix the last-layer classifier to be a Simplex ETF fornetwork training, which reduces memory cost by over $20\%$ on ResNet18 withoutsacrificing the generalization performance.

A Geometric Analysis of Neural Collapse with Unconstrained Features