English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When training deep neural networks for classification tasks, an intriguingempirical phenomenon has been widely observed in the last-layer classifiers andfeatures, where (i) the class means and the last-layer classifiers all collapseto the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and(ii) cross-example within-class variability of last-layer activations collapsesto zero. This phenomenon is called Neural Collapse (NC), which seems to takeplace regardless of the choice of loss functions. In this work, we justify NCunder the mean squared error (MSE) loss, where recent empirical evidence showsthat it performs comparably or even better than the de-facto cross-entropyloss. Under a simplified unconstrained feature model, we provide the firstglobal landscape analysis for vanilla nonconvex MSE loss and show that the(only!) global minimizers are neural collapse solutions, while all othercritical points are strict saddles whose Hessian exhibit negative curvaturedirections. Furthermore, we justify the usage of rescaled MSE loss by probingthe optimization landscape around the NC solutions, showing that the landscapecan be improved by tuning the rescaling hyperparameters. Finally, ourtheoretical findings are experimentally verified on practical networkarchitectures.

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features