English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural collapse describes the geometry of activation in the final layer of adeep neural network when it is trained beyond performance plateaus. Openquestions include whether neural collapse leads to better generalization and,if so, why and how training beyond the plateau helps. We model neural collapseas an information bottleneck (IB) problem in order to investigate whether sucha compact representation exists and discover its connection to generalization.We demonstrate that neural collapse leads to good generalization specificallywhen it approaches an optimal IB solution of the classification problem. Recentresearch has shown that two deep neural networks independently trained with thesame contrastive loss objective are linearly identifiable, meaning that theresulting representations are equivalent up to a matrix transformation. Weleverage linear identifiability to approximate an analytical solution of the IBproblem. This approximation demonstrates that when class means exhibit$K$-simplex Equiangular Tight Frame (ETF) behavior (e.g., $K$=10 for CIFAR10and $K$=100 for CIFAR100), they coincide with the critical phase transitions ofthe corresponding IB problem. The performance plateau occurs once the optimalsolution for the IB problem includes all of these phase transitions. We alsoshow that the resulting $K$-simplex ETF can be packed into a $K$-dimensionalGaussian distribution using supervised contrastive learning with a ResNet50backbone. This geometry suggests that the $K$-simplex ETF learned by supervisedcontrastive learning approximates the optimal features for source coding.Hence, there is a direct correspondence between optimal IB solutions andgeneralization in contrastive learning.

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method