English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Supervised contrastive loss (SCL) is a competitive and often superioralternative to the cross-entropy loss for classification. While prior studieshave demonstrated that both losses yield symmetric training representationsunder balanced data, this symmetry breaks under class imbalances. This paperpresents an intriguing discovery: the introduction of a ReLU activation at thefinal layer effectively restores the symmetry in SCL-learned representations.We arrive at this finding analytically, by establishing that the globalminimizers of an unconstrained features model with SCL loss and entry-wisenon-negativity constraints form an orthogonal frame. Extensive experimentsconducted across various datasets, architectures, and imbalance scenarioscorroborate our finding. Importantly, our experiments reveal that the inclusionof the ReLU activation restores symmetry without compromising test accuracy.This constitutes the first geometry characterization of SCL under imbalances.Additionally, our analysis and experiments underscore the pivotal role of batchselection strategies in representation geometry. By proving necessary andsufficient conditions for mini-batch choices that ensure invariant symmetricrepresentations, we introduce batch-binding as an efficient strategy thatguarantees these conditions hold.

Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching