English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Non-contrastive SSL methods like BYOL and SimSiam rely on asymmetricpredictor networks to avoid representational collapse without negative samples.Yet, how predictor networks facilitate stable learning is not fully understood.While previous theoretical analyses assumed Euclidean losses, most practicalimplementations rely on cosine similarity. To gain further theoretical insightinto non-contrastive SSL, we analytically study learning dynamics inconjunction with Euclidean and cosine similarity in the eigenspace ofclosed-form linear predictor networks. We show that both avoid collapse throughimplicit variance regularization albeit through different dynamical mechanisms.Moreover, we find that the eigenvalues act as effective learning ratemultipliers and propose a family of isotropic loss functions (IsoLoss) thatequalize convergence rates across eigenmodes. Empirically, IsoLoss speeds upthe initial learning dynamics and increases robustness, thereby allowing us todispense with the EMA target network typically used with non-contrastivemethods. Our analysis sheds light on the variance regularization mechanisms ofnon-contrastive SSL and lays the theoretical grounds for crafting novel lossfunctions that shape the learning dynamics of the predictor's spectrum.

Implicit variance regularization in non-contrastive SSL