English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Self-predictive unsupervised learning methods such as BYOL or SimSiam haveshown impressive results, and counter-intuitively, do not collapse to trivialrepresentations. In this work, we aim at exploring the simplest possiblemathematical arguments towards explaining the underlying mechanisms behindself-predictive unsupervised learning. We start with the observation that thosemethods crucially rely on the presence of a predictor network (andstop-gradient). With simple linear algebra, we show that when using a linearpredictor, the optimal predictor is close to an orthogonal projection, andpropose a general framework based on orthonormalization that enables tointerpret and give intuition on why BYOL works. In addition, this frameworkdemonstrates the crucial role of the exponential moving average andstop-gradient operator in BYOL as an efficient orthonormalization mechanism. Weuse these insights to propose four new \emph{closed-form predictor} variants ofBYOL to support our analysis. Our closed-form predictors outperform standardlinear trainable predictor BYOL at $100$ and $300$ epochs (top-$1$ linearaccuracy on ImageNet).

The Edge of Orthogonality: A Simple View of What Makes BYOL Tick