English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How can neural networks trained by contrastive learning extract features fromthe unlabeled data? Why does contrastive learning usually need much strongerdata augmentations than supervised learning to ensure good representations?These questions involve both the optimization and statistical aspects of deeplearning, but can hardly be answered by analyzing supervised learning, wherethe target functions are the highest pursuit. Indeed, in self-supervisedlearning, it is inevitable to relate to the optimization/generalization ofneural networks to how they can encode the latent structures in the data, whichwe refer to as the feature learning process.  In this work, we formally study how contrastive learning learns the featurerepresentations for neural networks by analyzing its feature learning process.We consider the case where our data are comprised of two types of features: themore semantically aligned sparse features which we want to learn from, and theother dense features we want to avoid. Theoretically, we prove that contrastivelearning using $\mathbf{ReLU}$ networks provably learns the desired sparsefeatures if proper augmentations are adopted. We present an underlyingprinciple called $\textbf{feature decoupling}$ to explain the effects ofaugmentations, where we theoretically characterize how augmentations can reducethe correlations of dense features between positive samples while keeping thecorrelations of sparse features intact, thereby forcing the neural networks tolearn from the self-supervision of sparse features. Empirically, we verifiedthat the feature decoupling principle matches the underlying mechanism ofcontrastive learning in practice.

Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning