English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Self-Supervised Learning (SSL) models rely on a pretext task to learnrepresentations. Because this pretext task differs from the downstream tasksused to evaluate the performance of these models, there is an inherentmisalignment or pretraining bias. A commonly used trick in SSL, shown to makedeep networks more robust to such bias, is the addition of a small projector(usually a 2 or 3 layer multi-layer perceptron) on top of a backbone networkduring training. In contrast to previous work that studied the impact of theprojector architecture, we here focus on a simpler, yet overlooked lever tocontrol the information in the backbone representation. We show that merelychanging its dimensionality -- by changing only the size of the backbone's verylast block -- is a remarkably effective technique to mitigate the pretrainingbias. It significantly improves downstream transfer performance for bothSelf-Supervised and Supervised pretrained models.

A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation