English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Self-supervised learning is an increasingly popular approach to unsupervisedlearning, achieving state-of-the-art results. A prevalent approach consists incontrasting data points and noise points within a classification task: thisrequires a good noise distribution which is notoriously hard to specify. Whilea comprehensive theory is missing, it is widely assumed that the optimal noisedistribution should in practice be made equal to the data distribution, as inGenerative Adversarial Networks (GANs). We here empirically and theoreticallychallenge this assumption. We turn to Noise-Contrastive Estimation (NCE) whichgrounds this self-supervised task as an estimation problem of an energy-basedmodel of the data. This ties the optimality of the noise distribution to thesample efficiency of the estimator, which is rigorously defined as itsasymptotic variance, or mean-squared error. In the special case where thenormalization constant only is unknown, we show that NCE recovers a family ofImportance Sampling estimators for which the optimal noise is indeed equal tothe data distribution. However, in the general case where the energy is alsounknown, we prove that the optimal noise density is the data density multipliedby a correction term based on the Fisher score. In particular, the optimalnoise distribution is different from the data distribution, and is even from adifferent family. Nevertheless, we soberly conclude that the optimal noise maybe hard to sample from, and the gain in efficiency can be modest compared tochoosing the noise distribution equal to the data's.

Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation