English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The vast majority of work in self-supervised learning, both theoretical andempirical (though mostly the latter), have largely focused on recovering goodfeatures for downstream tasks, with the definition of "good" often beingintricately tied to the downstream task itself. This lens is undoubtedly veryinteresting, but suffers from the problem that there isn't a "canonical" set ofdownstream tasks to focus on -- in practice, this problem is usually resolvedby competing on the benchmark dataset du jour.  In this paper, we present an alternative lens: one of parameteridentifiability. More precisely, we consider data coming from a parametricprobabilistic model, and train a self-supervised learning predictor with asuitably chosen parametric form. Then, we ask whether we can read off theground truth parameters of the probabilistic model from the optimal predictor.We focus on the widely used self-supervised learning method of predictingmasked tokens, which is popular for both natural languages and visual data.  While incarnations of this approach have already been successfully used forsimpler probabilistic models (e.g. learning fully-observed undirected graphicalmodels), we focus instead on latent-variable models capturing sequentialstructures -- namely Hidden Markov Models with both discrete and conditionallyGaussian observations. We show that there is a rich landscape of possibilities,out of which some prediction tasks yield identifiability, while others do not.Our results, borne of a theoretical grounding of self-supervised learning,could thus potentially beneficially inform practice. Moreover, we uncover closeconnections with uniqueness of tensor rank decompositions -- a widely used toolin studying identifiability through the lens of the method of moments.

Masked prediction tasks: a parameter identifiability view