English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

With the attention mechanism, transformers achieve significant empiricalsuccesses. Despite the intuitive understanding that transformers performrelational inference over long sequences to produce desirable representations,we lack a rigorous theory on how the attention mechanism achieves it. Inparticular, several intriguing questions remain open: (a) What makes adesirable representation? (b) How does the attention mechanism infer thedesirable representation within the forward pass? (c) How does a pretrainingprocedure learn to infer the desirable representation through the backwardpass?  We observe that, as is the case in BERT and ViT, input tokens are oftenexchangeable since they already include positional encodings. The notion ofexchangeability induces a latent variable model that is invariant to inputsizes, which enables our theoretical analysis.  - To answer (a) on representation, we establish the existence of a sufficientand minimal representation of input tokens. In particular, such arepresentation instantiates the posterior distribution of the latent variablegiven input tokens, which plays a central role in predicting output labels andsolving downstream tasks.  - To answer (b) on inference, we prove that attention with the desiredparameter infers the latent posterior up to an approximation error, which isdecreasing in input sizes. In detail, we quantify how attention approximatesthe conditional mean of the value given the key, which characterizes how itperforms relational inference over long sequences.  - To answer (c) on learning, we prove that both supervised andself-supervised objectives allow empirical risk minimization to learn thedesired parameter up to a generalization error, which is independent of inputsizes. Particularly, in the self-supervised setting, we identify a conditionnumber that is pivotal to solving downstream tasks.

An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models