English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Attention layers -- which map a sequence of inputs to a sequence of outputs-- are core building blocks of the Transformer architecture which has achievedsignificant breakthroughs in modern artificial intelligence. This paperpresents a rigorous theoretical study on the learning and generalization of asingle multi-head attention layer, with a sequence of key vectors and aseparate query vector as input. We consider the random feature setting wherethe attention layer has a large number of heads, with randomly sampled frozenquery and key matrices, and trainable value matrices. We show that such arandom-feature attention layer can express a broad class of target functionsthat are permutation invariant to the key vectors. We further providequantitative excess risk bounds for learning these target functions from finitesamples, using random feature attention with finitely many heads.  Our results feature several implications unique to the attention structurecompared with existing random features theory for neural networks, such as (1)Advantages in the sample complexity over standard two-layer random-featurenetworks; (2) Concrete and natural classes of functions that can be learnedefficiently by a random-feature attention layer; and (3) The effect of thesampling distribution of the query-key weight matrix (the product of the queryand key matrix), where Gaussian random weights with a non-zero mean result inbetter sample complexities over the zero-mean counterpart for learning certainnatural target functions. Experiments on simulated data corroborate ourtheoretical findings and further illustrate the interplay between the samplesize and the complexity of the target function.

What can a Single Attention Layer Learn? A Study Through the Random Features Lens