English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Several recent works demonstrate that transformers can implement algorithmslike gradient descent. By a careful construction of weights, these works showthat multiple layers of transformers are expressive enough to simulateiterations of gradient descent. Going beyond the question of expressivity, weask: Can transformers learn to implement such algorithms by training overrandom problem instances? To our knowledge, we make the first theoreticalprogress on this question via an analysis of the loss landscape for lineartransformers trained over random instances of linear regression. For a singleattention layer, we prove the global minimum of the training objectiveimplements a single iteration of preconditioned gradient descent. Notably, thepreconditioning matrix not only adapts to the input distribution but also tothe variance induced by data inadequacy. For a transformer with $L$ attentionlayers, we prove certain critical points of the training objective implement$L$ iterations of preconditioned gradient descent. Our results call for futuretheoretical studies on learning algorithms by training transformers.

Transformers learn to implement preconditioned gradient descent for in-context learning