English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Transformers are remarkably good at in-context learning (ICL) -- learningfrom demonstrations without parameter updates -- but how they perform ICLremains a mystery. Recent work suggests that Transformers may learn in-contextby internally running Gradient Descent, a first-order optimization method. Inthis paper, we instead demonstrate that Transformers learn to implementhigher-order optimization methods to perform ICL. Focusing on in-context linearregression, we show that Transformers learn to implement an algorithm verysimilar to Iterative Newton's Method, a higher-order optimization method,rather than Gradient Descent. Empirically, we show that predictions fromsuccessive Transformer layers closely match different iterations of Newton'sMethod linearly, with each middle layer roughly computing 3 iterations. Incontrast, exponentially more Gradient Descent steps are needed to match anadditional Transformers layer; this suggests that Transformers have ancomparable rate of convergence with high-order methods such as IterativeNewton, which are exponentially faster than Gradient Descent. We also show thatTransformers can learn in-context on ill-conditioned data, a setting whereGradient Descent struggles but Iterative Newton succeeds. Finally, we showtheoretical results which support our empirical findings and have a closecorrespondence with them: we prove that Transformers can implement $k$iterations of Newton's method with $\mathcal{O}(k)$ layers.

Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models