English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The emergence of In-Context Learning (ICL) in LLMs remains a significantphenomenon with little understanding. To explain ICL, recent studies try toshed light on ICL by connecting it to Gradient Descent (GD). However, thequestion is, do these hold up in practice in actual pre-trained models?  We highlight the limiting assumptions in prior works that make their contextconsiderably different from the practical context in which language models aretrained. For example, the theoretical hand-constructed weights used in thesestudies have properties that don't match those of real LLMs. Furthermore, theirexperimental verification uses \emph{ICL objective} (training models explicitlyfor ICL), which differs from the emergent ICL in the wild.  We also look for evidence in real models. We observe that ICL and GD havedifferent sensitivity to the order in which they observe demonstrations.Finally, we probe and compare the ICL vs. GD hypothesis in a natural setting.We conduct comprehensive empirical analyses on language models pre-trained onnatural data (LLaMa-7B). Our comparisons of three performance metrics highlightthe inconsistent behavior of ICL and GD as a function of various factors suchas datasets, models, and the number of demonstrations. We observe that ICL andGD modify the output distribution of language models differently. These resultsindicate that the equivalence between ICL and GD remains an open hypothesis andcalls for further studies.

Do pretrained Transformers Really Learn In-context by Gradient Descent?