English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer models, notably large language models (LLMs), have the remarkableability to perform in-context learning (ICL) -- to perform new tasks whenprompted with unseen input-output examples without any explicit model training.In this work, we study how effectively transformers can bridge between theirpretraining data mixture, comprised of multiple distinct task families, toidentify and learn new tasks in-context which are both inside and outside thepretraining distribution. Building on previous work, we investigate thisquestion in a controlled setting, where we study transformer models trained onsequences of $(x, f(x))$ pairs rather than natural language. Our empiricalresults show transformers demonstrate near-optimal unsupervised model selectioncapabilities, in their ability to first in-context identify different taskfamilies and in-context learn within them when the task families arewell-represented in their pretraining data. However when presented with tasksor functions which are out-of-domain of their pretraining data, we demonstratevarious failure modes of transformers and degradation of their generalizationfor even simple extrapolation tasks. Together our results highlight that theimpressive ICL abilities of high-capacity sequence models may be more closelytied to the coverage of their pretraining data mixtures than inductive biasesthat create fundamental generalization capabilities.

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models