English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In-context learning (ICL) is one of the most powerful and most unexpectedcapabilities to emerge in recent transformer-based large language models(LLMs). Yet the mechanisms that underlie it are poorly understood. In thispaper, we demonstrate that comparable ICL capabilities can be acquired by analternative sequence prediction learning method using clone-structured causalgraphs (CSCGs). Moreover, a key property of CSCGs is that, unliketransformer-based LLMs, they are {\em interpretable}, which considerablysimplifies the task of explaining how ICL works. Specifically, we show that ituses a combination of (a) learning template (schema) circuits for patterncompletion, (b) retrieving relevant templates in a context-sensitive manner,and (c) rebinding of novel tokens to appropriate slots in the templates. We goon to marshall evidence for the hypothesis that similar mechanisms underlie ICLin LLMs. For example, we find that, with CSCGs as with LLMs, differentcapabilities emerge at different levels of overparameterization, suggestingthat overparameterization helps in learning more complex template (schema)circuits. By showing how ICL can be achieved with small models and datasets, weopen up a path to novel architectures, and take a vital step towards a moregeneral understanding of the mechanics behind this important capability.

Schema-learning and rebinding as mechanisms of in-context learning and emergence