English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers trained on huge text corpora exhibit a remarkable set ofcapabilities, e.g., performing simple logical operations. Given the inherentcompositional nature of language, one can expect the model to learn to composethese capabilities, potentially yielding a combinatorial explosion of whatoperations it can perform on an input. Motivated by the above, we aim to assessin this paper "how capable can a transformer become?". Specifically, we trainautoregressive Transformer models on a data-generating process that involvescompositions of a set of well-defined monolithic capabilities. Through a seriesof extensive and systematic experiments on this data-generating process, weshow that: (1) autoregressive Transformers can learn compositional structuresfrom the training data and generalize to exponentially or even combinatoriallymany functions; (2) composing functions by generating intermediate outputs ismore effective at generalizing to unseen compositions, compared to generatingno intermediate outputs; (3) the training data has a significant impact on themodel's ability to compose unseen combinations of functions; and (4) theattention layers in the latter half of the model are critical tocompositionality.

How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks