English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Chain-of-thought (CoT) is a method that enables language models to handlecomplex reasoning tasks by decomposing them into simpler steps. Despite itssuccess, the underlying mechanics of CoT are not yet fully understood. In anattempt to shed light on this, our study investigates the impact of CoT on theability of transformers to in-context learn a simple to study, yet generalfamily of compositional functions: multi-layer perceptrons (MLPs). In thissetting, we find that the success of CoT can be attributed to breaking downin-context learning of a compositional function into two distinct phases:focusing on and filtering data related to each step of the composition andin-context learning the single-step composition function. Through bothexperimental and theoretical evidence, we demonstrate how CoT significantlyreduces the sample complexity of in-context learning (ICL) and facilitates thelearning of complex functions that non-CoT methods struggle with. Furthermore,we illustrate how transformers can transition from vanilla in-context learningto mastering a compositional function with CoT by simply incorporatingadditional layers that perform the necessary data-filtering for CoT via theattention mechanism. In addition to these test-time benefits, we show CoT helpsaccelerate pretraining by learning shortcuts to represent complex functions andfiltering plays an important role in this process. These findings collectivelyprovide insights into the mechanics of CoT, inviting further investigation ofits role in complex reasoning tasks.

Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning