English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recent studies have discovered that Chain-of-Thought prompting (CoT) candramatically improve the performance of Large Language Models (LLMs),particularly when dealing with complex tasks involving mathematics orreasoning. Despite the enormous empirical success, the underlying mechanismsbehind CoT and how it unlocks the potential of LLMs remain elusive. In thispaper, we take a first step towards theoretically answering these questions.Specifically, we examine the expressivity of LLMs with CoT in solvingfundamental mathematical and decision-making problems. By using circuitcomplexity theory, we first give impossibility results showing thatbounded-depth Transformers are unable to directly produce correct answers forbasic arithmetic/equation tasks unless the model size grows super-polynomiallywith respect to the input length. In contrast, we then prove by constructionthat autoregressive Transformers of constant size suffice to solve both tasksby generating CoT derivations using a commonly used math language format.Moreover, we show LLMs with CoT can handle a general class of decision-makingproblems known as Dynamic Programming, thus justifying its power in tacklingcomplex real-world tasks. Finally, an extensive set of experiments show that,while Transformers always fail to directly predict the answers, they canconsistently learn to generate correct solutions step-by-step given sufficientCoT demonstrations.

Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective