English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Mathematical reasoning in large language models (LMs) has garneredsignificant attention in recent work, but there is a limited understanding ofhow these models process and store information related to arithmetic taskswithin their architecture. In order to improve our understanding of this aspectof language models, we present a mechanistic interpretation ofTransformer-based LMs on arithmetic questions using a causal mediation analysisframework. By intervening on the activations of specific model components andmeasuring the resulting changes in predicted probabilities, we identify thesubset of parameters responsible for specific predictions. This providesinsights into how information related to arithmetic is processed by LMs. Ourexperimental results indicate that LMs process the input by transmitting theinformation relevant to the query from mid-sequence early layers to the finaltoken using the attention mechanism. Then, this information is processed by aset of MLP modules, which generate result-related information that isincorporated into the residual stream. To assess the specificity of theobserved activation dynamics, we compare the effects of different modelcomponents on arithmetic queries with other tasks, including number retrievalfrom prompts and factual knowledge questions.

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis