English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained language models can be surprisingly adept at tasks they were notexplicitly trained on, but how they implement these capabilities is poorlyunderstood. In this paper, we investigate the basic mathematical abilitiesoften acquired by pre-trained language models. Concretely, we use mechanisticinterpretability techniques to explain the (limited) mathematical abilities ofGPT-2 small. As a case study, we examine its ability to take in sentences suchas "The war lasted from the year 1732 to the year 17", and predict validtwo-digit end years (years > 32). We first identify a circuit, a small subsetof GPT-2 small's computational graph that computes this task's output. Then, weexplain the role of each circuit component, showing that GPT-2 small's finalmulti-layer perceptrons boost the probability of end years greater than thestart year. Finally, we find related tasks that activate our circuit. Ourresults suggest that GPT-2 small computes greater-than using a complex butgeneral mechanism that activates across diverse contexts.

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model