English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers, central to the successes in modern Natural Language Processing,often falter on arithmetic tasks despite their vast capabilities --whichparadoxically include remarkable coding abilities. We observe that a crucialchallenge is their naive reliance on positional information to solve arithmeticproblems with a small number of digits, leading to poor performance on largernumbers. Herein, we delve deeper into the role of positional encoding, andpropose several ways to fix the issue, either by modifying the positionalencoding directly, or by modifying the representation of the arithmetic task toleverage standard positional encoding differently. We investigate the value ofthese modifications for three tasks: (i) classical multiplication, (ii) lengthextrapolation in addition, and (iii) addition in natural language context. For(i) we train a small model on a small dataset (100M parameters and 300ksamples) with remarkable aptitude in (direct, no scratchpad) 15 digitsmultiplication and essentially perfect up to 12 digits, while usual training inthis context would give a model failing at 4 digits multiplication. In theexperiments on addition, we use a mere 120k samples to demonstrate: for (ii)extrapolation from 10 digits to testing on 12 digits numbers while usualtraining would have no extrapolation, and for (iii) almost perfect accuracy upto 5 digits while usual training would be correct only up to 3 digits (which isessentially memorization with a training set of 120k samples).

Positional Description Matters for Transformers Arithmetic