English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How do language models deal with the limited bandwidth of the residualstream? Prior work has suggested that some attention heads and MLP layers mayperform a "memory management" role. That is, clearing residual streamdirections set by earlier layers by reading in information and writing out thenegative version. In this work, we present concrete evidence for thisphenomenon in a 4-layer transformer. We identify several heads in layer 2 thatconsistently remove the output of a single layer 0 head. We then verify thatthis erasure causally depends on the original written direction. We furtherdemonstrate that direct logit attribution (DLA) suggests that writing anderasing heads directly contribute to predictions, when in fact their effectscancel out. Then we present adversarial prompts for which this effect isparticularly salient. These findings reveal that memory management can make DLAresults misleading. Accordingly, we make concrete recommendations for circuitanalysis to prevent interpretability illusions.

An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l