English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers are widely used to extract complex semantic meanings from inputtokens, yet they usually operate as black-box models. In this paper, we presenta simple yet informative decomposition of hidden states (or embeddings) oftrained transformers into interpretable components. For any layer, embeddingvectors of input sequence samples are represented by a tensor $\boldsymbol{h}\in \mathbb{R}^{C \times T \times d}$. Given embedding vector$\boldsymbol{h}_{c,t} \in \mathbb{R}^d$ at sequence position $t \le T$ in asequence (or context) $c \le C$, extracting the mean effects yields thedecomposition \[ \boldsymbol{h}_{c,t} = \boldsymbol{\mu} + \mathbf{pos}_t +\mathbf{ctx}_c + \mathbf{resid}_{c,t} \] where $\boldsymbol{\mu}$ is the globalmean vector, $\mathbf{pos}_t$ and $\mathbf{ctx}_c$ are the mean vectors acrosscontexts and across positions respectively, and $\mathbf{resid}_{c,t}$ is theresidual vector. For popular transformer architectures and diverse textdatasets, empirically we find pervasive mathematical structure: (1)$(\mathbf{pos}_t)_{t}$ forms a low-dimensional, continuous, and often spiralshape across layers, (2) $(\mathbf{ctx}_c)_c$ shows clear cluster structurethat falls into context topics, and (3) $(\mathbf{pos}_t)_{t}$ and$(\mathbf{ctx}_c)_c$ are mutually incoherent -- namely $\mathbf{pos}_t$ isalmost orthogonal to $\mathbf{ctx}_c$ -- which is canonical in compressedsensing and dictionary learning. This decomposition offers structural insightsabout input formats in in-context learning (especially for induction heads) andin arithmetic tasks.

Uncovering hidden geometry in Transformers via disentangling position and context