English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While there is much recent interest in studying why Transformer-based largelanguage models make predictions the way they do, the complex computationsperformed within each layer have made their behavior somewhat opaque. Tomitigate this opacity, this work presents a linear decomposition of finalhidden states from autoregressive language models based on each initial inputtoken, which is exact for virtually all contemporary Transformer architectures.This decomposition allows the definition of probability distributions thatablate the contribution of specific input tokens, which can be used to analyzetheir influence on model probabilities over a sequence of upcoming words withonly one forward pass from the model. Using the change in next-word probabilityas a measure of importance, this work first examines which context words makethe biggest contribution to language model predictions. Regression experimentssuggest that Transformer-based language models rely primarily on collocationalassociations, followed by linguistic factors such as syntactic dependencies andcoreference relationships in making next-word predictions. Additionally,analyses using these measures to predict syntactic dependencies and coreferentmention spans show that collocational association and repetitions of the sametoken largely explain the language models' predictions on these tasks.

Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions