English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Despite the growing use of transformer models in computer vision, amechanistic understanding of these networks is still needed. This workintroduces a method to reverse-engineer Vision Transformers trained to solveimage classification tasks. Inspired by previous research in NLP, wedemonstrate how the inner representations at any level of the hierarchy can beprojected onto the learned class embedding space to uncover how these networksbuild categorical representations for their predictions. We use our frameworkto show how image tokens develop class-specific representations that depend onattention mechanisms and contextual information, and give insights on howself-attention and MLP layers differentially contribute to this categoricalcomposition. We additionally demonstrate that this method (1) can be used todetermine the parts of an image that would be important for detecting the classof interest, and (2) exhibits significant advantages over traditional linearprobing approaches. Taken together, our results position our proposed frameworkas a powerful tool for mechanistic interpretability and explainabilityresearch.

Analyzing Vision Transformers for Image Classification in Class Embedding Space