English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision Transformer (ViT) has recently demonstrated promise in computer visionproblems. However, unlike Convolutional Neural Networks (CNN), it is known thatthe performance of ViT saturates quickly with depth increasing, due to theobserved attention collapse or patch uniformity. Despite a couple of empiricalsolutions, a rigorous framework studying on this scalability issue remainselusive. In this paper, we first establish a rigorous theory framework toanalyze ViT features from the Fourier spectrum domain. We show that theself-attention mechanism inherently amounts to a low-pass filter, whichindicates when ViT scales up its depth, excessive low-pass filtering will causefeature maps to only preserve their Direct-Current (DC) component. We thenpropose two straightforward yet effective techniques to mitigate theundesirable low-pass limitation. The first technique, termed AttnScale,decomposes a self-attention block into low-pass and high-pass components, thenrescales and combines these two filters to produce an all-pass self-attentionmatrix. The second technique, termed FeatScale, re-weights feature maps onseparate frequency bands to amplify the high-frequency signals. Both techniquesare efficient and hyperparameter-free, while effectively overcoming relevantViT training artifacts such as attention collapse and patch uniformity. Byseamlessly plugging in our techniques to multiple ViT variants, we demonstratethat they consistently help ViTs benefit from deeper architectures, bringing upto 1.1% performance gains "for free" (e.g., with little parameter overhead). Wepublicly release our codes and pre-trained models athttps://github.com/VITA-Group/ViT-Anti-Oversmoothing.

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice