English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision transformers (ViTs) have gained increasing popularity as they arecommonly believed to own higher modeling capacity and representationflexibility, than traditional convolutional networks. However, it isquestionable whether such potential has been fully unleashed in practice, asthe learned ViTs often suffer from over-smoothening, yielding likely redundantmodels. Recent works made preliminary attempts to identify and alleviate suchredundancy, e.g., via regularizing embedding similarity or re-injectingconvolution-like structures. However, a "head-to-toe assessment" regarding theextent of redundancy in ViTs, and how much we could gain by thoroughlymitigating such, has been absent for this field. This paper, for the firsttime, systematically studies the ubiquitous existence of redundancy at allthree levels: patch embedding, attention map, and weight space. In view ofthem, we advocate a principle of diversity for training ViTs, by presentingcorresponding regularizers that encourage the representation diversity andcoverage at each of those levels, that enabling capturing more discriminativeinformation. Extensive experiments on ImageNet with a number of ViT backbonesvalidate the effectiveness of our proposals, largely eliminating the observedViT redundancy and significantly boosting the model generalization. Forexample, our diversified DeiT obtains 0.70%~1.76% accuracy boosts on ImageNetwith highly reduced similarity. Our codes are fully available inhttps://github.com/VITA-Group/Diverse-ViT.

The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy