English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision transformers (ViTs) have recently obtained success in manyapplications, but their intensive computation and heavy memory usage at bothtraining and inference time limit their generalization. Previous compressionalgorithms usually start from the pre-trained dense models and only focus onefficient inference, while time-consuming training is still unavoidable. Incontrast, this paper points out that the million-scale training data isredundant, which is the fundamental reason for the tedious training. To addressthe issue, this paper aims to introduce sparsity into data and proposes anend-to-end efficient training framework from three sparse perspectives, dubbedTri-Level E-ViT. Specifically, we leverage a hierarchical data redundancyreduction scheme, by exploring the sparsity under three levels: number oftraining examples in the dataset, number of patches (tokens) in each example,and number of connections between tokens that lie in attention weights. Withextensive experiments, we demonstrate that our proposed technique cannoticeably accelerate training for various ViT architectures while maintainingaccuracy. Remarkably, under certain ratios, we are able to improve the ViTaccuracy rather than compromising it. For example, we can achieve 15.2% speedupwith 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1)Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training