English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision Transformers (ViT) have shown their competitive advantagesperformance-wise compared to convolutional neural networks (CNNs) though theyoften come with high computational costs. To this end, previous methods exploredifferent attention patterns by limiting a fixed number of spatially nearbytokens to accelerate the ViT's multi-head self-attention (MHSA) operations.However, such structured attention patterns limit the token-to-tokenconnections to their spatial relevance, which disregards learned semanticconnections from a full attention mask. In this work, we propose a novelapproach to learn instance-dependent attention patterns, by devising alightweight connectivity predictor module to estimate the connectivity score ofeach pair of tokens. Intuitively, two tokens have high connectivity scores ifthe features are considered relevant either spatially or semantically. As eachtoken only attends to a small number of other tokens, the binarizedconnectivity masks are often very sparse by nature and therefore provide theopportunity to accelerate the network via sparse computations. Equipped withthe learned unstructured attention pattern, sparse attention ViT (Sparsifiner)produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracyon ImageNet compared to token sparsity. Our method reduces 48% to 69% FLOPs ofMHSA while the accuracy drop is within 0.4%. We also show that combiningattention and token sparsity reduces ViT FLOPs by over 60%.

Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers