English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, transformers have shown great potential in image classification andestablished state-of-the-art results on the ImageNet benchmark. However,compared to CNNs, transformers converge slowly and are prone to overfitting inlow-data regimes due to the lack of spatial inductive biases. Such spatialinductive biases can be especially beneficial since the 2D structure of aninput image is not well preserved in transformers. In this work, we presentSpatial Prior-enhanced Self-Attention (SP-SA), a novel variant of vanillaSelf-Attention (SA) tailored for vision transformers. Spatial Priors (SPs) areour proposed family of inductive biases that highlight certain groups ofspatial relations. Unlike convolutional inductive biases, which are forced tofocus exclusively on hard-coded local regions, our proposed SPs are learned bythe model itself and take a variety of spatial relations into account.Specifically, the attention score is calculated with emphasis on certain kindsof spatial relations at each head, and such learned spatial foci can becomplementary to each other. Based on SP-SA we propose the SP-ViT family, whichconsistently outperforms other ViT models with similar GFlops or parameters.Our largest model SP-ViT-L achieves a record-breaking 86.3% Top-1 accuracy witha reduction in the number of parameters by almost 50% compared to previousstate-of-the-art model (150M for SP-ViT-L vs 271M for CaiT-M-36) among allImageNet-1K models trained on 224x224 and fine-tuned on 384x384 resolution w/oextra data.

SP-ViT: Learning 2D Spatial Priors for Vision Transformers