English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers have shown great potential in various computer vision tasksowing to their strong capability in modeling long-range dependency using theself-attention mechanism. Nevertheless, vision transformers treat an image as1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) inmodeling local visual structures and dealing with scale variance.Alternatively, they require large-scale training data and longer trainingschedules to learn the IB implicitly. In this paper, we propose a novel VisionTransformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE.Technically, ViTAE has several spatial pyramid reduction modules to downsampleand embed the input image into tokens with rich multi-scale context by usingmultiple convolutions with different dilation rates. In this way, it acquiresan intrinsic scale invariance IB and is able to learn robust featurerepresentation for objects at various scales. Moreover, in each transformerlayer, ViTAE has a convolution block in parallel to the multi-headself-attention module, whose features are fused and fed into the feed-forwardnetwork. Consequently, it has the intrinsic locality IB and is able to learnlocal features and global dependencies collaboratively. Experiments on ImageNetas well as downstream tasks prove the superiority of ViTAE over the baselinetransformer and concurrent works. Source code and pretrained models will beavailable at GitHub.

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias