English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Transformers have shown promising performance in various visiontasks. To reduce the quadratic computation complexity caused by the globalself-attention, various methods constrain the range of attention within a localregion to improve its efficiency. Consequently, their receptive fields in asingle attention layer are not large enough, resulting in insufficient contextmodeling. To address this issue, we propose a Pale-Shaped self-Attention(PS-Attention), which performs self-attention within a pale-shaped region.Compared to the global self-attention, PS-Attention can reduce the computationand memory costs significantly. Meanwhile, it can capture richer contextualinformation under the similar computation complexity with previous localself-attention mechanisms. Based on the PS-Attention, we develop a generalVision Transformer backbone with a hierarchical architecture, named PaleTransformer, which achieves 83.4%, 84.3%, and 84.9% Top-1 accuracy with themodel size of 22M, 48M, and 85M respectively for 224 ImageNet-1Kclassification, outperforming the previous Vision Transformer backbones. Fordownstream tasks, our Pale Transformer backbone performs better than the recentstate-of-the-art CSWin Transformer by a large margin on ADE20K semanticsegmentation and COCO object detection & instance segmentation. The code willbe released on https://github.com/BR-IDL/PaddleViT.

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention