English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Self-attention has become a defacto choice for capturing global context invarious vision applications. However, its quadratic computational complexitywith respect to image resolution limits its use in real-time applications,especially for deployment on resource-constrained mobile devices. Althoughhybrid approaches have been proposed to combine the advantages of convolutionsand self-attention for a better speed-accuracy trade-off, the expensive matrixmultiplication operations in self-attention remain a bottleneck. In this work,we introduce a novel efficient additive attention mechanism that effectivelyreplaces the quadratic matrix multiplication operations with linearelement-wise multiplications. Our design shows that the key-value interactioncan be replaced with a linear layer without sacrificing any accuracy. Unlikeprevious state-of-the-art methods, our efficient formulation of self-attentionenables its usage at all stages of the network. Using our proposed efficientadditive attention, we build a series of models called "SwiftFormer" whichachieves state-of-the-art performance in terms of both accuracy and mobileinference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracywith only 0.8 ms latency on iPhone 14, which is more accurate and 2x fastercompared to MobileViT-v2. Code: https://github.com/Amshaker/SwiftFormer

SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications