English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Currently, one main research line in designing a more efficient visiontransformer is reducing the computational cost of self attention modules byadopting sparse attention or using local attention windows. In contrast, wepropose a different approach that aims to improve the performance oftransformer-based architectures by densifying the attention pattern.Specifically, we proposed forward cross attention for hybrid vision transformer(FcaFormer), where tokens from previous blocks in the same stage are secondaryused. To achieve this, the FcaFormer leverages two innovative components:learnable scale factors (LSFs) and a token merge and enhancement module (TME).The LSFs enable efficient processing of cross tokens, while the TME generatesrepresentative cross tokens. By integrating these components, the proposedFcaFormer enhances the interactions of tokens across blocks with potentiallydifferent semantics, and encourages more information flows to the lower levels.Based on the forward cross attention (Fca), we have designed a series ofFcaFormer models that achieve the best trade-off between model size,computational cost, memory cost, and accuracy. For example, without the needfor knowledge distillation to strengthen training, our FcaFormer achieves 83.1%top-1 accuracy on Imagenet with only 16.3 million parameters and about 3.6billion MACs. This saves almost half of the parameters and a few computationalcosts while achieving 0.7% higher accuracy compared to distilledEfficientFormer.

Fcaformer: Forward Cross Attention in Hybrid Vision Transformer