English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The strong performance of vision transformers on image classification andother vision tasks is often attributed to the design of their multi-headattention layers. However, the extent to which attention is responsible forthis strong performance remains unclear. In this short report, we ask: is theattention layer even necessary? Specifically, we replace the attention layer ina vision transformer with a feed-forward layer applied over the patchdimension. The resulting architecture is simply a series of feed-forward layersapplied over the patch and feature dimensions in an alternating fashion. Inexperiments on ImageNet, this architecture performs surprisingly well: aViT/DeiT-base-sized model obtains 74.9\% top-1 accuracy, compared to 77.9\% and79.9\% for ViT and DeiT respectively. These results indicate that aspects ofvision transformers other than attention, such as the patch embedding, may bemore responsible for their strong performance than previously thought. We hopethese results prompt the community to spend more time trying to understand whyour current models are as effective as they are.

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet