English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The attention mechanism is considered the backbone of the widely-usedTransformer architecture. It contextualizes the input by computinginput-specific attention matrices. We find that this mechanism, while powerfuland elegant, is not as important as typically thought for pretrained languagemodels. We introduce PAPA, a new probing method that replaces theinput-dependent attention matrices with constant ones -- the average attentionweights over multiple inputs. We use PAPA to analyze several establishedpretrained Transformers on six downstream tasks. We find that without anyinput-dependent attention, all models achieve competitive performance -- anaverage relative drop of only 8% from the probing baseline. Further, little orno performance drop is observed when replacing half of the input-dependentattention matrices with constant (input-independent) ones. Interestingly, weshow that better-performing models lose more from applying our method thanweaker models, suggesting that the utilization of the input-dependent attentionmechanism might be a factor in their success. Our results motivate research onsimpler alternatives to input-dependent attention, as well as on methods forbetter utilization of this mechanism in the Transformer architecture.

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers