English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The great success of Transformer-based models benefits from the powerfulmulti-head self-attention mechanism, which learns token dependencies andencodes contextual information from the input. Prior work strives to attributemodel decisions to individual input features with different saliency measures,but they fail to explain how these input features interact with each other toreach predictions. In this paper, we propose a self-attention attributionmethod to interpret the information interactions inside Transformer. We takeBERT as an example to conduct extensive studies. Firstly, we applyself-attention attribution to identify the important attention heads, whileothers can be pruned with marginal performance degradation. Furthermore, weextract the most salient dependencies in each layer to construct an attributiontree, which reveals the hierarchical interactions inside Transformer. Finally,we show that the attribution results can be used as adversarial patterns toimplement non-targeted attacks towards BERT.

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer