English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained language models (PLM) have demonstrated their effectiveness for abroad range of information retrieval and natural language processing tasks. Asthe core part of PLM, multi-head self-attention is appealing for its ability tojointly attend to information from different positions. However, researchershave found that PLM always exhibits fixed attention patterns regardless of theinput (e.g., excessively paying attention to [CLS] or [SEP]), which we arguemight neglect important information in the other positions. In this work, wepropose a simple yet effective attention guiding mechanism to improve theperformance of PLM by encouraging attention towards the established goals.Specifically, we propose two kinds of attention guiding methods, i.e., mapdiscrimination guiding (MDG) and attention pattern decorrelation guiding (PDG).The former definitely encourages the diversity among multiple self-attentionheads to jointly attend to information from different representation subspaces,while the latter encourages self-attention to attend to as many differentpositions of the input as possible. We conduct experiments with multiplegeneral pre-trained models (i.e., BERT, ALBERT, and Roberta) anddomain-specific pre-trained models (i.e., BioBERT, ClinicalBERT, BlueBert, andSciBERT) on three benchmark datasets (i.e., MultiNLI, MedNLI, andCross-genre-IR). Extensive experimental results demonstrate that our proposedMDG and PDG bring stable performance improvements on all datasets with highefficiency and low cost.

Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding