English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

This paper studies the relative importance of attention heads inTransformer-based models to aid their interpretability in cross-lingual andmulti-lingual tasks. Prior research has found that only a few attention headsare important in each mono-lingual Natural Language Processing (NLP) task andpruning the remaining heads leads to comparable or improved performance of themodel. However, the impact of pruning attention heads is not yet clear incross-lingual and multi-lingual tasks. Through extensive experiments, we showthat (1) pruning a number of attention heads in a multi-lingualTransformer-based model has, in general, positive effects on its performance incross-lingual and multi-lingual tasks and (2) the attention heads to be prunedcan be ranked using gradients and identified with a few trial experiments. Ourexperiments focus on sequence labeling tasks, with potential applicability onother cross-lingual and multi-lingual tasks. For comprehensiveness, we examinetwo pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) andXLM-R, on three tasks across 9 languages each. We also discuss the validity ofour findings and their extensibility to truly resource-scarce languages andother task settings.

Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks