English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Transformer based models have shown competitive automatic speechrecognition (ASR) performance. One key factor in the success of these models isthe multi-head attention mechanism. However, for trained models, we havepreviously observed that many attention matrices are close to diagonal,indicating the redundancy of the corresponding attention heads. We have alsofound that some architectures with reduced numbers of attention heads havebetter performance. Since the search for the best structure is timeprohibitive, we propose to randomly remove attention heads during training andkeep all attention heads at test time, thus the final model is an ensemble ofmodels with different architectures. The proposed method also forces each headindependently learn the most useful patterns. We apply the proposed method totrain Transformer based and Convolution-augmented Transformer (Conformer) basedASR models. Our method gives consistent performance gains over strong baselineson the Wall Street Journal, AISHELL, Switchboard and AMI datasets. To the bestof our knowledge, we have achieved state-of-the-art end-to-end Transformerbased model performance on Switchboard and AMI.

Stochastic Attention Head Removal: A simple and effective method for improving Transformer Based ASR Models