English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently, powerful Transformer architectures have proven superior ingenerating high-quality sentences. Nevertheless, these models tend to producedull high-frequency phrases, severely hurting the diversity and novelty ofgenerated text. In this work, we dig into the intrinsic mechanism of thisproblem and found that sparser attention values in Transformer could improvediversity. To understand such a phenomenon, we first conduct both empirical andtheoretical analysis and then attribute it to representation degenerationcaused by the attentive mixture of the hidden states during training. We termthis process the Trap of Mediocrity. To escape from such a trap, we introduce anovel attention regularization loss to control the sharpness of the attentiondistribution, which is transparent to model structures and can be easilyimplemented within 20 lines of python code. We prove that this method could bemathematically regarded as learning a Bayesian approximation of posteriorattention. Experiments show that our method improved the diversity and noveltyof the generated text while maintaining comparable quality on a variety ofconditional and unconditional generation tasks.

Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention