English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent studies have determined that the learned token embeddings oflarge-scale neural language models are degenerated to be anisotropic with anarrow-cone shape. This phenomenon, called the representation degenerationproblem, facilitates an increase in the overall similarity between tokenembeddings that negatively affect the performance of the models. Although theexisting methods that address the degeneration problem based on observations ofthe phenomenon triggered by the problem improves the performance of the textgeneration, the training dynamics of token embeddings behind the degenerationproblem are still not explored. In this study, we analyze the training dynamicsof the token embeddings focusing on rare token embedding. We demonstrate thatthe specific part of the gradient for rare token embeddings is the key cause ofthe degeneration problem for all tokens during training stage. Based on theanalysis, we propose a novel method called, adaptive gradient gating (AGG). AGGaddresses the degeneration problem by gating the specific part of the gradientfor rare token embeddings. Experimental results from language modeling, wordsimilarity, and machine translation tasks quantitatively and qualitativelyverify the effectiveness of AGG.

Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings