English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

There are a number of diverging hypotheses about the neural text degenerationproblem, i.e., generating repetitive and dull loops, which makes this problemboth interesting and confusing. In this work, we aim to advance ourunderstanding by presenting a straightforward and fundamental explanation fromthe data perspective. Our preliminary investigation reveals a strongcorrelation between the degeneration issue and the presence of repetitions intraining data. Subsequent experiments also demonstrate that by selectivelydropping out the attention to repetitive words in training data, degenerationcan be significantly minimized. Furthermore, our empirical analysis illustratesthat prior works addressing the degeneration issue from various standpoints,such as the high-inflow words, the likelihood objective, and theself-reinforcement phenomenon, can be interpreted by one simple explanation.That is, penalizing the repetitions in training data is a common andfundamental factor for their effectiveness. Moreover, our experiments revealthat penalizing the repetitions in training data remains critical even whenconsidering larger model sizes and instruction tuning.

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective