English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It has been widely observed that exact or approximate MAP (mode-seeking)decoding from natural language generation (NLG) models consistently leads todegenerate outputs (Stahlberg and Byrne, 2019, Holtzman et al., 2019). This hasgenerally been attributed to either a fundamental inadequacy of modes in modelsor weaknesses in language modeling. Contrastingly in this work, we emphasizethat degenerate modes can even occur in the absence of any model error, due tocontamination of the training data. Specifically, we show that mixing even atiny amount of low-entropy noise with a population text distribution can causethe data distribution's mode to become degenerate, implying that any modelstrained on it will be as well. As the unconditional mode of NLG models willoften be degenerate, we therefore propose to apply MAP decoding to the model'sdistribution conditional on avoiding specific degeneracies. Using exact-search,we empirically verify that the length-conditional modes of machine translationmodels and language models are indeed more fluent and topical than theirunconditional modes. For the first time, we also share many examples of exactmodal sequences from these models, and from several variants of the LLaMA-7Bmodel. Notably, the modes of the LLaMA models are still degenerate, showingthat improvements in modeling have not fixed this issue. Because of the cost ofexact mode finding algorithms, we develop an approximate mode finding approach,ACBS, which finds sequences that are both high-likelihood and high-quality. Weapply this approach to LLaMA-7B, a model which was not trained for instructionfollowing, and find that we are able to elicit reasonable outputs without anyfinetuning.

MAP's not dead yet: Uncovering true language model modes by conditioning away degeneracy