English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Reinforcement learning (RL) is frequently employed in fine-tuning largelanguage models (LMs), such as GPT-3, to penalize them for undesirable featuresof generated sequences, such as offensiveness, social bias, harmfulness orfalsehood. The RL formulation involves treating the LM as a policy and updatingit to maximise the expected value of a reward function which captures humanpreferences, such as non-offensiveness. In this paper, we analyze challengesassociated with treating a language model as an RL policy and show how avoidingthose challenges requires moving beyond the RL paradigm. We start by observingthat the standard RL approach is flawed as an objective for fine-tuning LMsbecause it leads to distribution collapse: turning the LM into a degeneratedistribution. Then, we analyze KL-regularised RL, a widely used recipe forfine-tuning LMs, which additionally constrains the fine-tuned LM to stay closeto its original distribution in terms of Kullback-Leibler (KL) divergence. Weshow that KL-regularised RL is equivalent to variational inference:approximating a Bayesian posterior which specifies how to update a prior LM toconform with evidence provided by the reward function. We argue that thisBayesian inference view of KL-regularised RL is more insightful than thetypically employed RL perspective. The Bayesian inference view explains howKL-regularised RL avoids the distribution collapse problem and offers afirst-principles derivation for its objective. While this objective happens tobe equivalent to RL (with a particular choice of parametric reward), thereexist other objectives for fine-tuning LMs which are no longer equivalent toRL. That observation leads to a more general point: RL is not an adequateformal framework for problems such as fine-tuning language models. Theseproblems are best viewed as Bayesian inference: approximating a pre-definedtarget distribution.

RL with KL penalties is better viewed as Bayesian inference