English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Preference learning is a key technology for aligning language models withhuman values. Reinforcement Learning from Human Feedback (RLHF) is a modelbased algorithm to optimize preference learning, which first fitting a rewardmodel for preference score, and then optimizing generating policy withon-policy PPO algorithm to maximize the reward. The processing of RLHF iscomplex, time-consuming and unstable. Direct Preference Optimization (DPO)algorithm using off-policy algorithm to direct optimize generating policy andeliminating the need for reward model, which is data efficient and stable. DPOuse Bradley-Terry model and log-loss which leads to over-fitting to thepreference data at the expense of ignoring KL-regularization term whenpreference is deterministic. IPO uses a root-finding MSE loss to solve theignoring KL-regularization problem. In this paper, we'll figure out, althoughIPO fix the problem when preference is deterministic, but both DPO and IPOfails the KL-regularization term because the support of preference distributionnot equal to reference distribution. Then, we design a simple and intuitiveoff-policy preference optimization algorithm from an importance sampling view,which we call Maximum Preference Optimization (MPO), and add off-policyKL-regularization terms which makes KL-regularization truly effective. Theobjective of MPO bears resemblance to RLHF's objective, and likes IPO, MPO isoff-policy. So, MPO attains the best of both worlds. To simplify the learningprocess and save memory usage, MPO eliminates the needs for both reward modeland reference policy.

Preference as Reward, Maximum Preference Optimization with Importance Sampling