English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We provide a theoretical framework for Reinforcement Learning with HumanFeedback (RLHF). Our analysis shows that when the true reward function islinear, the widely used maximum likelihood estimator (MLE) converges under boththe Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However,we show that when training a policy based on the learned reward model, MLEfails while a pessimistic MLE provides policies with improved performance undercertain coverage assumptions. Additionally, we demonstrate that under the PLmodel, the true MLE and an alternative MLE that splits the $K$-wise comparisoninto pairwise comparisons both converge. Moreover, the true MLE isasymptotically more efficient. Our results validate the empirical success ofexisting RLHF algorithms in InstructGPT and provide new insights for algorithmdesign. Furthermore, our results unify the problem of RLHF and max-entropyInverse Reinforcement Learning (IRL), and provide the first sample complexitybound for max-entropy IRL.

Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons