English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

The dominant paradigm for RLHF is online and on-policy RL: synchronouslygenerating from the large language model (LLM) policy, labelling with a rewardmodel, and learning using feedback on the LLM's own outputs. While performant,this paradigm is computationally inefficient. Inspired by classical deep RLliterature, we propose separating generation and learning in RLHF. This enablesasynchronous generation of new samples while simultaneously training on oldsamples, leading to faster training and more compute-optimal scaling. However,asynchronous training relies on an underexplored regime, online but off-policyRLHF: learning on samples from previous iterations of our model which give aworse training signal. We tackle the fundamental challenge in this regime: howmuch off-policyness can we tolerate for asynchronous training to speed uplearning but maintain performance? Among several RLHF algorithms we test,online DPO is found to be most robust to off-policy data, and robustnessincreases with the scale of the policy model. We study further computeoptimizations for asynchronous RLHF but find that they come at a performancecost, giving rise to a trade-off. We verify the scalability of asynchronousRLHF by training a general-purpose chatbot from LLaMA 3.1 8B on aninstruction-following task ~40% faster than a synchronous run while matchingfinal performance. Finally, we extend our results to math and reasoning todemonstrate asynchronous RL can finetune Rho 1B on GSM8k ~70% faster whilematching synchronous accuracy.

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models