English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Reward models play a key role in aligning language model applications towardshuman preferences. However, this setup creates an incentive for the languagemodel to exploit errors in the reward model to achieve high estimated reward, aphenomenon often termed \emph{reward hacking}. A natural mitigation is to trainan ensemble of reward models, aggregating over model outputs to obtain a morerobust reward estimate. We explore the application of reward ensembles toalignment at both training time (through reinforcement learning) and inferencetime (through reranking). First, we show that reward models are\emph{underspecified}: reward models that perform similarly in-distribution canyield very different rewards when used in alignment, due to distribution shift.Second, underspecification results in overoptimization, where alignment to onereward model does not improve reward as measured by another reward modeltrained on the same data. Third, overoptimization is mitigated by the use ofreward ensembles, and ensembles that vary by their \emph{pretraining} seedslead to better generalization than ensembles that differ only by their\emph{fine-tuning} seeds, with both outperforming individual reward models.However, even pretrain reward ensembles do not eliminate reward hacking: weshow several qualitative reward hacking phenomena that are not mitigated byensembling because all reward models in the ensemble exhibit similar errorpatterns.

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking