English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In order to solve a task using reinforcement learning, it is necessary tofirst formalise the goal of that task as a reward function. However, for manyreal-world tasks, it is very difficult to manually specify a reward functionthat never incentivises undesirable behaviour. As a result, it is increasinglypopular to use reward learning algorithms, which attempt to learn a rewardfunction from data. However, the theoretical foundations of reward learning arenot yet well-developed. In particular, it is typically not known when a givenreward learning algorithm with high probability will learn a reward functionthat is safe to optimise. This means that reward learning algorithms generallymust be evaluated empirically, which is expensive, and that their failure modesare difficult to predict in advance. One of the roadblocks to deriving bettertheoretical guarantees is the lack of good methods for quantifying thedifference between reward functions. In this paper we provide a solution tothis problem, in the form of a class of pseudometrics on the space of allreward functions that we call STARC (STAndardised Reward Comparison) metrics.We show that STARC metrics induce both an upper and a lower bound on worst-caseregret, which implies that our metrics are tight, and that any metric with thesame properties must be bilipschitz equivalent to ours. Moreover, we alsoidentify a number of issues with reward metrics proposed by earlier works.Finally, we evaluate our metrics empirically, to demonstrate their practicalefficacy. STARC metrics can be used to make both theoretical and empiricalanalysis of reward learning algorithms both easier and more principled.

STARC: A General Framework For Quantifying Differences Between Reward Functions