English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When inferring reward functions from human behavior (be it demonstrations,comparisons, physical corrections, or e-stops), it has proven useful to modelthe human as making noisy-rational choices, with a "rationality coefficient"capturing how much noise or entropy we expect to see in the human behavior.Prior work typically sets the rationality level to a constant value, regardlessof the type, or quality, of human feedback. However, in many settings, givingone type of feedback (e.g. a demonstration) may be much more difficult than adifferent type of feedback (e.g. answering a comparison query). Thus, we expectto see more or less noise depending on the type of human feedback. In thiswork, we advocate that grounding the rationality coefficient in real data foreach feedback type, rather than assuming a default value, has a significantpositive effect on reward learning. We test this in both simulated experimentsand in a user study with real human feedback. We find that overestimating humanrationality can have dire effects on reward learning accuracy and regret. Wealso find that fitting the rationality coefficient to human data enables betterreward learning, even when the human deviates significantly from thenoisy-rational choice model due to systematic biases. Further, we find that therationality level affects the informativeness of each feedback type:surprisingly, demonstrations are not always the most informative -- when thehuman acts very suboptimally, comparisons actually become more informative,even when the rationality level is the same for both. Ultimately, our resultsemphasize the importance and advantage of paying attention to the assumedhuman-rationality level, especially when agents actively learn from multipletypes of human feedback.

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types