English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite empirical success, the theory of reinforcement learning (RL) withvalue function approximation remains fundamentally incomplete. Prior work hasidentified a variety of pathological behaviours that arise in RL algorithmsthat combine approximate on-policy evaluation and greedification. One prominentexample is policy oscillation, wherein an algorithm may cycle indefinitelybetween policies, rather than converging to a fixed point. What is not wellunderstood however is the quality of the policies in the region of oscillation.In this paper we present simple examples illustrating that in addition topolicy oscillation and multiple fixed points -- the same basic issue can leadto convergence to the worst possible policy for a given approximation. Suchbehaviours can arise when algorithms optimize evaluation accuracy weighted bythe distribution of states that occur under the current policy, but greedifybased on the value of states which are rare or nonexistent under thisdistribution. This means the values used for greedification are unreliable andcan steer the policy in undesirable directions. Our observation that this canlead to the worst possible policy shows that in a general sense such algorithmsare unreliable. The existence of such examples helps to narrow the kind oftheoretical guarantees that are possible and the kind of algorithmic ideas thatare likely to be helpful. We demonstrate analytically and experimentally thatsuch pathological behaviours can impact a wide range of RL and dynamicprogramming algorithms; such behaviours can arise both with and withoutbootstrapping, and with linear function approximation as well as with morecomplex parameterized functions like neural networks.

Understanding the Pathologies of Approximate Policy Evaluation when Combined with Greedification in Reinforcement Learning