English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Actor-critic algorithms learn an explicit policy (actor), and an accompanyingvalue function (critic). The actor performs actions in the environment, whilethe critic evaluates the actor's current policy. However, despite theirstability and promising convergence properties, current actor-critic algorithmsdo not outperform critic-only ones in practice. We believe that the fact thatthe critic learns Q^pi, instead of the optimal Q-function Q*, preventsstate-of-the-art robust and sample-efficient off-policy learning algorithmsfrom being used. In this paper, we propose an elegant solution, theActor-Advisor architecture, in which a Policy Gradient actor learns fromunbiased Monte-Carlo returns, while being shaped (or advised) by the Softmaxpolicy arising from an off-policy critic. The critic can be learnedindependently from the actor, using any state-of-the-art algorithm. Beingadvised by a high-quality critic, the actor quickly and robustly learns thetask, while its use of the Monte-Carlo return helps overcome any bias thecritic may have. In addition to a new Actor-Critic formulation, theActor-Advisor, a method that allows an external advisory policy to shape aPolicy Gradient actor, can be applied to many other domains. By varying thesource of advice, we demonstrate the wide applicability of the Actor-Advisor tothree other important subfields of RL: safe RL with backup policies, efficientleverage of domain knowledge, and transfer learning in RL. Our experimentalresults demonstrate the benefits of the Actor-Advisor compared tostate-of-the-art actor-critic methods, illustrate its applicability to thethree other application scenarios listed above, and show that many importantchallenges of RL can now be solved using a single elegant solution.

The Actor-Advisor: Policy Gradient With Off-Policy Advice