English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Meta-gradient Reinforcement Learning (RL) allows agents to self-tune theirhyper-parameters in an online fashion during training. In this paper, weidentify a bias in the meta-gradient of current meta-gradient RL approaches.This bias comes from using the critic that is trained using the meta-learneddiscount factor for the advantage estimation in the outer objective whichrequires a different discount factor. Because the meta-learned discount factoris typically lower than the one used in the outer objective, the resulting biascan cause the meta-gradient to favor myopic policies. We propose a simplesolution to this issue: we eliminate this bias by using an alternative,\emph{outer} value function in the estimation of the outer loss. To obtain thisouter value function we add a second head to the critic network and train italongside the classic critic, using the outer loss discount factor. On anillustrative toy problem, we show that the bias can cause catastrophic failureof current meta-gradient RL approaches, and show that our proposed solutionfixes it. We then apply our method to a more complex environment anddemonstrate that fixing the meta-gradient bias can significantly improveperformance.

Debiasing Meta-Gradient Reinforcement Learning by Learning the Outer Value Function