English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider an offline reinforcement learning (RL) setting where the agentneed to learn from a dataset collected by rolling out multiple behaviorpolicies. There are two challenges for this setting: 1) The optimal trade-offbetween optimizing the RL signal and the behavior cloning (BC) signal changeson different states due to the variation of the action coverage induced bydifferent behavior policies. Previous methods fail to handle this by onlycontrolling the global trade-off. 2) For a given state, the action distributiongenerated by different behavior policies may have multiple modes. The BCregularizers in many previous methods are mean-seeking, resulting in policiesthat select out-of-distribution (OOD) actions in the middle of the modes. Inthis paper, we address both challenges by using adaptively weighted reverseKullback-Leibler (KL) divergence as the BC regularizer based on the TD3algorithm. Our method not only trades off the RL and BC signals with per-stateweights (i.e., strong BC regularization on the states with narrow actioncoverage, and vice versa) but also avoids selecting OOD actions thanks to themode-seeking property of reverse KL. Empirically, our algorithm can outperformexisting offline RL algorithms in the MuJoCo locomotion tasks with the standardD4RL datasets as well as the mixed datasets that combine the standard datasets.

TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets