English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

It is desirable for policies to optimistically explore new states andbehaviors during online reinforcement learning (RL) or fine-tuning, especiallywhen prior offline data does not provide enough state coverage. However,exploration bonuses can bias the learned policy, and our experiments find thatnaive, yet standard use of such bonuses can fail to recover a performantpolicy. Concurrently, pessimistic training in offline RL has enabled recoveryof performant policies from static datasets. Can we leverage offline RL torecover better policies from online interaction? We make a simple observationthat a policy can be trained from scratch on all interaction data withpessimistic objectives, thereby decoupling the policies used for datacollection and for evaluation. Specifically, we propose offline retraining, apolicy extraction step at the end of online fine-tuning in ourOffline-to-Online-to-Offline (OOO) framework for reinforcement learning (RL).An optimistic (exploration) policy is used to interact with the environment,and a separate pessimistic (exploitation) policy is trained on all the observeddata for evaluation. Such decoupling can reduce any bias from onlineinteraction (intrinsic rewards, primacy bias) in the evaluation policy, and canallow more exploratory behaviors during online interaction which in turn cangenerate better data for exploitation. OOO is complementary to severaloffline-to-online RL and online RL methods, and improves their averageperformance by 14% to 26% in our fine-tuning experiments, achievesstate-of-the-art performance on several environments in the D4RL benchmarks,and improves online RL performance by 165% on two OpenAI gym environments.Further, OOO can enable fine-tuning from incomplete offline datasets whereprior methods can fail to recover a performant policy. Implementation:https://github.com/MaxSobolMark/OOO

Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias