English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Offline-to-online Reinforcement Learning (O2O RL) aims to improve theperformance of offline pretrained policy using only a few online samples. Builton offline RL algorithms, most O2O methods focus on the balance between RLobjective and pessimism, or the utilization of offline and online samples. Inthis paper, from a novel perspective, we systematically study the challengesthat remain in O2O RL and identify that the reason behind the slow improvementof the performance and the instability of online finetuning lies in theinaccurate Q-value estimation inherited from offline pretraining. Specifically,we demonstrate that the estimation bias and the inaccurate rank of Q-valuecause a misleading signal for the policy update, making the standard offline RLalgorithms, such as CQL and TD3-BC, ineffective in the online finetuning. Basedon this observation, we address the problem of Q-value estimation by twotechniques: (1) perturbed value update and (2) increased frequency of Q-valueupdates. The first technique smooths out biased Q-value estimation with sharppeaks, preventing early-stage policy exploitation of sub-optimal actions. Thesecond one alleviates the estimation bias inherited from offline pretraining byaccelerating learning. Extensive experiments on the MuJoco and Adroitenvironments demonstrate that the proposed method, named SO2, significantlyalleviates Q-value estimation issues, and consistently improves the performanceagainst the state-of-the-art methods by up to 83.1%.

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning