English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The divergence of the Q-value estimation has been a prominent issue inoffline RL, where the agent has no access to real dynamics. Traditional beliefsattribute this instability to querying out-of-distribution actions whenbootstrapping value targets. Though this issue can be alleviated with policyconstraints or conservative Q estimation, a theoretical understanding of theunderlying mechanism causing the divergence has been absent. In this work, weaim to thoroughly comprehend this mechanism and attain an improved solution. Wefirst identify a fundamental pattern, self-excitation, as the primary cause ofQ-value estimation divergence in offline RL. Then, we propose a novelSelf-Excite Eigenvalue Measure (SEEM) metric based on Neural Tangent Kernel(NTK) to measure the evolving property of Q-network at training, which providesan intriguing explanation of the emergence of divergence. For the first time,our theory can reliably decide whether the training will diverge at an earlystage, and even predict the order of the growth for the estimated Q-value, themodel's norm, and the crashing step when an SGD optimizer is used. Theexperiments demonstrate perfect alignment with this theoretic analysis.Building on our insights, we propose to resolve divergence from a novelperspective, namely improving the model's architecture for better extrapolatingbehavior. Through extensive empirical studies, we identify LayerNorm as a goodsolution to effectively avoid divergence without introducing detrimental bias,leading to superior performance. Experimental results prove that it can stillwork in some most challenging settings, i.e. using only 1 transitions of thedataset, where all previous methods fail. Moreover, it can be easily pluggedinto modern offline RL methods and achieve SOTA results on many challengingtasks. We also give unique insights into its effectiveness.

Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL