English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Various acceleration approaches for Policy Gradient (PG) have been analyzedwithin the realm of Reinforcement Learning (RL). However, the theoreticalunderstanding of the widely used momentum-based acceleration method on PGremains largely open. In response to this gap, we adapt the celebratedNesterov's accelerated gradient (NAG) method to policy optimization in RL,termed \textit{Accelerated Policy Gradient} (APG). To demonstrate the potentialof APG in achieving fast convergence, we formally prove that with the truegradient and under the softmax policy parametrization, APG converges to anoptimal policy at rates: (i) $\tilde{O}(1/t^2)$ with constant step sizes; (ii)$O(e^{-ct})$ with exponentially-growing step sizes. To the best of ourknowledge, this is the first characterization of the convergence rates of NAGin the context of RL. Notably, our analysis relies on one interesting finding:Regardless of the parameter initialization, APG ends up entering a locallynearly-concave regime, where APG can significantly benefit from the momentum,within finite iterations. Through numerical validation and experiments on theAtari 2600 benchmarks, we confirm that APG exhibits a $\tilde{O}(1/t^2)$ ratewith constant step sizes and a linear convergence rate withexponentially-growing step sizes, significantly improving convergence over thestandard PG.

Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning