English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite the wide applications of Adam in reinforcement learning (RL), thetheoretical convergence of Adam-type RL algorithms has not been established.This paper provides the first such convergence analysis for two fundamental RLalgorithms of policy gradient (PG) and temporal difference (TD) learning thatincorporate AMSGrad updates (a standard alternative of Adam in theoreticalanalysis), referred to as PG-AMSGrad and TD-AMSGrad, respectively. Moreover,our analysis focuses on Markovian sampling for both algorithms. We show thatunder general nonlinear function approximation, PG-AMSGrad with a constantstepsize converges to a neighborhood of a stationary point at the rate of$\mathcal{O}(1/T)$ (where $T$ denotes the number of iterations), and with adiminishing stepsize converges exactly to a stationary point at the rate of$\mathcal{O}(\log^2 T/\sqrt{T})$. Furthermore, under linear functionapproximation, TD-AMSGrad with a constant stepsize converges to a neighborhoodof the global optimum at the rate of $\mathcal{O}(1/T)$, and with a diminishingstepsize converges exactly to the global optimum at the rate of$\mathcal{O}(\log T/\sqrt{T})$. Our study develops new techniques for analyzingthe Adam-type RL algorithms under Markovian sampling.

Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling