English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The actor-critic (AC) reinforcement learning algorithms have been thepowerhouse behind many challenging applications. Nevertheless, its convergenceis fragile in general. To study its instability, existing works mostly considerthe uncommon double-loop variant or basic models with finite state and actionspace. We investigate the more practical single-sample two-timescale AC forsolving the canonical linear quadratic regulator (LQR) problem, where the actorand the critic update only once with a single sample in each iteration on anunbounded continuous state and action space. Existing analysis cannot concludethe convergence for such a challenging case. We develop a new analysisframework that allows establishing the global convergence to an$\epsilon$-optimal solution with at most an $\mathcal{O}(\epsilon^{-2.5})$sample complexity. To our knowledge, this is the first finite-time convergenceanalysis for the single sample two-timescale AC for solving LQR with globaloptimality. The sample complexity improves those of other variants by orders,which sheds light on the practical wisdom of single sample algorithms. We alsofurther validate our theoretical findings via comprehensive simulationcomparisons.

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator