English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Two-timescale stochastic approximation (TTSA) is among the most generalframeworks for iterative stochastic algorithms. This includes well-knownstochastic optimization methods such as SGD variants and those designed forbilevel or minimax problems, as well as reinforcement learning like the familyof gradient-based temporal difference (GTD) algorithms. In this paper, weconduct an in-depth asymptotic analysis of TTSA under controlled Markoviannoise via central limit theorem (CLT), uncovering the coupled dynamics of TTSAinfluenced by the underlying Markov chain, which has not been addressed byprevious CLT results of TTSA only with Martingale difference noise. Buildingupon our CLT, we expand its application horizon of efficient samplingstrategies from vanilla SGD to a wider TTSA context in distributed learning,thus broadening the scope of Hu et al. (2022). In addition, we leverage our CLTresult to deduce the statistical properties of GTD algorithms with nonlinearfunction approximation using Markovian samples and show their identicalasymptotic performance, a perspective not evident from current finite-timebounds.

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications