English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Gradient Temporal Difference (GTD) algorithms (Sutton et al., 2008, 2009) arethe first $O(d)$ ($d$ is the number features) algorithms that have convergenceguarantees for off-policy learning with linear function approximation. Liu etal. (2015) and Dalal et. al. (2018) proved the convergence rates of GTD, GTD2and TDC are $O(t^{-\alpha/2})$ for some $\alpha \in (0,1)$. This bound is tight(Dalal et al., 2020), and slower than $O(1/\sqrt{t})$. GTD algorithms also havetwo step-size parameters, which are difficult to tune. In literature, there isa "single-time-scale" formulation of GTD. However, this formulation still hastwo step-size parameters.  This paper presents a truly single-time-scale GTD algorithm for minimizingthe Norm of Expected td Update (NEU) objective, and it has only one step-sizeparameter. We prove that the new algorithm, called Impression GTD, converges atleast as fast as $O(1/t)$. Furthermore, based on a generalization of theexpected smoothness (Gower et al. 2019), called $L$-$\lambda$ smoothness, weare able to prove that the new GTD converges even faster, in fact, with alinear rate. Our rate actually also improves Gower et al.'s result with atighter bound under a weaker assumption. Besides Impression GTD, we also provethe rates of three other GTD algorithms, one by Yao and Liu (2008), anothercalled A-transpose-TD (Sutton et al., 2008), and a counterpart ofA-transpose-TD. The convergence rates of all the four GTD algorithms are provedin a single generic GTD framework to which $L$-$\lambda$ smoothness applies.Empirical results on Random walks, Boyan chain, and Baird counterexample showthat Impression GTD converges much faster than existing GTD algorithms for bothon-policy and off-policy learning problems, with well-performing step-sizes ina big range.

A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using $L$-$\lambda$ Smoothness