English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider off-policy temporal-difference (TD) learning methods for policyevaluation in Markov decision processes with finite spaces and discountedreward criteria, and we present a collection of convergence results for severalgradient-based TD algorithms with linear function approximation. The algorithmswe analyze include: (i) two basic forms of two-time-scale gradient-based TDalgorithms, which we call GTD and which minimize the mean squared projectedBellman error using stochastic gradient-descent; (ii) their "robustified"biased variants; (iii) their mirror-descent versions which combine themirror-descent idea with TD learning; and (iv) a single-time-scale version ofGTD that solves minimax problems formulated for approximate policy evaluation.  We derive convergence results for three types of stepsizes: constantstepsize, slowly diminishing stepsize, as well as the standard type ofdiminishing stepsize with a square-summable condition. For the first two typesof stepsizes, we apply the weak convergence method from stochasticapproximation theory to characterize the asymptotic behavior of the algorithms,and for the standard type of stepsize, we analyze the algorithmic behavior withrespect to a stronger mode of convergence, almost sure convergence. Ourconvergence results are for the aforementioned TD algorithms with three generalways of setting their $\lambda$-parameters: (i) state-dependent $\lambda$; (ii)a recently proposed scheme of using history-dependent $\lambda$ to keep theeligibility traces of the algorithms bounded while allowing for relativelylarge values of $\lambda$; and (iii) a composite scheme of setting the$\lambda$-parameters that combines the preceding two schemes and allows abroader class of generalized Bellman operators to be used for approximatepolicy evaluation with TD methods.

On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learning