English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In temporal difference (TD) learning, off-policy sampling is known to be morepractical than on-policy sampling, and by decoupling learning from datacollection, it enables data reuse. It is known that policy evaluation(including multi-step off-policy importance sampling) has the interpretation ofsolving a generalized Bellman equation. In this paper, we derive finite-samplebounds for any general off-policy TD-like stochastic approximation algorithmthat solves for the fixed-point of this generalized Bellman operator. Our keystep is to show that the generalized Bellman operator is simultaneously acontraction mapping with respect to a weighted $\ell_p$-norm for each $p$ in$[1,\infty)$, with a common contraction factor.  Off-policy TD-learning is known to suffer from high variance due to theproduct of importance sampling ratios. A number of algorithms (e.g.$Q^\pi(\lambda)$, Tree-Backup$(\lambda)$, Retrace$(\lambda)$, and $Q$-trace)have been proposed in the literature to address this issue. Our resultsimmediately imply finite-sample bounds of these algorithms. In particular, weprovide first-known finite-sample guarantees for $Q^\pi(\lambda)$,Tree-Backup$(\lambda)$, and Retrace$(\lambda)$, and improve the best knownbounds of $Q$-trace in [19]. Moreover, we show the bias-variance trade-offs ineach of these algorithms.

Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators