English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Baird counterexample was proposed by Leemon Baird in 1995, first used to showthat the Temporal Difference (TD(0)) algorithm diverges on this example. Sincethen, it is often used to test and compare off-policy learning algorithms.Gradient TD algorithms solved the divergence issue of TD on Bairdcounterexample. However, their convergence on this example is still very slow,and the nature of the slowness is not well understood, e.g., see (Sutton andBarto 2018).  This note is to understand in particular, why TDC is slow on this example,and provide a debugging analysis to understand this behavior. Our debuggingtechnique can be used to study the convergence behavior of two-time-scalestochastic approximation algorithms. We also provide empirical results of therecent Impression GTD algorithm on this example, showing the convergence isvery fast, in fact, in a linear rate. We conclude that Baird counterexample issolved, by an algorithm with the convergence guarantee to the TD solution ingeneral, and a fast convergence rate.

Baird Counterexample is Solved: with an example of How to Debug a Two-time-scale Algorithm