English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we provide a unified analysis of temporal difference learningalgorithms with linear function approximators by exploiting their connectionsto Markov jump linear systems (MJLS). We tailor the MJLS theory developed inthe control community to characterize the exact behaviors of the first andsecond order moments of a large family of temporal difference learningalgorithms. For both the IID and Markov noise cases, we show that the evolutionof some augmented versions of the mean and covariance matrix of the TDestimation error exactly follows the trajectory of a deterministic lineartime-invariant (LTI) dynamical system. Applying the well-known LTI systemtheory, we obtain closed-form expressions for the mean and covariance matrix ofthe TD estimation error at any time step. We provide a tight matrix spectralradius condition to guarantee the convergence of the covariance matrix of theTD estimation error, and perform a perturbation analysis to characterize thedependence of the TD behaviors on learning rate. For the IID case, we providean exact formula characterizing how the mean and covariance matrix of the TDestimation error converge to the steady state values. For the Markov case, weuse our formulas to explain how the behaviors of TD learning algorithms areaffected by learning rate and the underlying Markov chain. For both cases,upper and lower bounds for the mean square TD error are provided. The meansquare TD error is shown to converge linearly to an exact limit.

Characterizing the Exact Behaviors of Temporal Difference Learning Algorithms Using Markov Jump Linear System Theory