English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The stochastic approximation (SA) algorithm is a widely used probabilisticmethod for finding a zero or a fixed point of a vector-valued funtion, whenonly noisy measurements of the function are available. In the literature todate, one makes a distinction between ``synchronous'' updating, whereby everycomponent of the current guess is updated at each time, and ``asynchronous''updating, whereby only one component is updated. In this paper, we study anintermediate situation that we call ``batch asynchronous stochasticapproximation'' (BASA), in which, at each time instant, \textit{some but notall} components of the current estimated solution are updated. BASA allows theuser to trade off memory requirements against time complexity. We develop ageneral methodology for proving that such algorithms converge to the fixedpoint of the map under study. These convergence proofs make use of weakerhypotheses than existing results. Specifically, existing convergence proofsrequire that the measurement noise is a zero-mean i.i.d\ sequence or amartingale difference sequence. In the present paper, we permit biasedmeasurements, that is, measurement noises that have nonzero conditional mean.Also, all convergence results to date assume that the stochastic step sizessatisfy a probabilistic analog of the well-known Robbins-Monro conditions. Wereplace this assumption by a purely deterministic condition on theirreducibility of the underlying Markov processes.  As specific applications to Reinforcement Learning, we analyze the temporaldifference algorithm $TD(\lambda)$ for value iteration, and the $Q$-learningalgorithm for finding the optimal action-value function. In both cases, weestablish the convergence of these algorithms, under milder conditions than inthe existing literature.

Convergence of Batch Asynchronous Stochastic Approximation With Applications to Reinforcement Learning