English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we establish the global optimality and convergence rate of anoff-policy actor critic algorithm in the tabular setting without using densityratio to correct the discrepancy between the state distribution of the behaviorpolicy and that of the target policy. Our work goes beyond existing works onthe optimality of policy gradient methods in that existing works use the exactpolicy gradient for updating the policy parameters while we use an approximateand stochastic update step. Our update step is not a gradient update because wedo not use a density ratio to correct the state distribution, which aligns wellwith what practitioners do. Our update is approximate because we use a learnedcritic instead of the true value function. Our update is stochastic because ateach step the update is done for only the current state action pair. Moreover,we remove several restrictive assumptions from existing works in our analysis.Central to our work is the finite sample analysis of a generic stochasticapproximation algorithm with time-inhomogeneous update operators ontime-inhomogeneous Markov chains, based on its uniform contraction properties.

Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch