English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In recent years, reinforcement learning (RL) systems with general goalsbeyond a cumulative sum of rewards have gained traction, such as in constrainedproblems, exploration, and acting upon prior experiences. In this paper, weconsider policy optimization in Markov Decision Problems, where the objectiveis a general concave utility function of the state-action occupancy measure,which subsumes several of the aforementioned examples as special cases. Suchgenerality invalidates the Bellman equation. As this means that dynamicprogramming no longer works, we focus on direct policy search. Analogously tothe Policy Gradient Theorem \cite{sutton2000policy} available for RL withcumulative rewards, we derive a new Variational Policy Gradient Theorem for RLwith general utilities, which establishes that the parametrized policy gradientmay be obtained as the solution of a stochastic saddle point problem involvingthe Fenchel dual of the utility function. We develop a variational Monte Carlogradient estimation algorithm to compute the policy gradient based on samplepaths. We prove that the variational policy gradient scheme converges globallyto the optimal policy for the general objective, though the optimizationproblem is nonconvex. We also establish its rate of convergence of the order$O(1/t)$ by exploiting the hidden convexity of the problem, and proves that itconverges exponentially when the problem admits hidden strong convexity. Ouranalysis applies to the standard RL problem with cumulative rewards as aspecial case, in which case our result improves the available convergence rate.

Variational Policy Gradient Method for Reinforcement Learning with General Utilities