English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study policy optimization in an infinite horizon, $\gamma$-discountedconstrained Markov decision process (CMDP). Our objective is to return a policythat achieves large expected reward with a small constraint violation. Weconsider the online setting with linear function approximation and assumeglobal access to the corresponding features. We propose a generic primal-dualframework that allows us to bound the reward sub-optimality and constraintviolation for arbitrary algorithms in terms of their primal and dual regret ononline linear optimization problems. We instantiate this framework to usecoin-betting algorithms and propose the Coin Betting Politex (CBP) algorithm.Assuming that the action-value functions are $\varepsilon_b$-close to the spanof the $d$-dimensional state-action features and no sampling errors, we provethat $T$ iterations of CBP result in an $O\left(\frac{1}{(1 - \gamma)^3\sqrt{T}} + \frac{\varepsilon_b\sqrt{d}}{(1 - \gamma)^2} \right)$ rewardsub-optimality and an $O\left(\frac{1}{(1 - \gamma)^2 \sqrt{T}} +\frac{\varepsilon_b \sqrt{d}}{1 - \gamma} \right)$ constraint violation.Importantly, unlike gradient descent-ascent and other recent methods, CBP doesnot require extensive hyperparameter tuning. Via experiments on synthetic andCartpole environments, we demonstrate the effectiveness and robustness of CBP.

Towards Painless Policy Optimization for Constrained MDPs