English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In safe reinforcement learning (SRL) problems, an agent explores theenvironment to maximize an expected total reward and meanwhile avoids violationof certain constraints on a number of expected total costs. In general, suchSRL problems have nonconvex objective functions subject to multiple nonconvexconstraints, and hence are very challenging to solve, particularly to provide aglobally optimal policy. Many popular SRL algorithms adopt a primal-dualstructure which utilizes the updating of dual variables for satisfying theconstraints. In contrast, we propose a primal approach, calledconstraint-rectified policy optimization (CRPO), which updates the policyalternatingly between objective improvement and constraint satisfaction. CRPOprovides a primal-type algorithmic framework to solve SRL problems, where eachpolicy update can take any variant of policy optimization step. To demonstratethe theoretical performance of CRPO, we adopt natural policy gradient (NPG) foreach policy update step and show that CRPO achieves an$\mathcal{O}(1/\sqrt{T})$ convergence rate to the global optimal policy in theconstrained policy set and an $\mathcal{O}(1/\sqrt{T})$ error bound onconstraint satisfaction. This is the first finite-time analysis of primal SRLalgorithms with global optimality guarantee. Our empirical results demonstratethat CRPO can outperform the existing primal-dual baseline algorithmssignificantly.

CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee