English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Policy optimization is a fundamental principle for designing reinforcementlearning algorithms, and one example is the proximal policy optimizationalgorithm with a clipped surrogate objective (PPO-Clip), which has beenpopularly used in deep reinforcement learning due to its simplicity andeffectiveness. Despite its superior empirical performance, PPO-Clip has notbeen justified via theoretical proof up to date. In this paper, we establishthe first global convergence rate of PPO-Clip under neural functionapproximation. We identify the fundamental challenges of analyzing PPO-Clip andaddress them with the two core ideas: (i) We reinterpret PPO-Clip from theperspective of hinge loss, which connects policy improvement with solving alarge-margin classification problem with hinge loss and offers a generalizedversion of the PPO-Clip objective. (ii) Based on the above viewpoint, wepropose a two-step policy improvement scheme, which facilitates the convergenceanalysis by decoupling policy search from the complex neural policyparameterization with the help of entropic mirror descent and aregression-based policy update scheme. Moreover, our theoretical resultsprovide the first characterization of the effect of the clipping mechanism onthe convergence of PPO-Clip. Through experiments, we empirically validate thereinterpretation of PPO-Clip and the generalized objective with variousclassifiers on various RL benchmark tasks.

Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective