English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Policy-based reinforcement learning algorithms are widely used in variousfields. Among them, mainstream policy optimization algorithms such as TRPO andPPO introduce importance sampling into policy iteration, which allows the reuseof historical data. However, this can also lead to a high variance of thesurrogate objective and indirectly affects the stability and convergence of thealgorithm. In this paper, we first derived an upper bound of the surrogateobjective variance, which can grow quadratically with the increase of thesurrogate objective. Next, we proposed the dropout technique to avoid theexcessive increase of the surrogate objective variance caused by importancesampling. Then, we introduced a general reinforcement learning frameworkapplicable to mainstream policy optimization methods, and applied the dropouttechnique to the PPO algorithm to obtain the D-PPO variant. Finally, we conductcomparative experiments between D-PPO and PPO algorithms in the Atari 2600environment, and the results show that D-PPO achieved significant performanceimprovements compared to PPO, and effectively limited the excessive increase ofthe surrogate objective variance during training.

Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods