English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Centralised training with decentralised execution (CT-DE) serves as thefoundation of many leading multi-agent reinforcement learning (MARL)algorithms. Despite its popularity, it suffers from a critical drawback due toits reliance on learning from a single sample of the joint-action at a givenstate. As agents explore and update their policies during training, thesesingle samples may poorly represent the actual joint-policy of the system ofagents leading to high variance gradient estimates that hinder learning. Toaddress this problem, we propose an enhancement tool that accommodates anyactor-critic MARL method. Our framework, Performance Enhancing ReinforcementLearning Apparatus (PERLA), introduces a sampling technique of the agents'joint-policy into the critics while the agents train. This leads to TD updatesthat closely approximate the true expected value under the current joint-policyrather than estimates from a single sample of the joint-action at a givenstate. This produces low variance and precise estimates of expected returns,minimising the variance in the critic estimators which typically hinderslearning. Moreover, as we demonstrate, by eliminating much of the criticvariance from the single sampling of the joint policy, PERLA enables CT-DEmethods to scale more efficiently with the number of agents. Theoretically, weprove that PERLA reduces variance in value estimates similar to that ofdecentralised training while maintaining the benefits of centralised training.Empirically, we demonstrate PERLA's superior performance and ability to reduceestimator variance in a range of benchmarks including Multi-agent Mujoco, andStarCraft II Multi-agent Challenge.

Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction