English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider the problem of learning a set of probability distributions fromthe Bellman dynamics in distributional reinforcement learning~(RL) that learnsthe whole return distribution compared with only its expectation in classicalRL. Despite its success to obtain superior performance, we still have a poorunderstanding of how the value distribution in distributional RL works. In thisstudy, we analyze the optimization benefits of distributional RL by leverage ofadditional value distribution information over classical RL in the NeuralFitted Z-Iteration~(Neural FZI) framework. To begin with, we demonstrate thatthe distribution loss of distributional RL has desirable smoothnesscharacteristics and hence enjoys stable gradients, which is in line with itstendency to promote optimization stability. Furthermore, the accelerationeffect of distributional RL is revealed by decomposing the return distribution.It turns out that distributional RL can perform favorably if the valuedistribution approximation is appropriate, measured by the variance of gradientestimates in each environment for any specific distributional RL algorithm.Rigorous experiments validate the stable optimization behaviors ofdistributional RL, contributing to its acceleration effects compared toclassical RL. The findings of our research illuminate how the valuedistribution in distributional RL algorithms helps the optimization.

How Does Value Distribution in Distributional Reinforcement Learning Help Optimization?