English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep reinforcement learning (RL) works impressively in some environments andfails catastrophically in others. Ideally, RL theory should be able to providean understanding of why this is, i.e. bounds predictive of practicalperformance. Unfortunately, current theory does not quite have this ability. Wecompare standard deep RL algorithms to prior sample complexity bounds byintroducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs fromcommon deep RL benchmarks, along with their corresponding tabularrepresentations, which enables us to exactly compute instance-dependent bounds.We choose to focus on deterministic environments because they share manyinteresting properties of stochastic environments, but are easier to analyze.Using BRIDGE, we find that prior bounds do not correlate well with when deep RLsucceeds vs. fails, but discover a surprising property that does. When actionswith the highest Q-values under the random policy also have the highestQ-values under the optimal policy (i.e. when it is optimal to be greedy on therandom policy's Q function), deep RL tends to succeed; when they don't, deep RLtends to fail. We generalize this property into a new complexity measure of anMDP that we call the effective horizon, which roughly corresponds to how manysteps of lookahead search would be needed in that MDP in order to identify thenext optimal action, when leaf nodes are evaluated with random rollouts. UsingBRIDGE, we show that the effective horizon-based bounds are more closelyreflective of the empirical performance of PPO and DQN than prior samplecomplexity bounds across four metrics. We also find that, unlike existingbounds, the effective horizon can predict the effects of using reward shapingor a pre-trained exploration policy. Our code and data are available athttps://github.com/cassidylaidlaw/effective-horizon

Bridging RL Theory and Practice with the Effective Horizon