English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper provides a theoretical study of deep neural function approximationin reinforcement learning (RL) with the $\epsilon$-greedy exploration under theonline setting. This problem setting is motivated by the successful deepQ-networks (DQN) framework that falls in this regime. In this work, we providean initial attempt on theoretical understanding deep RL from the perspective offunction class and neural networks architectures (e.g., width and depth) beyondthe ``linear'' regime. To be specific, we focus on the value based algorithmwith the $\epsilon$-greedy exploration via deep (and two-layer) neural networksendowed by Besov (and Barron) function spaces, respectively, which aims atapproximating an $\alpha$-smooth Q-function in a $d$-dimensional feature space.We prove that, with $T$ episodes, scaling the width $m =\widetilde{\mathcal{O}}(T^{\frac{d}{2\alpha + d}})$ and the depth$L=\mathcal{O}(\log T)$ of the neural network for deep RL is sufficient forlearning with sublinear regret in Besov spaces. Moreover, for a two layerneural network endowed by the Barron space, scaling the width$\Omega(\sqrt{T})$ is sufficient. To achieve this, the key issue in ouranalysis is how to estimate the temporal difference error under deep neuralfunction approximation as the $\epsilon$-greedy exploration is not enough toensure ``optimism''. Our analysis reformulates the temporal difference error inan $L^2(\mathrm{d}\mu)$-integrable space over a certain averaged measure $\mu$,and transforms it to a generalization problem under the non-iid setting. Thismight have its own interest in RL theory for better understanding$\epsilon$-greedy exploration in deep RL.

Understanding Deep Neural Function Approximation in Reinforcement Learning via $\epsilon$-Greedy Exploration