English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We propose a novel algorithm for offline reinforcement learning called ValueIteration with Perturbed Rewards (VIPeR), which amalgamates the pessimismprinciple with random perturbations of the value function. Most current offlineRL algorithms explicitly construct statistical confidence regions to obtainpessimism via lower confidence bounds (LCB), which cannot easily scale tocomplex problems where a neural network is used to estimate the valuefunctions. Instead, VIPeR implicitly obtains pessimism by simply perturbing theoffline data multiple times with carefully-designed i.i.d. Gaussian noises tolearn an ensemble of estimated state-action {value functions} and actinggreedily with respect to the minimum of the ensemble. The estimatedstate-action values are obtained by fitting a parametric model (e.g., neuralnetworks) to the perturbed datasets using gradient descent. As a result, VIPeRonly needs $\mathcal{O}(1)$ time complexity for action selection, whileLCB-based algorithms require at least $\Omega(K^2)$, where $K$ is the totalnumber of trajectories in the offline data. We also propose a noveldata-splitting technique that helps remove a factor involving the log of thecovering number in our bound. We prove that VIPeR yields a provable uncertaintyquantifier with overparameterized neural networks and enjoys a bound onsub-optimality of $\tilde{\mathcal{O}}( { \kappa H^{5/2} \tilde{d}}/{\sqrt{K}})$, where $\tilde{d}$ is the effective dimension, $H$ is thehorizon length and $\kappa$ measures the distributional shift. We corroboratethe statistical and computational efficiency of VIPeR with an empiricalevaluation on a wide set of synthetic and real-world datasets. To the best ofour knowledge, VIPeR is the first algorithm for offline RL that is provablyefficient for general Markov decision processes (MDPs) with neural networkfunction approximation.

VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximation