English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the offline reinforcement learning (offline RL) problem, where thegoal is to learn a reward-maximizing policy in an unknown Markov DecisionProcess (MDP) using the data coming from a policy $\mu$. In particular, weconsider the sample complexity problems of offline RL for finite-horizon MDPs.Prior works study this problem based on different data-coverage assumptions,and their learning guarantees are expressed by the covering coefficients whichlack the explicit characterization of system quantities. In this work, weanalyze the Adaptive Pessimistic Value Iteration (APVI) algorithm and derivethe suboptimality upper bound that nearly matches \[O\left(\sum_{h=1}^H\sum_{s_h,a_h}d^{\pi^\star}_h(s_h,a_h)\sqrt{\frac{\mathrm{Var}_{P_{s_h,a_h}}{(V^\star_{h+1}+r_h)}}{d^\mu_h(s_h,a_h)}}\sqrt{\frac{1}{n}}\right).\] In complementary, we also prove a per-instance information-theoretical lowerbound under the weak assumption that $d^\mu_h(s_h,a_h)>0$ if$d^{\pi^\star}_h(s_h,a_h)>0$. Different from the previous minimax lower bounds,the per-instance lower bound (via local minimaxity) is a much strongercriterion as it applies to individual instances separately. Here $\pi^\star$ isa optimal policy, $\mu$ is the behavior policy and $d_h^\mu$ is the marginalstate-action probability. We call the above equation the intrinsic offlinereinforcement learning bound since it directly implies all the existing optimalresults: minimax rate under uniform data-coverage assumption, horizon-freesetting, single policy concentrability, and the tight problem-dependentresults. Later, we extend the result to the assumption-free regime (where wemake no assumption on $ \mu$) and obtain the assumption-free intrinsic bound.Due to its generic form, we believe the intrinsic bound could help illuminatewhat makes a specific problem hard and reveal the fundamental challenges inoffline RL.

Towards Instance-Optimal Offline Reinforcement Learning with Pessimism