English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Offline reinforcement learning (RL), which refers to decision-making from apreviously-collected dataset of interactions, has received significantattention over the past years. Much effort has focused on improving offline RLpracticality by addressing the prevalent issue of partial data coverage throughvarious forms of conservative policy learning. While the majority of algorithmsdo not have finite-sample guarantees, several provable conservative offline RLalgorithms are designed and analyzed within the single-policy concentrabilityframework that handles partial coverage. Yet, in the nonlinear functionapproximation setting where confidence intervals are difficult to obtain,existing provable algorithms suffer from computational intractability,prohibitively strong assumptions, and suboptimal statistical rates. In thispaper, we leverage the marginalized importance sampling (MIS) formulation of RLand present the first set of offline RL algorithms that are statisticallyoptimal and practical under general function approximation and single-policyconcentrability, bypassing the need for uncertainty quantification. We identifythat the key to successfully solving the sample-based approximation of the MISproblem is ensuring that certain occupancy validity constraints are nearlysatisfied. We enforce these constraints by a novel application of the augmentedLagrangian method and prove the following result: with the MIS formulation,augmented Lagrangian is enough for statistically optimal offline RL. In starkcontrast to prior algorithms that induce additional conservatism throughmethods such as behavior regularization, our approach provably eliminates thisneed and reinterprets regularizers as "enforcers of occupancy validity" than"promoters of conservatism."

Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian