English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we study distributionally robust offline reinforcementlearning (robust offline RL), which seeks to find an optimal policy purely froman offline dataset that can perform well in perturbed environments. Inspecific, we propose a generic algorithm framework called Doubly PessimisticModel-based Policy Optimization ($P^2MPO$), which features a novel combinationof a flexible model estimation subroutine and a doubly pessimistic policyoptimization step. Notably, the double pessimism principle is crucial toovercome the distributional shifts incurred by (i) the mismatch between thebehavior policy and the target policies; and (ii) the perturbation of thenominal model. Under certain accuracy conditions on the model estimationsubroutine, we prove that $P^2MPO$ is sample-efficient with robust partialcoverage data, which only requires the offline data to have good coverage ofthe distributions induced by the optimal robust policy and the perturbed modelsaround the nominal model.  By tailoring specific model estimation subroutines for concrete examples ofRMDPs, including tabular RMDPs, factored RMDPs, kernel and neural RMDPs, weprove that $P^2MPO$ enjoys a $\tilde{\mathcal{O}}(n^{-1/2})$ convergence rate,where $n$ is the dataset size. We highlight that all these examples, excepttabular RMDPs, are first identified and proven tractable by this work.Furthermore, we continue our study of robust offline RL in the robust Markovgames (RMGs). By extending the double pessimism principle identified forsingle-agent RMDPs, we propose another algorithm framework that can efficientlyfind the robust Nash equilibria among players using only robust unilateral(partial) coverage data. To our best knowledge, this work proposes the firstgeneral learning principle -- double pessimism -- for robust offline RL andshows that it is provably efficient with general function approximation.

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage