English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider the problem of solving robust Markov decision process (MDP),which involves a set of discounted, finite state, finite action space MDPs withuncertain transition kernels. The goal of planning is to find a robust policythat optimizes the worst-case values against the transition uncertainties, andthus encompasses the standard MDP planning as a special case. For$(\mathbf{s},\mathbf{a})$-rectangular uncertainty sets, we establish severalstructural observations on the robust objective, which facilitates thedevelopment of a policy-based first-order method, namely the robust policymirror descent (RPMD). An $\mathcal{O}(\log(1/\epsilon))$ iteration complexityfor finding an $\epsilon$-optimal policy is established with linearlyincreasing stepsizes. We further develop a stochastic variant of the robustpolicy mirror descent method, named SRPMD, when the first-order information isonly available through online interactions with the nominal environment. Weshow that the optimality gap converges linearly up to the noise level, andconsequently establish an $\tilde{\mathcal{O}}(1/\epsilon^2)$ sample complexityby developing a temporal difference learning method for policy evaluation. Bothiteration and sample complexities are also discussed for RPMD with a constantstepsize. To the best of our knowledge, all the aforementioned results appearto be new for policy-based first-order methods applied to the robust MDPproblem.

First-order Policy Optimization for Robust Markov Decision Process