English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider online reinforcement learning (RL) in episodic Markov decisionprocesses (MDPs) under the linear $q^\pi$-realizability assumption, where it isassumed that the action-values of all policies can be expressed as linearfunctions of state-action features. This class is known to be more general thanlinear MDPs, where the transition kernel and the reward function are assumed tobe linear functions of the feature vectors. As our first contribution, we showthat the difference between the two classes is the presence of states inlinearly $q^\pi$-realizable MDPs where for any policy, all the actions haveapproximately equal values, and skipping over these states by following anarbitrarily fixed policy in those states transforms the problem to a linearMDP. Based on this observation, we derive a novel (computationally inefficient)learning algorithm for linearly $q^\pi$-realizable MDPs that simultaneouslylearns what states should be skipped over and runs another learning algorithmon the linear MDP hidden in the problem. The method returns an$\epsilon$-optimal policy after $\text{polylog}(H, d)/\epsilon^2$ interactionswith the MDP, where $H$ is the time horizon and $d$ is the dimension of thefeature vectors, giving the first polynomial-sample-complexity online RLalgorithm for this setting. The results are proved for the misspecified case,where the sample complexity is shown to degrade gracefully with themisspecification error.

Online RL in Linearly $q^\pi$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore