English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study a novel setting in offline reinforcement learning (RL) where anumber of distributed machines jointly cooperate to solve the problem but onlyone single round of communication is allowed and there is a budget constrainton the total number of information (in terms of bits) that each machine cansend out. For value function prediction in contextual bandits, and bothepisodic and non-episodic MDPs, we establish information-theoretic lower boundson the minimax risk for distributed statistical estimators; this reveals theminimum amount of communication required by any offline RL algorithms.Specifically, for contextual bandits, we show that the number of bits mustscale at least as $\Omega(AC)$ to match the centralised minimax optimal rate,where $A$ is the number of actions and $C$ is the context dimension; meanwhile,we reach similar results in the MDP settings. Furthermore, we develop learningalgorithms based on least-squares estimates and Monte-Carlo return estimatesand provide a sharp analysis showing that they can achieve optimal risk up tologarithmic factors. Additionally, we also show that temporal difference isunable to efficiently utilise information from all available devices under thesingle-round communication setting due to the initial bias of this method. Toour best knowledge, this paper presents the first minimax lower bounds fordistributed offline RL problems.

Settling the Communication Complexity for Distributed Offline Reinforcement Learning