English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In the maximum state entropy exploration framework, an agent interacts with areward-free environment to learn a policy that maximizes the entropy of theexpected state visitations it is inducing. Hazan et al. (2019) noted that theclass of Markovian stochastic policies is sufficient for the maximum stateentropy objective, and exploiting non-Markovianity is generally consideredpointless in this setting. In this paper, we argue that non-Markovianity isinstead paramount for maximum state entropy exploration in a finite-sampleregime. Especially, we recast the objective to target the expected entropy ofthe induced state visitations in a single trial. Then, we show that the classof non-Markovian deterministic policies is sufficient for the introducedobjective, while Markovian policies suffer non-zero regret in general. However,we prove that the problem of finding an optimal non-Markovian policy isNP-hard. Despite this negative result, we discuss avenues to address theproblem in a tractable way and how non-Markovian exploration could benefit thesample efficiency of online reinforcement learning in future works.

The Importance of Non-Markovianity in Maximum State Entropy Exploration