English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Reinforcement learning is generally difficult for partially observable Markovdecision processes (POMDPs), which occurs when the agent's observation ispartial or noisy. To seek good performance in POMDPs, one strategy is to endowthe agent with a finite memory, whose update is governed by the policy.However, policy optimization is non-convex in that case and can lead to poortraining performance for random initialization. The performance can beempirically improved by constraining the memory architecture, then sacrificingoptimality to facilitate training. Here we study this trade-off in atwo-hypothesis testing problem, akin to the two-arm bandit problem. We comparetwo extreme cases: (i) the random access memory where any transitions between$M$ memory states are allowed and (ii) a fixed memory where the agent canaccess its last $m$ actions and rewards. For (i), the probability $q$ to playthe worst arm is known to be exponentially small in $M$ for the optimal policy.Our main result is to show that similar performance can be reached for (ii) aswell, despite the simplicity of the memory architecture: using a conjecture onGray-ordered binary necklaces, we find policies for which $q$ is exponentiallysmall in $2^m$, i.e. $q\sim\alpha^{2^m}$ with $\alpha < 1$. In addition, weobserve empirically that training from random initialization leads to very poorresults for (i), and significantly better results for (ii) thanks to theconstraints on the memory architecture.

How memory architecture affects learning in a simple POMDP: the two-hypothesis testing problem