English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

One of the key behavioral characteristics used in neuroscience to determinewhether the subject of study -- be it a rodent or a human -- exhibitsmodel-based learning is effective adaptation to local changes in theenvironment, a particular form of adaptivity that is the focus of this work. Inreinforcement learning, however, recent work has shown that modern deepmodel-based reinforcement-learning (MBRL) methods adapt poorly to localenvironment changes. An explanation for this mismatch is that MBRL methods aretypically designed with sample-efficiency on a single task in mind and therequirements for effective adaptation are substantially higher, both in termsof the learned world model and the planning routine. One particularlychallenging requirement is that the learned world model has to be sufficientlyaccurate throughout relevant parts of the state-space. This is challenging fordeep-learning-based world models due to catastrophic forgetting. And while areplay buffer can mitigate the effects of catastrophic forgetting, thetraditional first-in-first-out replay buffer precludes effective adaptation dueto maintaining stale data. In this work, we show that a conceptually simplevariation of this traditional replay buffer is able to overcome thislimitation. By removing only samples from the buffer from the localneighbourhood of the newly observed samples, deep world models can be builtthat maintain their accuracy across the state-space, while also being able toeffectively adapt to local changes in the reward function. We demonstrate thisby applying our replay-buffer variation to a deep version of the classical Dynamethod, as well as to recent methods such as PlaNet and DreamerV2,demonstrating that deep model-based methods can adapt effectively as well tolocal changes in the environment.

Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learning