English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While it is established that neural networks suffer from catastrophicforgetting ``at the output level'', it is debated whether this is also the caseat the level of representations. Some studies ascribe a certain level of innaterobustness to representations, that they only forget minimally and no criticalinformation, while others claim that representations are also severely affectedby forgetting. To settle this debate, we first discuss how this apparentdisagreement might stem from the coexistence of two phenomena that affect thequality of continually learned representations: knowledge accumulation andfeature forgetting. We then show that, even though it is true that featureforgetting can be small in absolute terms, newly learned information isforgotten just as catastrophically at the level of representations as it is atthe output level. Next we show that this feature forgetting is problematic asit substantially slows down knowledge accumulation. We further show thatrepresentations that are continually learned through both supervised andself-supervised learning suffer from feature forgetting. Finally, we study howfeature forgetting and knowledge accumulation are affected by different typesof continual learning methods.

Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting