English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Continual learning (CL) algorithms strive to acquire new knowledge whilepreserving prior information. However, this stability-plasticity trade-offremains a central challenge. This paper introduces a framework that dissectsthis trade-off, offering valuable insights into CL algorithms. TheReadout-Decomposition of Activation Change (RDAC) framework first addresses thestability-plasticity dilemma and its relation to catastrophic forgetting. Itrelates learning-induced activation changes in the range of prior readouts tothe degree of stability and changes in the null space to the degree ofplasticity. In deep non-linear networks tackling split-CIFAR-110 tasks, theframework clarifies the stability-plasticity trade-offs of the popularregularization algorithms Synaptic intelligence (SI), Elastic-weightconsolidation (EWC), and learning without Forgetting (LwF), and replay-basedalgorithms Gradient episodic memory (GEM), and data replay. GEM and data replaypreserved stability and plasticity, while SI, EWC, and LwF traded offplasticity for stability. The inability of the regularization algorithms tomaintain plasticity was linked to them restricting the change of activations inthe null space of the prior readout. Additionally, for one-hidden-layer linearneural networks, we derived a gradient decomposition algorithm to restrictactivation change only in the range of the prior readouts, to maintain highstability while not further sacrificing plasticity. Results demonstrate thatthe algorithm maintained stability without significant plasticity loss. TheRDAC framework informs the behavior of existing CL algorithms and paves the wayfor novel CL approaches. Finally, it sheds light on the connection betweenlearning-induced activation/representation changes and the stability-plasticitydilemma, also offering insights into representational drift in biologicalsystems.

Balancing stability and plasticity in continual learning: the readout-decomposition of activation change (RDAC) framework