English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Continual learning algorithms strive to acquire new knowledge whilepreserving prior information. Often, these algorithms emphasise stability andrestrict network updates upon learning new tasks. In many cases, suchrestrictions come at a cost to the model's plasticity, i.e. the model's abilityto adapt to the requirements of a new task. But is all change detrimental?Here, we approach this question by proposing that activation spaces in neuralnetworks can be decomposed into two subspaces: a readout range in which changeaffects prior tasks and a null space in which change does not alter priorperformance. Based on experiments with this novel technique, we show that,indeed, not all activation change is associated with forgetting. Instead, onlychange in the subspace visible to the readout of a task can lead to decreasedstability, while restricting change outside of this subspace is associated onlywith a loss of plasticity. Analysing various commonly used algorithms, we showthat regularisation-based techniques do not fully disentangle the two spacesand, as a result, restrict plasticity more than need be. We expand our resultsby investigating a linear model in which we can manipulate learning in the twosubspaces directly and thus causally link activation changes to stability andplasticity. For hierarchical, nonlinear cases, we present an approximation thatenables us to estimate functionally relevant subspaces at every layer of a deepnonlinear network, corroborating our previous insights. Together, this workprovides novel means to derive insights into the mechanisms behind stabilityand plasticity in continual learning and may serve as a diagnostic tool toguide developments of future continual learning algorithms that stabiliseinference while allowing maximal space for learning.

Keep Moving: identifying task-relevant subspaces to maximise plasticity for newly learned tasks