English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A primary focus area in continual learning research is alleviating the"catastrophic forgetting" problem in neural networks by designing newalgorithms that are more robust to the distribution shifts. While the recentprogress in continual learning literature is encouraging, our understanding ofwhat properties of neural networks contribute to catastrophic forgetting isstill limited. To address this, instead of focusing on continual learningalgorithms, in this work, we focus on the model itself and study the impact of"width" of the neural network architecture on catastrophic forgetting, and showthat width has a surprisingly significant effect on forgetting. To explain thiseffect, we study the learning dynamics of the network from various perspectivessuch as gradient orthogonality, sparsity, and lazy training regime. We providepotential explanations that are consistent with the empirical results acrossdifferent architectures and continual learning benchmarks.

Wide Neural Networks Forget Less Catastrophically