English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Sequential training from task to task is becoming one of the major objects indeep learning applications such as continual learning and transfer learning.Nevertheless, it remains unclear under what conditions the trained model'sperformance improves or deteriorates. To deepen our understanding of sequentialtraining, this study provides a theoretical analysis of generalizationperformance in a solvable case of continual learning. We consider neuralnetworks in the neural tangent kernel (NTK) regime that continually learntarget functions from task to task, and investigate the generalization by usingan established statistical mechanical analysis of kernel ridge-less regression.We first show characteristic transitions from positive to negative transfer.More similar targets above a specific critical value can achieve positiveknowledge transfer for the subsequent task while catastrophic forgetting occurseven with very similar targets. Next, we investigate a variant of continuallearning which supposes the same target function in multiple tasks. Even forthe same target, the trained model shows some transfer and forgetting dependingon the sample size of each task. We can guarantee that the generalization errormonotonically decreases from task to task for equal sample sizes whileunbalanced sample sizes deteriorate the generalization. We respectively referto these improvement and deterioration as self-knowledge transfer andforgetting, and empirically confirm them in realistic training of deep neuralnetworks as well.

Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting