English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Continual learning (CL) aims to incrementally learn different tasks (such asclassification) in a non-stationary data stream without forgetting old ones.Most CL works focus on tackling catastrophic forgetting under alearning-from-scratch paradigm. However, with the increasing prominence offoundation models, pre-trained models equipped with informative representationshave become available for various downstream requirements. Several CL methodsbased on pre-trained models have been explored, either utilizing pre-extractedfeatures directly (which makes bridging distribution gaps challenging) orincorporating adaptors (which may be subject to forgetting). In this paper, wepropose a concise and effective approach for CL with pre-trained models. Giventhat forgetting occurs during parameter updating, we contemplate an alternativeapproach that exploits training-free random projectors and class-prototypeaccumulation, which thus bypasses the issue. Specifically, we inject a frozenRandom Projection layer with nonlinear activation between the pre-trainedmodel's feature representations and output head, which captures interactionsbetween features with expanded dimensionality, providing enhanced linearseparability for class-prototype-based CL. We also demonstrate the importanceof decorrelating the class-prototypes to reduce the distribution disparity whenusing pre-trained representations. These techniques prove to be effective andcircumvent the problem of forgetting for both class- and domain-incrementalcontinual learning. Compared to previous methods applied to pre-trainedViT-B/16 models, we reduce final error rates by between 20% and 62% on sevenclass-incremental benchmarks, despite not using any rehearsal memory. Weconclude that the full potential of pre-trained models for simple, effective,and fast CL has not hitherto been fully tapped. Code is atgithub.com/RanPAC/RanPAC.

RanPAC: Random Projections and Pre-trained Models for Continual Learning