English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

With the success of pretraining techniques in representation learning, anumber of continual learning methods based on pretrained models have beenproposed. Some of these methods design continual learning mechanisms on thepre-trained representations and only allow minimum updates or even no updatesof the backbone models during the training of continual learning. In thispaper, we question whether the complexity of these models is needed to achievegood performance by comparing them to a simple baseline that we designed. Weargue that the pretrained feature extractor itself can be strong enough toachieve a competitive or even better continual learning performance onSplit-CIFAR100 and CoRe 50 benchmarks. To validate this, we conduct a verysimple baseline that 1) use the frozen pretrained model to extract imagefeatures for every class encountered during the continual learning stage andcompute their corresponding mean features on training data, and 2) predict theclass of the input based on the nearest neighbor distance between test samplesand mean features of the classes; i.e., Nearest Mean Classifier (NMC). Thisbaseline is single-headed, exemplar-free, and can be task-free (by updating themeans continually). This baseline achieved 88.53% on 10-Split-CIFAR-100,surpassing most state-of-the-art continual learning methods that are allinitialized using the same pretrained transformer model. We hope our baselinemay encourage future progress in designing learning systems that cancontinually add quality to the learning representations even if they startedfrom some pretrained weights.

A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning