English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In class incremental learning (CIL) setting, groups of classes are introducedto a model in each learning phase. The goal is to learn a unified modelperformant on all the classes observed so far. Given the recent popularity ofVision Transformers (ViTs) in conventional classification settings, aninteresting question is to study their continual learning behaviour. In thiswork, we develop a Debiased Dual Distilled Transformer for CIL dubbed$\textrm{D}^3\textrm{Former}$. The proposed model leverages a hybrid nested ViTdesign to ensure data efficiency and scalability to small as well as largedatasets. In contrast to a recent ViT based CIL approach, our$\textrm{D}^3\textrm{Former}$ does not dynamically expand its architecture whennew tasks are learned and remains suitable for a large number of incrementaltasks. The improved CIL behaviour of $\textrm{D}^3\textrm{Former}$ owes to twofundamental changes to the ViT design. First, we treat the incremental learningas a long-tail classification problem where the majority samples from newclasses vastly outnumber the limited exemplars available for old classes. Toavoid the bias against the minority old classes, we propose to dynamicallyadjust logits to emphasize on retaining the representations relevant to oldtasks. Second, we propose to preserve the configuration of spatial attentionmaps as the learning progresses across tasks. This helps in reducingcatastrophic forgetting by constraining the model to retain the attention onthe most discriminative regions. $\textrm{D}^3\textrm{Former}$ obtainsfavorable results on incremental versions of CIFAR-100, MNIST, SVHN, andImageNet datasets. Code is available at https://tinyurl.com/d3former

D3Former: Debiased Dual Distilled Transformer for Incremental Learning