English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we investigate the continual learning of Vision Transformers(ViT) for the challenging exemplar-free scenario, with special focus on how toefficiently distill the knowledge of its crucial self-attention mechanism(SAM). Our work takes an initial step towards a surgical investigation of SAMfor designing coherent continual learning methods in ViTs. We first carry outan evaluation of established continual learning regularization techniques. Wethen examine the effect of regularization when applied to two key enablers ofSAM: (a) the contextualized embedding layers, for their ability to capturewell-scaled representations with respect to the values, and (b) the prescaledattention maps, for carrying value-independent global contextual information.We depict the perks of each distilling strategy on two image recognitionbenchmarks (CIFAR100 and ImageNet-32) -- while (a) leads to a better overallaccuracy, (b) helps enhance the rigidity by maintaining competitiveperformances. Furthermore, we identify the limitation imposed by the symmetricnature of regularization losses. To alleviate this, we propose an asymmetricvariant and apply it to the pooled output distillation (POD) loss adapted forViTs. Our experiments confirm that introducing asymmetry to POD boosts itsplasticity while retaining stability across (a) and (b). Moreover, weacknowledge low forgetting measures for all the compared methods, indicatingthat ViTs might be naturally inclined continual learner

Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization