English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision Transformers (ViTs) emerge to achieve impressive performance on manydata-abundant computer vision tasks by capturing long-range dependencies amonglocal features. However, under few-shot learning (FSL) settings on smalldatasets with only a few labeled data, ViT tends to overfit and suffers fromsevere performance degradation due to its absence of CNN-alike inductive bias.Previous works in FSL avoid such problem either through the help ofself-supervised auxiliary losses, or through the dextile uses of labelinformation under supervised settings. But the gap between self-supervised andsupervised few-shot Transformers is still unfilled. Inspired by recent advancesin self-supervised knowledge distillation and masked image modeling (MIM), wepropose a novel Supervised Masked Knowledge Distillation model (SMKD) forfew-shot Transformers which incorporates label information intoself-distillation frameworks. Compared with previous self-supervised methods,we allow intra-class knowledge distillation on both class and patch tokens, andintroduce the challenging task of masked patch tokens reconstruction acrossintra-class images. Experimental results on four few-shot classificationbenchmark datasets show that our method with simple design outperforms previousmethods by a large margin and achieves a new start-of-the-art. Detailedablation studies confirm the effectiveness of each component of our model. Codefor this paper is available here: https://github.com/HL-hanlin/SMKD.

Supervised Masked Knowledge Distillation for Few-Shot Transformers