English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge Distillation (KD) uses the teacher's prediction logits as softlabels to guide the student, while self-KD does not need a real teacher torequire the soft labels. This work unifies the formulations of the two tasks bydecomposing and reorganizing the generic KD loss into a Normalized KD (NKD)loss and customized soft labels for both target class (image's category) andnon-target classes named Universal Self-Knowledge Distillation (USKD). Wedecompose the KD loss and find the non-target loss from it forces the student'snon-target logits to match the teacher's, but the sum of the two non-targetlogits is different, preventing them from being identical. NKD normalizes thenon-target logits to equalize their sum. It can be generally used for KD andself-KD to better use the soft labels for distillation loss. USKD generatescustomized soft labels for both target and non-target classes without ateacher. It smooths the target logit of the student as the soft target labeland uses the rank of the intermediate feature to generate the soft non-targetlabels with Zipf's law. For KD with teachers, our NKD achieves state-of-the-artperformance on CIFAR-100 and ImageNet datasets, boosting the ImageNet Top-1accuracy of ResNet18 from 69.90% to 71.96% with a ResNet-34 teacher. Forself-KD without teachers, USKD is the first self-KD method that can beeffectively applied to both CNN and ViT models with negligible additional timeand memory cost, resulting in new state-of-the-art results, such as 1.17% and0.55% accuracy gains on ImageNet for MobileNet and DeiT-Tiny, respectively. Ourcodes are available at https://github.com/yzd-v/cls_KD.

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels