English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation, which involves extracting the "dark knowledge" from ateacher network to guide the learning of a student network, has emerged as anessential technique for model compression and transfer learning. Unlikeprevious works that focus on the accuracy of student network, here we study alittle-explored but important question, i.e., knowledge distillationefficiency. Our goal is to achieve a performance comparable to conventionalknowledge distillation with a lower computation cost during training. We showthat the UNcertainty-aware mIXup (UNIX) can serve as a clean yet effectivesolution. The uncertainty sampling strategy is used to evaluate theinformativeness of each training sample. Adaptive mixup is applied to uncertainsamples to compact knowledge. We further show that the redundancy ofconventional knowledge distillation lies in the excessive learning of easysamples. By combining uncertainty and mixup, our approach reduces theredundancy and makes better use of each query to the teacher network. Wevalidate our approach on CIFAR100 and ImageNet. Notably, with only 79%computation cost, we outperform conventional knowledge distillation on CIFAR100and achieve a comparable result on ImageNet.

Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup