English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Knowledge distillation is an effective approach to leverage a well-trainednetwork or an ensemble of them, named as the teacher, to guide the training ofa student network. The outputs from the teacher network are used as soft labelsfor supervising the training of a new network. Recent studies\citep{muller2019does,yuan2020revisiting} revealed an intriguing property ofthe soft labels that making labels soft serves as a good regularization to thestudent network. From the perspective of statistical learning, regularizationaims to reduce the variance, however how bias and variance change is not clearfor training with soft labels. In this paper, we investigate the bias-variancetradeoff brought by distillation with soft labels. Specifically, we observethat during training the bias-variance tradeoff varies sample-wisely. Further,under the same distillation temperature setting, we observe that thedistillation performance is negatively associated with the number of somespecific samples, which are named as regularization samples since these sampleslead to bias increasing and variance decreasing. Nevertheless, we empiricallyfind that completely filtering out regularization samples also deterioratesdistillation performance. Our discoveries inspired us to propose the novelweighted soft labels to help the network adaptively handle the sample-wisebias-variance tradeoff. Experiments on standard evaluation benchmarks validatethe effectiveness of our method. Our code is available at\url{https://github.com/bellymonster/Weighted-Soft-Label-Distillation}.

Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective