English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD), transferring knowledge from a cumbersome teachermodel to a lightweight student model, has been investigated to design efficientneural architectures. Generally, the objective function of KD is theKullback-Leibler (KL) divergence loss between the softened probabilitydistributions of the teacher model and the student model with the temperaturescaling hyperparameter tau. Despite its widespread use, few studies havediscussed the influence of such softening on generalization. Here, wetheoretically show that the KL divergence loss focuses on the logit matchingwhen tau increases and the label matching when tau goes to 0 and empiricallyshow that the logit matching is positively correlated to performanceimprovement in general. From this observation, we consider an intuitive KD lossfunction, the mean squared error (MSE) between the logit vectors, so that thestudent model can directly learn the logit of the teacher model. The MSE lossoutperforms the KL divergence loss, explained by the difference in thepenultimate layer representations between the two losses. Furthermore, we showthat sequential distillation can improve performance and that KD, particularlywhen using the KL divergence loss with small tau, mitigates the label noise.The code to reproduce the experiments is publicly available online athttps://github.com/jhoon-oh/kd_data/.

Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation