English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD) has been widely-used to improve the test accuracyof a ``student'' network by training the student to mimic soft probabilities ofa trained "teacher" network. Yet, it has been shown in recent work that,despite being trained to fit the teacher's probabilities, the student not onlysignificantly deviates from these probabilities, but also performs even betterthan the teacher. Our work aims to reconcile this seemingly paradoxicalobservation by characterizing the precise nature of the student-teacherdeviations, and by arguing how they can co-occur with better generalization.First, through experiments on image and language data, we identify that thesedeviations correspond to the student systematically exaggerating the confidencelevels of the teacher. Next, we theoretically and empirically establish in somesimple settings that KD also exaggerates the implicit bias of gradient descentin converging faster along the top eigendirections of the data. Finally, wedemonstrate that this exaggerated bias effect can simultaneously result in both(a) the exaggeration of confidence and (b) the improved generalization of thestudent, thus offering a resolution to the apparent paradox. Our analysisbrings existing theory and practice closer by considering the role of gradientdescent in KD and by demonstrating the exaggerated bias effect in boththeoretical and empirical settings.

On student-teacher deviations in distillation: does it pay to disobey?