English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Knowledge Distillation (KD) transfers knowledge from a large teacher model toa smaller student model by minimizing the divergence between their outputdistributions, typically using forward Kullback-Leibler divergence (FKLD) orreverse KLD (RKLD). It has become an effective training paradigm due to thebroader supervision information provided by the teacher distribution comparedto one-hot labels. We identify that the core challenge in KD lies in balancingtwo mode-concentration effects: the \textbf{\textit{Hardness-Concentration}}effect, which refers to focusing on modes with large errors, and the\textbf{\textit{Confidence-Concentration}} effect, which refers to focusing onmodes with high student confidence. Through an analysis of how probabilitiesare reassigned during gradient updates, we observe that these two effects areentangled in FKLD and RKLD, but in extreme forms. Specifically, both are tooweak in FKLD, causing the student to fail to concentrate on the target class.In contrast, both are too strong in RKLD, causing the student to overlyemphasize the target class while ignoring the broader distributionalinformation from the teacher. To address this imbalance, we propose ABKD, ageneric framework with $\alpha$-$\beta$-divergence. Our theoretical resultsshow that ABKD offers a smooth interpolation between FKLD and RKLD, achievingan effective trade-off between these effects. Extensive experiments on 17language/vision datasets with 12 teacher-student settings confirm its efficacy.The code is available at https://github.com/ghwang-s/abkd.

ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence