English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is believed that in knowledge distillation (KD), the role of the teacheris to provide an estimate for the unknown Bayes conditional probabilitydistribution (BCPD) to be used in the student training process. Conventionally,this estimate is obtained by training the teacher using maximum log-likelihood(MLL) method. To improve this estimate for KD, in this paper we introduce theconcept of conditional mutual information (CMI) into the estimation of BCPD andpropose a novel estimator called the maximum CMI (MCMI) method. Specifically,in MCMI estimation, both the log-likelihood and CMI of the teacher aresimultaneously maximized when the teacher is trained. Through Eigen-CAM, it isfurther shown that maximizing the teacher's CMI value allows the teacher tocapture more contextual information in an image cluster. Via conducting athorough set of experiments, we show that by employing a teacher trained viaMCMI estimation rather than one trained via MLL estimation in variousstate-of-the-art KD frameworks, the student's classification accuracyconsistently increases, with the gain of up to 3.32\%. This suggests that theteacher's BCPD estimate provided by MCMI method is more accurate than thatprovided by MLL method. In addition, we show that such improvements in thestudent's accuracy are more drastic in zero-shot and few-shot settings.Notably, the student's accuracy increases with the gain of up to 5.72\% when5\% of the training samples are available to the student (few-shot), andincreases from 0\% to as high as 84\% for an omitted class (zero-shot). Thecode is available at \url{https://github.com/iclr2024mcmi/ICLRMCMI}.

Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information