English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation is a generalized logits matching technique for modelcompression. Their equivalence is previously established on the condition of$\textit{infinity temperature}$ and $\textit{zero-mean normalization}$. In thispaper, we prove that with only $\textit{infinity temperature}$, the effect ofknowledge distillation equals to logits matching with an extra regularization.Furthermore, we reveal that an additional weaker condition --$\textit{equal-mean initialization}$ rather than the $\textit{zero-meannormalization}$ already suffices to set up the equivalence. The key to ourproof is we realize that in neural networks with the cross-entropy loss andsoftmax activation, the mean of back-propagated gradient on logits always keepszero.

Exploring the Connection between Knowledge Distillation and Logits Matching