English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Although deep neural networks have enjoyed remarkable success across a widevariety of tasks, their ever-increasing size also imposes significant overheadon deployment. To compress these models, knowledge distillation was proposed totransfer knowledge from a cumbersome (teacher) network into a lightweight(student) network. However, guidance from a teacher does not always improve thegeneralization of students, especially when the size gap between student andteacher is large. Previous works argued that it was due to the high certaintyof the teacher, resulting in harder labels that were difficult to fit. Tosoften these labels, we present a pruning method termed Prediction UncertaintyEnlargement (PrUE) to simplify the teacher. Specifically, our method aims todecrease the teacher's certainty about data, thereby generating softpredictions for students. We empirically investigate the effectiveness of theproposed method with experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet.Results indicate that student networks trained with sparse teachers achievebetter performance. Besides, our method allows researchers to distill knowledgefrom deeper networks to improve students further. Our code is made public at:\url{https://github.com/wangshaopu/prue}.

PrUE: Distilling Knowledge from Sparse Teacher Networks