English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Knowledge distillation aims at obtaining a compact and effective model bylearning the mapping function from a much larger one. Due to the limitedcapacity of the student, the student would underfit the teacher. Therefore,student performance would unexpectedly drop when distilling from an oversizedteacher, termed the capacity gap problem. We investigate this problem by studythe gap of confidence between teacher and student. We find that the magnitudeof confidence is not necessary for knowledge distillation and could harm thestudent performance if the student are forced to learn confidence. We proposeSpherical Knowledge Distillation to eliminate this gap explicitly, which easesthe underfitting problem. We find this novel knowledge representation canimprove compact models with much larger teachers and is robust to temperature.We conducted experiments on both CIFAR100 and ImageNet, and achieve significantimprovement. Specifically, we train ResNet18 to 73.0 accuracy, which is asubstantial improvement over previous SOTA and is on par with resnet34 almosttwice the student size. The implementation has been shared athttps://github.com/forjiuzhou/Spherical-Knowledge-Distillation.

Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation