English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep neural networks have achieved remarkable performance for artificialintelligence tasks. The success behind intelligent systems often relies onlarge-scale models with high computational complexity and storage costs. Theover-parameterized networks are often easy to optimize and can achieve betterperformance. However, it is challenging to deploy them over resource-limitededge-devices. Knowledge Distillation (KD) aims to optimize a lightweightnetwork from the perspective of over-parameterized training. The traditionaloffline KD transfers knowledge from a cumbersome teacher to a small and faststudent network. When a sizeable pre-trained teacher network is unavailable,online KD can improve a group of models by collaborative or mutual learning.Without needing extra models, Self-KD boosts the network itself using attachedauxiliary architectures. KD mainly involves knowledge extraction anddistillation strategies these two aspects. Beyond KD schemes, various KDalgorithms are widely used in practical applications, such as multi-teacher KD,cross-modal KD, attention-based KD, data-free KD and adversarial KD. This paperprovides a comprehensive KD survey, including knowledge categories,distillation schemes and algorithms, as well as some empirical studies onperformance comparison. Finally, we discuss the open challenges of existing KDworks and prospect the future directions.

Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation