English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Model compression has been widely adopted to obtain light-weighted deepneural networks. Most prevalent methods, however, require fine-tuning withsufficient training data to ensure accuracy, which could be challenged byprivacy and security issues. As a compromise between privacy and performance,in this paper we investigate few shot network compression: given few samplesper class, how can we effectively compress the network with negligibleperformance drop? The core challenge of few shot network compression lies inhigh estimation errors from the original network during inference, since thecompressed network can easily over-fits on the few training instances. Theestimation errors could propagate and accumulate layer-wisely and finallydeteriorate the network output. To address the problem, we propose crossdistillation, a novel layer-wise knowledge distillation approach. Byinterweaving hidden layers of teacher and student network, layer-wiselyaccumulated estimation errors can be effectively reduced.The proposed methodoffers a general framework compatible with prevalent network compressiontechniques such as pruning. Extensive experiments on benchmark datasetsdemonstrate that cross distillation can significantly improve the studentnetwork's accuracy when only a few training instances are available.

Few Shot Network Compression via Cross Distillation