English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

With the increasing popularity of deep learning on edge devices, compressinglarge neural networks to meet the hardware requirements of resource-constraineddevices became a significant research direction. Numerous compressionmethodologies are currently being used to reduce the memory sizes and energyconsumption of neural networks. Knowledge distillation (KD) is among suchmethodologies and it functions by using data samples to transfer the knowledgecaptured by a large model (teacher) to a smaller one(student). However, due tovarious reasons, the original training data might not be accessible at thecompression stage. Therefore, data-free model compression is an ongoingresearch problem that has been addressed by various works. In this paper, wepoint out that catastrophic forgetting is a problem that can potentially beobserved in existing data-free distillation methods. Moreover, the samplegeneration strategies in some of these methods could result in a mismatchbetween the synthetic and real data distributions. To prevent such problems, wepropose a data-free KD framework that maintains a dynamic collection ofgenerated samples over time. Additionally, we add the constraint of matchingthe real data distribution in sample generation strategies that target maximuminformation gain. Our experiments demonstrate that we can improve the accuracyof the student models obtained via KD when compared with state-of-the-artapproaches on the SVHN, Fashion MNIST and CIFAR100 datasets.

Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data