English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Data-Free Knowledge Distillation (KD) allows knowledge transfer from atrained neural network (teacher) to a more compact one (student) in the absenceof original training data. Existing works use a validation set to monitor theaccuracy of the student over real data and report the highest performancethroughout the entire process. However, validation data may not be available atdistillation time either, making it infeasible to record the student snapshotthat achieved the peak accuracy. Therefore, a practical data-free KD methodshould be robust and ideally provide monotonically increasing student accuracyduring distillation. This is challenging because the student experiencesknowledge degradation due to the distribution shift of the synthetic data. Astraightforward approach to overcome this issue is to store and rehearse thegenerated samples periodically, which increases the memory footprint andcreates privacy concerns. We propose to model the distribution of thepreviously observed synthetic samples with a generative network. In particular,we design a Variational Autoencoder (VAE) with a training objective that iscustomized to learn the synthetic data representations optimally. The studentis rehearsed by the generative pseudo replay technique, with samples producedby the VAE. Hence knowledge degradation can be prevented without storing anysamples. Experiments on image classification benchmarks show that our methodoptimizes the expected value of the distilled model accuracy while eliminatingthe large memory overhead incurred by the sample-storing methods.

Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay