English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Data-free Knowledge Distillation (DFKD) has gained popularity recently, withthe fundamental idea of carrying out knowledge transfer from a Teacher neuralnetwork to a Student neural network in the absence of training data. However,in the Adversarial DFKD framework, the student network's accuracy, suffers dueto the non-stationary distribution of the pseudo-samples under multiplegenerator updates. To this end, at every generator update, we aim to maintainthe student's performance on previously encountered examples while acquiringknowledge from samples of the current distribution. Thus, we propose ameta-learning inspired framework by treating the task of Knowledge-Acquisition(learning from newly generated samples) and Knowledge-Retention (retainingknowledge on previously met samples) as meta-train and meta-test, respectively.Hence, we dub our method as Learning to Retain while Acquiring. Moreover, weidentify an implicit aligning factor between the Knowledge-Retention andKnowledge-Acquisition tasks indicating that the proposed student updatestrategy enforces a common gradient direction for both tasks, alleviatinginterference between the two objectives. Finally, we support our hypothesis byexhibiting extensive evaluation and comparison of our method with prior arts onmultiple datasets.

Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation