English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD) is a successful approach for deep neural networkacceleration, with which a compact network (student) is trained by mimickingthe softmax output of a pre-trained high-capacity network (teacher). Intradition, KD usually relies on access to the training samples and theparameters of the white-box teacher to acquire the transferred knowledge.However, these prerequisites are not always realistic due to storage costs orprivacy issues in real-world applications. Here we propose the concept ofdecision-based black-box (DB3) knowledge distillation, with which the studentis trained by distilling the knowledge from a black-box teacher (parameters arenot accessible) that only returns classes rather than softmax outputs. We startwith the scenario when the training set is accessible. We represent a sample'srobustness against other classes by computing its distances to the teacher'sdecision boundaries and use it to construct the soft label for each trainingsample. After that, the student can be trained via standard KD. We then extendthis approach to a more challenging scenario in which even accessing thetraining data is not feasible. We propose to generate pseudo samplesdistinguished by the teacher's decision boundaries to the largest extent andconstruct soft labels for them, which are used as the transfer set. We evaluateour approaches on various benchmark networks and datasets and experimentresults demonstrate their effectiveness. Codes are available at:https://github.com/zwang84/zsdb3kd.

Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model