English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD) has been widely used for model compression andknowledge transfer. Typically, a big teacher model trained on sufficient datatransfers knowledge to a small student model. However, despite the success ofKD, little effort has been made to study whether KD leaks the training data ofthe teacher model. In this paper, we experimentally reveal that KD suffers fromthe risk of privacy leakage. To alleviate this issue, we propose a novelknowledge distillation method, swing distillation, which can effectivelyprotect the private information of the teacher model from flowing to thestudent model. In our framework, the temperature coefficient is dynamically andadaptively adjusted according to the degree of private information contained inthe data, rather than a predefined constant hyperparameter. It assignsdifferent temperatures to tokens according to the likelihood that a token in aposition contains private information. In addition, we inject noise into softtargets provided to the student model, in order to avoid unshielded knowledgetransfer. Experiments on multiple datasets and tasks demonstrate that theproposed swing distillation can significantly reduce (by over 80% in terms ofcanary exposure) the risk of privacy leakage in comparison to KD withcompetitive or better performance. Furthermore, swing distillation is robustagainst the increasing privacy budget.

Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework