English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, various intermediate layer distillation (ILD) objectives have beenshown to improve compression of BERT models via Knowledge Distillation (KD).However, a comprehensive evaluation of the objectives in both task-specific andtask-agnostic settings is lacking. To the best of our knowledge, this is thefirst work comprehensively evaluating distillation objectives in both settings.We show that attention transfer gives the best performance overall. We alsostudy the impact of layer choice when initializing the student from the teacherlayers, finding a significant impact on the performance in task-specificdistillation. For vanilla KD and hidden states transfer, initialisation withlower layers of the teacher gives a considerable improvement over higherlayers, especially on the task of QNLI (up to an absolute percentage change of17.8 in accuracy). Attention transfer behaves consistently under differentinitialisation settings. We release our code as an efficient transformer-basedmodel distillation framework for further studies.

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives