English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, knowledge distillation (KD) has shown great success in BERTcompression. Instead of only learning from the teacher's soft label as inconventional KD, researchers find that the rich information contained in thehidden layers of BERT is conducive to the student's performance. To betterexploit the hidden knowledge, a common practice is to force the student todeeply mimic the teacher's hidden states of all the tokens in a layer-wisemanner. In this paper, however, we observe that although distilling theteacher's hidden state knowledge (HSK) is helpful, the performance gain(marginal utility) diminishes quickly as more HSK is distilled. To understandthis effect, we conduct a series of analysis. Specifically, we divide the HSKof BERT into three dimensions, namely depth, length and width. We firstinvestigate a variety of strategies to extract crucial knowledge for eachsingle dimension and then jointly compress the three dimensions. In this way,we show that 1) the student's performance can be improved by extracting anddistilling the crucial HSK, and 2) using a tiny fraction of HSK can achieve thesame performance as extensive HSK distillation. Based on the second finding, wefurther propose an efficient KD paradigm to compress BERT, which does notrequire loading the teacher during the training of student. For two kinds ofstudent models and computing devices, the proposed KD paradigm gives rise totraining speedup of 2.7x ~ 3.4x.

Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation