English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Knowledge distillation (KD) has been a ubiquitous method for modelcompression to strengthen the capability of a lightweight model with thetransferred knowledge from the teacher. In particular, KD has been employed inquantization-aware training (QAT) of Transformer encoders like BERT to improvethe accuracy of the student model with the reduced-precision weight parameters.However, little is understood about which of the various KD approaches bestfits the QAT of Transformers. In this work, we provide an in-depth analysis ofthe mechanism of KD on attention recovery of quantized large Transformers. Inparticular, we reveal that the previously adopted MSE loss on the attentionscore is insufficient for recovering the self-attention information. Therefore,we propose two KD methods; attention-map and attention-output losses.Furthermore, we explore the unification of both losses to addresstask-dependent preference between attention-map and output losses. Theexperimental results on various Transformer encoder models demonstrate that theproposed KD methods achieve state-of-the-art accuracy for QAT with sub-2-bitweight quantization.

Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders