English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper studies compressing pre-trained language models, like BERT (Devlinet al.,2019), via teacher-student knowledge distillation. Previous worksusually force the student model to strictly mimic the smoothed labels predictedby the teacher BERT. As an alternative, we propose a new method for BERTdistillation, i.e., asking the teacher to generate smoothed word ids, ratherthan labels, for teaching the student model in knowledge distillation. We callthis kind of methodTextSmoothing. Practically, we use the softmax prediction ofthe Masked Language Model(MLM) in BERT to generate word distributions for giventexts and smooth those input texts using that predicted soft word ids. Weassume that both the smoothed labels and the smoothed texts can implicitlyaugment the input corpus, while text smoothing is intuitively more efficientsince it can generate more instances in one neural network forwardstep.Experimental results on GLUE and SQuAD demonstrate that our solution canachieve competitive results compared with existing BERT distillation methods.

Distilling Knowledge from Pre-trained Language Models via Text Smoothing