English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretrained language models (PLMs) such as BERT adopt a training paradigmwhich first pretrain the model in general data and then finetune the model ontask-specific data, and have recently achieved great success. However, PLMs arenotorious for their enormous parameters and hard to be deployed on real-lifeapplications. Knowledge distillation has been prevailing to address thisproblem by transferring knowledge from a large teacher to a much smallerstudent over a set of data. We argue that the selection of thee three keycomponents, namely teacher, training data, and learning objective, is crucialto the effectiveness of distillation. We, therefore, propose a four-stageprogressive distillation framework ERNIE-Tiny to compress PLM, which varies thethree components gradually from general level to task-specific level.Specifically, the first stage, General Distillation, performs distillation withguidance from pretrained teacher, gerenal data and latent distillation loss.Then, General-Enhanced Distillation changes teacher model from pretrainedteacher to finetuned teacher. After that, Task-Adaptive Distillation shiftstraining data from general data to task-specific data. In the end,Task-Specific Distillation, adds two additional losses, namely Soft-Label andHard-Label loss onto the last stage. Empirical results demonstrate theeffectiveness of our framework and generalization gain brought by ERNIE-Tiny.Inparticular, experiments show that a 4-layer ERNIE-Tiny maintains over98.0%performance of its 12-layer teacher BERT base on GLUE benchmark,surpassing state-of-the-art (SOTA) by 1.0% GLUE score with the same amount ofparameters. Moreover, ERNIE-Tiny achieves a new compression SOTA on fiveChinese NLP tasks, outperforming BERT base by 0.4% accuracy with 7.5x fewerparameters and9.4x faster inference speed.

ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression