English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

While pre-training and fine-tuning, e.g., BERT~\citep{devlin2018bert},GPT-2~\citep{radford2019language}, have achieved great success in languageunderstanding and generation tasks, the pre-trained models are usually too bigfor online deployment in terms of both memory cost and inference speed, whichhinders them from practical online usage. In this paper, we propose LightPAFF,a Lightweight Pre-training And Fine-tuning Framework that leverages two-stageknowledge distillation to transfer knowledge from a big teacher model to alightweight student model in both pre-training and fine-tuning stages. In thisway the lightweight model can achieve similar accuracy as the big teachermodel, but with much fewer parameters and thus faster online inference speed.LightPAFF can support different pre-training methods (such as BERT, GPT-2 andMASS~\citep{song2019mass}) and be applied to many downstream tasks. Experimentson three language understanding tasks, three language modeling tasks and threesequence to sequence generation tasks demonstrate that while achieving similaraccuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the modelsize by nearly 5x and improves online inference speed by 5x-7x.

LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning