English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Modern Natural Language Generation (NLG) models come with massivecomputational and storage requirements. In this work, we study the potential ofcompressing them, which is crucial for real-world applications serving millionsof users. We focus on Knowledge Distillation (KD) techniques, in which a smallstudent model learns to imitate a large teacher model, allowing to transferknowledge from the teacher to the student. In contrast to much of the previouswork, our goal is to optimize the model for a specific NLG task and a specificdataset. Typically in real-world applications, in addition to labeled datathere is abundant unlabeled task-specific data, which is crucial for attaininghigh compression rates via KD. In this work, we conduct a systematic study oftask-specific KD techniques for various NLG tasks under realistic assumptions.We discuss the special characteristics of NLG distillation and particularly theexposure bias problem. Following, we derive a family of Pseudo-Target (PT)augmentation methods, substantially extending prior work on sequence-level KD.We propose the Joint-Teaching method, which applies word-level KD to multiplePTs generated by both the teacher and the student. Finally, we validate ourfindings in an extreme setup with no labeled examples using GPT-4 as theteacher. Our study provides practical model design observations anddemonstrates the effectiveness of PT training for task-specific KD in NLG.

A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training