English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Incremental language learning with pseudo-data can alleviate catastrophicforgetting in neural networks. However, to obtain better performance, formermethods have higher demands for pseudo-data of the previous tasks. Theperformance dramatically decreases when fewer pseudo-data are employed. Inaddition, the distribution of pseudo-data gradually deviates from the real datawith the sequential learning of different tasks. The deviation will be greaterwith more tasks learned, which results in more serious catastrophic forgetting.To address these issues, we propose reminding incremental language model viadata-free self-distillation (DFSD), which includes self-distillation based onthe Earth Mover's Distance and hidden data augmentation. By estimating theknowledge distribution in all layers of GPT-2 and transforming it from teachermodel to student model, the Self-distillation based on the Earth Mover'sDistance can significantly reduce the demand for pseudo-data. Hidden dataaugmentation can greatly alleviate the catastrophic forgetting caused bydeviations via modeling the generation of pseudo-data as a hidden dataaugmentation process, where each sample is a mixture of all trained task data.The experimental results demonstrate that our DFSD can exceed the previousstate-of-the-art methods even if the maximum decrease in pseudo-data is 90%.

Reminding the Incremental Language Model via Data-Free Self-Distillation