English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

A recent trend in Natural Language Processing is the exponential growth inLanguage Model (LM) size, which prevents research groups without a necessaryhardware infrastructure from participating in the development process. Thisstudy investigates methods for Knowledge Distillation (KD) to provide efficientalternatives to large-scale models. In this context, KD means extractinginformation about language encoded in a Neural Network and Lexical KnowledgeDatabases. We developed two methods to test our hypothesis that efficientarchitectures can gain knowledge from LMs and extract valuable information fromlexical sources. First, we present a technique to learn confident probabilitydistribution for Masked Language Modeling by prediction weighting of multipleteacher networks. Second, we propose a method for Word Sense Disambiguation(WSD) and lexical KD that is general enough to be adapted to many LMs. Ourresults show that KD with multiple teachers leads to improved trainingconvergence. When using our lexical pre-training method, LM characteristics arenot lost, leading to increased performance in Natural Language Understanding(NLU) tasks over the state-of-the-art while adding no parameters. Moreover, theimproved semantic understanding of our model increased the task performancebeyond WSD and NLU in a real-problem scenario (Plagiarism Detection). Thisstudy suggests that sophisticated training methods and network architecturescan be superior over scaling trainable parameters. On this basis, we suggestthe research area should encourage the development and use of efficient modelsand rate impacts resulting from growing LM size equally against taskperformance.

A Cohesive Distillation Architecture for Neural Language Models