English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pretrained language models (PreLMs) are revolutionizing naturallanguage processing across all benchmarks. However, their sheer size isprohibitive for small laboratories or for deployment on mobile devices.Approaches like pruning and distillation reduce the model size but typicallyretain the same model architecture. In contrast, we explore distilling PreLMsinto a different, more efficient architecture, Continual Multiplication ofWords (CMOW), which embeds each word as a matrix and uses matrix multiplicationto encode sequences. We extend the CMOW architecture and its CMOW/CBOW-Hybridvariant with a bidirectional component for more expressive power, per-tokenrepresentations for a general (task-agnostic) distillation during pretraining,and a two-sequence encoding scheme that facilitates downstream tasks onsentence pairs, such as sentence similarity and natural language inference. Ourmatrix-based bidirectional CMOW/CBOW-Hybrid model is competitive to DistilBERTon question similarity and recognizing textual entailment, but uses only halfof the number of parameters and is three times faster in terms of inferencespeed. We match or exceed the scores of ELMo for all tasks of the GLUEbenchmark except for the sentiment analysis task SST-2 and the linguisticacceptability task CoLA. However, compared to previous cross-architecturedistillation approaches, we demonstrate a doubling of the scores on detectinglinguistic acceptability. This shows that matrix-based embeddings can be usedto distill large PreLM into competitive models and motivates further researchin this direction.

General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings