English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

We present results from a large-scale experiment on pretraining encoders withnon-embedding parameter counts ranging from 700M to 9.3B, their subsequentdistillation into smaller models ranging from 17M-170M parameters, and theirapplication to the Natural Language Understanding (NLU) component of a virtualassistant system. Though we train using 70% spoken-form data, our teachermodels perform comparably to XLM-R and mT5 when evaluated on the written-formCross-lingual Natural Language Inference (XNLI) corpus. We perform a secondstage of pretraining on our teacher models using in-domain data from oursystem, improving error rates by 3.86% relative for intent classification and7.01% relative for slot filling. We find that even a 170M-parameter modeldistilled from our Stage 2 teacher model has 2.88% better intent classificationand 7.69% better slot filling error rates when compared to the 2.3B-parameterteacher trained only on public data (Stage 1), emphasizing the importance ofin-domain data for pretraining. When evaluated offline using labeled NLU data,our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85Mparams) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally,we present results from a full virtual assistant experimentation platform,where we find that models trained using our pretraining and distillationpipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91%on an automatic measurement of full-system user dissatisfaction.

Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems