English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, large pre-trained models have significantly improved theperformance of various Natural LanguageProcessing (NLP) tasks but they areexpensive to serve due to long serving latency and large memory usage. Tocompress these models, knowledge distillation has attracted an increasingamount of interest as one of the most effective methods for model compression.However, existing distillation methods have not yet addressed the uniquechallenges of model serving in datacenters, such as handling fast evolvingmodels, considering serving performance, and optimizing for multipleobjectives. To solve these problems, we propose AutoDistill, an end-to-endmodel distillation framework integrating model architecture exploration andmulti-objective optimization for building hardware-efficient NLP pre-trainedmodels. We use Bayesian Optimization to conduct multi-objective NeuralArchitecture Search for selecting student model architectures. The proposedsearch comprehensively considers both prediction accuracy and serving latencyon target hardware. The experiments on TPUv4i show the finding of seven modelarchitectures with better pre-trained accuracy (up to 3.2% higher) and lowerinference latency (up to 1.44x faster) than MobileBERT. By running downstreamNLP tasks in the GLUE benchmark, the model distilled for pre-training byAutoDistill with 28.5M parameters achieves an 81.69 average score, which ishigher than BERT_BASE, DistillBERT, TinyBERT, NAS-BERT, and MobileBERT. Themost compact model found by AutoDistill contains only 20.6M parameters butstill outperform BERT_BASE(109M), DistillBERT(67M), TinyBERT(67M), andMobileBERT(25.3M) regarding the average GLUE score. By evaluating on SQuAD, amodel found by AutoDistill achieves an 88.4% F1 score with 22.8M parameters,which reduces parameters by more than 62% while maintaining higher accuracythan DistillBERT, TinyBERT, and NAS-BERT.

AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models