English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pre-trained models have achieved great success in many natural languageprocessing tasks. However, when they are applied in specific domains, thesemodels suffer from domain shift and bring challenges in fine-tuning and onlineserving for latency and capacity constraints. In this paper, we present ageneral approach to developing small, fast and effective pre-trained models forspecific domains. This is achieved by adapting the off-the-shelf generalpre-trained models and performing task-agnostic knowledge distillation intarget domains. Specifically, we propose domain-specific vocabulary expansionin the adaptation stage and employ corpus level occurrence probability tochoose the size of incremental vocabulary automatically. Then we systematicallyexplore different strategies to compress the large pre-trained models forspecific domains. We conduct our experiments in the biomedical and computerscience domain. The experimental results demonstrate that our approach achievesbetter performance over the BERT BASE model in domain-specific tasks while 3.3xsmaller and 5.1x faster than BERT BASE. The code and pre-trained models areavailable at https://aka.ms/adalm.

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains