English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Transformer-based language models have demonstrated remarkableperformance across many NLP domains. However, the unsupervised pre-trainingstep of these models suffers from unbearable overall computational expenses.Current methods for accelerating the pre-training either rely on massiveparallelism with advanced hardware or are not applicable to language modeling.In this work, we propose a method based on progressive layer dropping thatspeeds the training of Transformer-based language models, not at the cost ofexcessive hardware resources but from model architecture change and trainingtechnique boosted efficiency. Extensive experiments on BERT show that theproposed method achieves a 24% time reduction on average per sample and allowsthe pre-training to be 2.5 times faster than the baseline to get a similaraccuracy on downstream tasks. While being faster, our pre-trained models areequipped with strong knowledge transferability, achieving comparable andsometimes higher GLUE score than the baseline when pre-trained with the samenumber of samples.

Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping