English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The pre-training of masked language models (MLMs) consumes massivecomputation to achieve good results on downstream NLP tasks, resulting in alarge carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act asplaceholders and gather the contextualized information from unmasked tokens torestore the corrupted information. It raises the question of whether we canappend [MASK]s at a later layer, to reduce the sequence length for earlierlayers and make the pre-training more efficient. We show: (1) [MASK]s canindeed be appended at a later layer, being disentangled from the wordembedding; (2) The gathering of contextualized information from unmasked tokenscan be conducted with a few layers. By further increasing the masking rate from15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch withonly 78% and 68% of the original computational budget without any degradationon the GLUE benchmark. When pre-training with the original budget, our methodoutperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.

Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token