English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Masked language models (MLMs) conventionally mask 15% of tokens due to thebelief that more masking would leave insufficient context to learn goodrepresentations; this masking rate has been widely used, regardless of modelsizes or masking strategies. In this work, we revisit this important choice ofMLM pre-training. We first establish that 15% is not universally optimal, andlarger models should adopt a higher masking rate. Specifically, we find thatmasking 40% outperforms 15% for BERT-large size models on GLUE and SQuAD.Interestingly, an extremely high masking rate of 80% can still preserve 95%fine-tuning performance and most of the accuracy in linguistic probing,challenging the conventional wisdom about the role of the masking rate. We thenexamine the interplay between masking rates and masking strategies and findthat uniform masking requires a higher masking rate compared to sophisticatedmasking strategies such as span or PMI masking. Finally, we argue thatincreasing the masking rate has two distinct effects: it leads to morecorruption, which makes the prediction task more difficult; it also enablesmore predictions, which benefits optimization. Using this framework, we revisitBERT's 80-10-10 corruption strategy. Together, our results contribute to abetter understanding of MLM pre-training.

Should You Mask 15% in Masked Language Modeling?