English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Masked language modeling (MLM) is a widely used self-supervised pretrainingobjective, where a model needs to predict an original token that is replacedwith a mask given contexts. Although simpler and computationally efficientpretraining objectives, e.g., predicting the first character of a masked token,have recently shown comparable results to MLM, no objectives with a maskingscheme actually outperform it in downstream tasks. Motivated by the assumptionthat their lack of complexity plays a vital role in the degradation, wevalidate whether more complex masked objectives can achieve better results andinvestigate how much complexity they should have to perform comparably to MLM.Our results using GLUE, SQuAD, and Universal Dependencies benchmarksdemonstrate that more complicated objectives tend to show better downstreamresults with at least half of the MLM complexity needed to perform comparablyto MLM. Finally, we discuss how we should pretrain a model using a maskedobjective from the task complexity perspective.

How does the task complexity of masked pretraining objectives affect downstream performance?