English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Masking tokens uniformly at random constitutes a common flaw in thepretraining of Masked Language Models (MLMs) such as BERT. We show that suchuniform masking allows an MLM to minimize its training objective by latchingonto shallow local signals, leading to pretraining inefficiency and suboptimaldownstream performance. To address this flaw, we propose PMI-Masking, aprincipled masking strategy based on the concept of Pointwise MutualInformation (PMI), which jointly masks a token n-gram if it exhibits highcollocation over the corpus. PMI-Masking motivates, unifies, and improves uponprior more heuristic approaches that attempt to address the drawback of randomuniform token masking, such as whole-word masking, entity/phrase masking, andrandom-span masking. Specifically, we show experimentally that PMI-Maskingreaches the performance of prior masking approaches in half the training time,and consistently improves performance at the end of training.

PMI-Masking: Principled masking of correlated spans