English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretrained Masked Language Models (MLMs) have revolutionised NLP in recentyears. However, previous work has indicated that off-the-shelf MLMs are noteffective as universal lexical or sentence encoders without furthertask-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasksusing annotated task data. In this work, we demonstrate that it is possible toturn MLMs into effective universal lexical and sentence encoders even withoutany additional data and without any supervision. We propose an extremelysimple, fast and effective contrastive learning technique, termed Mirror-BERT,which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30seconds without any additional external knowledge. Mirror-BERT relies on fullyidentical or slightly modified string pairs as positive (i.e., synonymous)fine-tuning examples, and aims to maximise their similarity during identityfine-tuning. We report huge gains over off-the-shelf MLMs with Mirror-BERT inboth lexical-level and sentence-level tasks, across different domains anddifferent languages. Notably, in the standard sentence semantic similarity(STS) tasks, our self-supervised Mirror-BERT model even matches the performanceof the task-tuned Sentence-BERT models from prior work. Finally, we delvedeeper into the inner workings of MLMs, and suggest some evidence on why thissimple approach can yield effective universal lexical and sentence encoders.

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders