English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Dense retrieval (DR) has shown promising results in information retrieval. Inessence, DR requires high-quality text representations to support effectivesearch in the representation space. Recent studies have shown that pre-trainedautoencoder-based language models with a weak decoder can provide high-qualitytext representations, boosting the effectiveness and few-shot ability of DRmodels. However, even a weak autoregressive decoder has the bypass effect onthe encoder. More importantly, the discriminative ability of learnedrepresentations may be limited since each token is treated equally important indecoding the input texts. To address the above problems, in this paper, wepropose a contrastive pre-training approach to learn a discriminativeautoencoder with a lightweight multi-layer perception (MLP) decoder. The basicidea is to generate word distributions of input text in a non-autoregressivefashion and pull the word distributions of two masked versions of one textclose while pushing away from others. We theoretically show that ourcontrastive strategy can suppress the common words and highlight therepresentative words in decoding, leading to discriminative representations.Empirical results show that our method can significantly outperform thestate-of-the-art autoencoder-based language models and other pre-trained modelsfor dense retrieval.

A Contrastive Pre-training Approach to Learn Discriminative Autoencoder for Dense Retrieval