English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Dense retrieval requires high-quality text sequence embeddings to supporteffective search in the representation space. Autoencoder-based language modelsare appealing in dense retrieval as they train the encoder to outputhigh-quality embedding that can reconstruct the input texts. However, in thispaper, we provide theoretical analyses and show empirically that an autoencoderlanguage model with a low reconstruction loss may not provide good sequencerepresentations because the decoder may take shortcuts by exploiting languagepatterns. To address this, we propose a new self-learning method thatpre-trains the autoencoder using a \textit{weak} decoder, with restrictedcapacity and attention flexibility to push the encoder to provide better textrepresentations. Our experiments on web search, news recommendation, and opendomain question answering show that our pre-trained model significantly booststhe effectiveness and few-shot ability of dense retrieval models. Our code isavailable at https://github.com/microsoft/SEED-Encoder/.

Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder