English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained language models (PLMs) have established the new paradigm in thefield of NLP. For more powerful PLMs, one of the most popular and successfulway is to continuously scale up sizes of the models and the pre-trainingcorpora. These large corpora are generally obtained by converging smaller onesfrom multiple sources, they are thus growing increasingly diverse. However, theside-effects of these colossal converged corpora remain understudied. In thispaper, we identify the disadvantage of heterogeneous corpora from multiplesources for pre-training PLMs. Towards coordinated pre-training on diversecorpora, we further propose source prompts (SP), which explicitly prompt themodel of the data source at the pre-training and fine-tuning stages. Results ofextensive experiments demonstrate that PLMs pre-trained with SP on diversecorpora gain significant improvement in various downstream tasks.

Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources