English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Pre-training language models (LMs) on large-scale unlabeled text data makesthe model much easier to achieve exceptional downstream performance than theircounterparts directly trained on the downstream tasks. In this work, we studywhat specific traits in the pre-training data, other than the semantics, make apre-trained LM superior to their counterparts trained from scratch ondownstream tasks. We propose to use artificially constructed datasets as thepre-training data to exclude the effect of semantics, and further control whatcharacteristics the pre-training corpora have. By fine-tuning the pre-trainedmodels on GLUE benchmark, we can learn how beneficial it is to transfer theknowledge from the model trained on the dataset possessing that specific trait.We define and discuss three different characteristics in the artificialdataset: 1) matching the token's uni-gram or bi-gram distribution betweenpre-training and downstream fine-tuning, 2) the presence of the explicitdependencies among the tokens in a sequence, 3) the length of the implicitdependencies among the tokens in a sequence. Our experiments show that theexplicit dependencies in the sequences of the pre-training data are critical tothe downstream performance. Our results also reveal that models achieve betterdownstream performance when pre-trained on a dataset with a longer range ofimplicit dependencies. Based on our analysis, we find that models pre-trainedwith artificial datasets are prone to learn spurious correlation in downstreamtasks. Our work reveals that even if the LMs are not pre-trained on naturallanguage, they still gain transferability on certain human language downstreamtasks once the LMs learn to model the token dependencies in the sequences. Thisresult helps us understand the exceptional transferability of pre-trained LMs.

On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets