English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretraining is the preliminary and fundamental step in developing capablelanguage models (LM). Despite this, pretraining data design is criticallyunder-documented and often guided by empirically unsupported intuitions. Toaddress this, we pretrain 28 1.5B parameter decoder-only models, training ondata curated (1) at different times, (2) with varying toxicity and qualityfilters, and (3) with different domain compositions. First, we quantify theeffect of pretraining data age. A temporal shift between evaluation data andpretraining data leads to performance degradation, which is not overcome byfinetuning. Second, we explore the effect of quality and toxicity filters,showing a trade-off between performance on standard benchmarks and risk oftoxic generations. Our findings indicate there does not exist aone-size-fits-all solution to filtering training data. We also find that theeffects of different types of filtering are not predictable from text domaincharacteristics. Lastly, we empirically validate that the inclusion ofheterogeneous data sources, like books and web, is broadly beneficial andwarrants greater prioritization. These findings constitute the largest set ofexperiments to validate, quantify, and expose many undocumented intuitionsabout text pretraining, which we hope will help support more informeddata-centric decisions in LM development.

A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity