English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large volumes of text data have contributed significantly to the developmentof large language models (LLMs) in recent years. This data is typicallyacquired by scraping the internet, leading to pretraining datasets comprised ofnoisy web text. To date, efforts to prune these datasets down to a higherquality subset have relied on hand-crafted heuristics encoded as rule-basedfilters. In this work, we take a wider view and explore scalable estimates ofdata quality that can be used to systematically measure the quality ofpretraining data. We perform a rigorous comparison at scale of the simple dataquality estimator of perplexity, as well as more sophisticated andcomputationally intensive estimates of the Error L2-Norm and memorization.These metrics are used to rank and prune pretraining corpora, and wesubsequently compare LLMs trained on these pruned datasets. Surprisingly, wefind that the simple technique of perplexity outperforms our morecomputationally expensive scoring methods. We improve over our no-pruningbaseline while training on as little as 30% of the original training dataset.Our work sets the foundation for unexplored strategies in automaticallycurating high quality corpora and suggests the majority of pretraining data canbe removed while retaining performance.

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale