English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Current trends in pre-training Large Language Models (LLMs) primarily focuson the scaling of model and dataset size. While the quality of pre-trainingdata is considered an important factor for training powerful LLMs, it remains anebulous concept that has not been rigorously characterized. To this end, wepropose a formalization of one key aspect of data quality -- measuring thevariability of natural language data -- specifically via a measure we call thediversity coefficient. Our empirical analysis shows that the proposed diversitycoefficient aligns with the intuitive properties of diversity and variability,e.g., it increases as the number of latent concepts increases. Then, we measurethe diversity coefficient of publicly available pre-training datasets anddemonstrate that their formal diversity is high compared to theoretical lowerand upper bounds. Finally, we conduct a comprehensive set of controlledinterventional experiments with GPT-2 and LLaMAv2 that demonstrate thediversity coefficient of pre-training data characterizes useful aspects ofdownstream model evaluation performance -- totaling 44 models of various sizes(51M to 7B parameters). We conclude that our formal notion of diversity is animportant aspect of data quality that captures variability and causally leadsto improved evaluation performance.

Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data