English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Current trends to pre-train capable Large Language Models (LLMs) mostly focuson scaling of model and dataset size. However, the quality of pre-training datais an important factor for training powerful LLMs, yet it is a nebulous conceptthat has not been fully characterized. Therefore, we use the recently proposedTask2Vec diversity coefficient to ground and understand formal aspects of dataquality, to go beyond scale alone. Specifically, we measure the diversitycoefficient of publicly available pre-training datasets to demonstrate thattheir formal diversity is high when compared to theoretical lower and upperbounds. In addition, to build confidence in the diversity coefficient, weconduct interpretability experiments and find that the coefficient aligns withintuitive properties of diversity, e.g., it increases as the number of latentconcepts increases. We conclude the diversity coefficient is reliable, showit's high for publicly available LLM datasets, and conjecture it can be used tobuild useful diverse datasets for LLMs.

Beyond Scale: the Diversity Coefficient as a Data Quality Metric Demonstrates LLMs are Pre-trained on Formally Diverse Data