English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

In this paper we combine statistical analysis of large text databases andsimple stochastic models to explain the appearance of scaling laws in thestatistics of word frequencies. Besides the sublinear scaling of the vocabularysize with database size (Heaps' law), here we report a new scaling of thefluctuations around this average (fluctuation scaling analysis). We explainboth scaling laws by modeling the usage of words by simple stochastic processesin which the overall distribution of word-frequencies is fat tailed (Zipf'slaw) and the frequency of a single word is subject to fluctuations acrossdocuments (as in topic models). In this framework, the mean and the variance ofthe vocabulary size can be expressed as quenched averages, implying that: i)the inhomogeneous dissemination of words cause a reduction of the averagevocabulary size in comparison to the homogeneous case, and ii) correlations inthe co-occurrence of words lead to an increase in the variance and thevocabulary size becomes a non-self-averaging quantity. We address theimplications of these observations to the measurement of lexical richness. Wetest our results in three large text databases (Google-ngram, EnlgishWikipedia, and a collection of scientific articles).

Scaling laws and fluctuations in the statistics of word frequencies