English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The formation of sentences is a highly structured and history-dependentprocess. The probability of using a specific word in a sentence stronglydepends on the 'history' of word-usage earlier in that sentence. We study asimple history-dependent model of text generation assuming that thesample-space of word usage reduces along sentence formation, on average. Wefirst show that the model explains the approximate Zipf law found in wordfrequencies as a direct consequence of sample-space reduction. We thenempirically quantify the amount of sample-space reduction in the sentences often famous English books, by analysis of corresponding word-transition tablesthat capture which words can follow any given word in a text. We find a highlynested structure in these transition tables and show that this `nestedness' istightly related to the power law exponents of the observed word frequencydistributions. With the proposed model it is possible to understand that thenestedness of a text can be the origin of the actual scaling exponent, and thatdeviations from the exact Zipf law can be understood by variations of thedegree of nestedness on a book-by-book basis. On a theoretical level we areable to show that in case of weak nesting, Zipf's law breaks down in a fasttransition. Unlike previous attempts to understand Zipf's law in language thesample-space reducing model is not based on assumptions of multiplicative,preferential, or self-organised critical mechanisms behind language formation,but simply used the empirically quantifiable parameter 'nestedness' tounderstand the statistics of word frequencies.

Understanding Zipf's law of word frequencies through sample-space collapse in sentence formation