English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Maintaining the integrity of long-term data collection is an essentialscientific practice. As a field evolves, so too will that field's measurementinstruments and data storage systems, as they are invented, improved upon, andmade obsolete. For data streams generated by opaque sociotechnical systemswhich may have episodic and unknown internal rule changes, detecting andaccounting for shifts in historical datasets requires vigilance and creativeanalysis. Here, we show that around 10\% of day-scale word usage frequency timeseries for Twitter collected in real time for a set of roughly 10,000frequently used words for over 10 years come from tweets with, in effect,corrupted language labels. We describe how we uncovered problematic signalswhile comparing word usage over varying time frames. We locate time pointswhere Twitter switched on or off different kinds of language identificationalgorithms, and where data formats may have changed. We then show how we createa statistic for identifying and removing words with pathological time series.While our resulting process for removing `bad' time series from ensembles oftime series is particular, the approach leading to its construction may begeneralizeable.

Long-term word frequency dynamics derived from Twitter are corrupted: A  bespoke approach to detecting and removing pathologies in ensembles of time  series