English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

During the development of large language models (LLMs), the scale and qualityof the pre-training data play a crucial role in shaping LLMs' capabilities. Toaccelerate the research of LLMs, several large-scale datasets, such as C4 [1],Pile [2], RefinedWeb [3] and WanJuan [4], have been released to the public.However, most of the released corpus focus mainly on English, and there isstill lack of complete tool-chain for extracting clean texts from web data.Furthermore, fine-grained information of the corpus, e.g. the quality of eachtext, is missing. To address these challenges, we propose in this paper a newcomplete tool-chain EvalWeb to extract Chinese clean texts from noisy web data.First, similar to previous work, manually crafted rules are employed to discardexplicit noisy texts from the raw crawled web contents. Second, a well-designedevaluation model is leveraged to assess the remaining relatively clean data,and each text is assigned a specific quality score. Finally, we can easilyutilize an appropriate threshold to select the high-quality pre-training datafor Chinese. Using our proposed approach, we release the largest and latestlarge-scale high-quality Chinese web text ChineseWebText, which consists of1.42 TB and each text is associated with a quality score, facilitating the LLMresearchers to choose the data according to the desired quality thresholds. Wealso release a much cleaner subset of 600 GB Chinese data with the qualityexceeding 90%.

ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model