English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Instruction tuning is a burgeoning method to elicit the general intelligenceof Large Language Models (LLMs). However, the creation of instruction data isstill largely heuristic, leading to significant variation in quality anddistribution across existing datasets. Experimental conclusions drawn fromthese datasets are also inconsistent, with some studies emphasizing theimportance of scaling instruction numbers, while others argue that a limitednumber of samples suffice. To better understand data construction guidelines,we deepen our focus from the overall model performance to the growth of eachunderlying ability, such as creative writing, code generation, and logicalreasoning. We systematically investigate the effects of data volume, parametersize, and data construction methods on the development of various abilities,using hundreds of model checkpoints (7b to 33b) fully instruction-tuned on anew collection of over 40k human-curated instruction data. This proposeddataset is stringently quality-controlled and categorized into ten distinct LLMabilities. Our study reveals three primary findings: (i) Despite data volumeand parameter scale directly impacting models' overall performance, someabilities are more responsive to their increases and can be effectively trainedusing limited data, while some are highly resistant to these changes. (ii)Human-curated data strongly outperforms synthetic data from GPT-4 in efficiencyand can constantly enhance model performance with volume increases, but isunachievable with synthetic data. (iii) Instruction data brings powerfulcross-ability generalization, with evaluation results on out-of-domain datamirroring the first two observations. Furthermore, we demonstrate how thesefindings can guide more efficient data constructions, leading to practicalperformance improvements on public benchmarks.

Dynamics of Instruction Tuning: Each Ability of Large Language Models Has Its Own Growth Pace