English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) have exhibited remarkable reasoning capabilitiesand become the foundation of language technologies. Inspired by the greatsuccess of code data in training LLMs, we naturally wonder at which trainingstage introducing code data can really help LLMs reasoning. To this end, thispaper systematically explores the impact of code data on LLMs at differentstages. Concretely, we introduce the code data at the pre-training stage,instruction-tuning stage, and both of them, respectively. Then, the reasoningcapability of LLMs is comprehensively and fairly evaluated via six reasoningtasks in five domains. We critically analyze the experimental results andprovide conclusions with insights. First, pre-training LLMs with the mixture ofcode and text can significantly enhance LLMs' general reasoning capabilityalmost without negative transfer on other tasks. Besides, at theinstruction-tuning stage, code data endows LLMs the task-specific reasoningcapability. Moreover, the dynamic mixing strategy of code and text data assistsLLMs to learn reasoning capability step-by-step during training. These insightsdeepen the understanding of LLMs regarding reasoning ability for theirapplication, such as scientific question answering, legal support, etc. Thesource code and model parameters are released at thelink:~\url{https://github.com/yingweima2022/CodeLLM}.

At Which Training Stage Does Code Data Help LLMs Reasoning?