English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) have demonstrated exceptional coding capability.However, as another critical component of programming proficiency, thedebugging capability of LLMs remains relatively unexplored. Previousevaluations of LLMs' debugging ability are significantly limited by the risk ofdata leakage, the scale of the dataset, and the variety of tested bugs. Toovercome these deficiencies, we introduce `DebugBench', an LLM debuggingbenchmark consisting of 4,253 instances. It covers four major bug categoriesand 18 minor types in C++, Java, and Python. To construct DebugBench, wecollect code snippets from the LeetCode community, implant bugs into sourcedata with GPT-4, and assure rigorous quality checks. We evaluate two commercialand three open-source models in a zero-shot scenario. We find that (1) whileclosed-source models like GPT-4 exhibit inferior debugging performance comparedto humans, open-source models such as Code Llama fail to attain any pass ratescores; (2) the complexity of debugging notably fluctuates depending on the bugcategory; (3) incorporating runtime feedback has a clear impact on debuggingperformance which is not always helpful. As an extension, we also compare LLMdebugging and code generation, revealing a strong correlation between them forclosed-source models. These findings will benefit the development of LLMs indebugging.

DebugBench: Evaluating Debugging Capability of Large Language Models