English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Program synthesis has been long studied with recent approaches focused ondirectly using the power of Large Language Models (LLMs) to generate code.Programming benchmarks, with curated synthesis problems and test-cases, areused to measure the performance of various LLMs on code synthesis. However,these test-cases can be limited in both quantity and quality for fullyassessing the functional correctness of the generated code. Such limitation inthe existing benchmarks begs the following question: In the era of LLMs, is thecode generated really correct? To answer this, we propose EvalPlus -- a codesynthesis evaluation framework to rigorously benchmark the functionalcorrectness of LLM-synthesized code. EvalPlus augments a given evaluationdataset with large amounts of test-cases newly produced by an automatic testinput generator, powered by both LLM- and mutation-based strategies. WhileEvalPlus is general, we extend the test-cases of the popular HumanEvalbenchmark by 80x to build HumanEval+. Our extensive evaluation across 26popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able tocatch significant amounts of previously undetected wrong code synthesized byLLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found thattest insufficiency can lead to mis-ranking. For example, bothWizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+,while none of them could on HumanEval. Our work not only indicates that priorpopular code synthesis evaluation results do not accurately reflect the trueperformance of LLMs for code synthesis, but also opens up a new direction toimprove such programming benchmarks through automated testing. We haveopen-sourced our tools, enhanced datasets as well as all LLM-generated code athttps://github.com/evalplus/evalplus to facilitate and accelerate futureLLM-for-code research.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation