English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) have shown remarkable capabilities in processingboth natural and programming languages, which have enabled various applicationsin software engineering, such as requirement engineering, code generation, andsoftware testing. However, existing code generation benchmarks do notnecessarily assess the code understanding performance of LLMs, especially forthe subtle inconsistencies that may arise between code and its semanticsdescribed in natural language.  In this paper, we propose a novel method to systematically assess the codeunderstanding performance of LLMs, particularly focusing on subtle differencesbetween code and its descriptions, by introducing code mutations to existingcode generation datasets. Code mutations are small changes that alter thesemantics of the original code, creating a mismatch with the natural languagedescription. We apply different types of code mutations, such as operatorreplacement and statement deletion, to generate inconsistent code-descriptionpairs. We then use these pairs to test the ability of LLMs to correctly detectthe inconsistencies.  We propose a new LLM testing method, called Mutation-based ConsistencyTesting (MCT), and conduct a case study on the two popular LLMs, GPT-3.5 andGPT-4, using the state-of-the-art code generation benchmark, HumanEval-X, whichconsists of six programming languages (Python, C++, Java, Go, JavaScript, andRust). We compare the performance of the LLMs across different types of codemutations and programming languages and analyze the results. We find that theLLMs show significant variation in their code understanding performance andthat they have different strengths and weaknesses depending on the mutationtype and language.

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs