English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Automatically detecting software failures is an important task and alongstanding challenge. It requires finding failure-inducing test cases whosetest input can trigger the software's fault, and constructing an automatedoracle to detect the software's incorrect behaviors. Recent advancement oflarge language models (LLMs) motivates us to study how far this challenge canbe addressed by ChatGPT, a state-of-the-art LLM. Unfortunately, our study showsthat ChatGPT has a low probability (28.8%) of finding correct failure-inducingtest cases for buggy programs. A possible reason is that findingfailure-inducing test cases requires analyzing the subtle code differencesbetween a buggy program and its correct version. When these two versions havesimilar syntax, ChatGPT is weak at recognizing subtle code differences. Ourinsight is that ChatGPT's performance can be substantially enhanced whenChatGPT is guided to focus on the subtle code difference. We have aninteresting observation that ChatGPT is effective in inferring the intendedbehaviors of a buggy program. The intended behavior can be leveraged tosynthesize programs, in order to make the subtle code difference between abuggy program and its correct version (i.e., the synthesized program) explicit.Driven by this observation, we propose a novel approach that synergisticallycombines ChatGPT and differential testing to find failure-inducing test cases.We evaluate our approach on Quixbugs (a benchmark of buggy programs), andcompare it with state-of-the-art baselines, including direct use of ChatGPT andPynguin. The experimental result shows that our approach has a much higherprobability (77.8%) of finding correct failure-inducing test cases, 2.7X as thebest baseline.

Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting