English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Identifying and resolving logic errors can be one of the most frustratingchallenges for novices programmers. Unlike syntax errors, for which a compileror interpreter can issue a message, logic errors can be subtle. In certainconditions, buggy code may even exhibit correct behavior -- in other cases, theissue might be about how a problem statement has been interpreted. Such errorscan be hard to spot when reading the code, and they can also at times be missedby automated tests. There is great educational potential in automaticallydetecting logic errors, especially when paired with suitable feedback fornovices. Large language models (LLMs) have recently demonstrated surprisingperformance for a range of computing tasks, including generating and explainingcode. These capabilities are closely linked to code syntax, which aligns withthe next token prediction behavior of LLMs. On the other hand, logic errorsrelate to the runtime performance of code and thus may not be as well suited toanalysis by LLMs. To explore this, we investigate the performance of twopopular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendlyexplanation of logic errors. We compare LLM performance with a large cohort ofintroductory computing students $(n=964)$ solving the same error detectiontask. Through a mixed-methods analysis of student and model responses, weobserve significant improvement in logic error identification between theprevious and current generation of LLMs, and find that both LLM generationssignificantly outperform students. We outline how such models could beintegrated into computing education tools, and discuss their potential forsupporting students when learning programming.

Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models