English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) have been gaining increasing attention anddemonstrated promising performance across a variety of Software Engineering(SE) tasks, such as Automated Program Repair (APR), code summarization, andcode completion. For example, ChatGPT, the latest black-box LLM, has beeninvestigated by numerous recent research studies and has shown impressiveperformance in various tasks. However, there exists a potential risk of dataleakage since these LLMs are usually close-sourced with unknown specifictraining details, e.g., pre-training datasets.  In this paper, we seek to review the bug-fixing capabilities of ChatGPT on aclean APR benchmark with different research objectives. We first introduce{\benchmark}, a new benchmark with buggy and the corresponding fixed programsfrom competitive programming problems starting from 2023, after the trainingcutoff point of ChatGPT. The results on {\benchmark} show that ChatGPT is ableto fix 109 out of 151 buggy programs using the basic prompt within 35independent rounds, outperforming state-of-the-art LLMs CodeT5 and PLBART by27.5\% and 62.4\% prediction accuracy. We also investigate the impact of threetypes of prompts, i.e., problem description, error feedback, and buglocalization, leading to additional 34 fixed bugs. Besides, we provideadditional discussion from the interactive nature of ChatGPT to illustrate thecapacity of a dialog-based repair workflow with 9 additional fixed bugs.Inspired by the findings, we further pinpoint various challenges andopportunities for advanced SE study equipped with such LLMs (e.g.,~ChatGPT) inthe near future. More importantly, our work calls for more research on thereevaluation of the achievements obtained by existing black-box LLMs acrossvarious SE tasks, not limited to ChatGPT on APR.

A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair