English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Using large language models (LLMs) for source code has recently gainedattention. LLMs, such as Transformer-based models like Codex and ChatGPT, havebeen shown to be highly capable of solving a wide range of programmingproblems. However, the extent to which LLMs understand problem descriptions andgenerate programs accordingly or just retrieve source code from the mostrelevant problem in training data based on superficial cues has not beendiscovered yet. To explore this research question, we conduct experiments tounderstand the robustness of several popular LLMs, CodeGen and GPT-3.5 seriesmodels, capable of tackling code generation tasks in introductory programmingproblems. Our experimental results show that CodeGen and Codex are sensitive tothe superficial modifications of problem descriptions and significantly impactcode generation performance. Furthermore, we observe that Codex relies onvariable names, as randomized variables decrease the solved rate significantly.However, the state-of-the-art (SOTA) models, such as InstructGPT and ChatGPT,show higher robustness to superficial modifications and have an outstandingcapability for solving programming problems. This highlights the fact thatslight modifications to the prompts given to the LLMs can greatly affect codegeneration performance, and careful formatting of prompts is essential forhigh-quality code generation, while the SOTA models are becoming more robust toperturbations.

Exploring the Robustness of Large Language Models for Solving Programming Problems