English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In this paper, we investigate the effectiveness of state-of-the-art LLM,i.e., GPT-4, with three different prompting engineering techniques (i.e., basicprompting, in-context learning, and task-specific prompting) against 18fine-tuned LLMs on three typical ASE tasks, i.e., code generation, codesummarization, and code translation. Our quantitative analysis of theseprompting strategies suggests that prompt engineering GPT-4 cannot necessarilyand significantly outperform fine-tuning smaller/older LLMs in all three tasks.For comment generation, GPT-4 with the best prompting strategy (i.e.,task-specific prompt) had outperformed the first-ranked fine-tuned model by8.33% points on average in BLEU. However, for code generation, the first-rankedfine-tuned model outperforms GPT-4 with best prompting by 16.61% and 28.3%points, on average in BLEU. For code translation, GPT-4 and fine-tunedbaselines tie as they outperform each other on different translation tasks. Toexplore the impact of different prompting strategies, we conducted a user studywith 27 graduate students and 10 industry practitioners. From our qualitativeanalysis, we find that the GPT-4 with conversational prompts (i.e., when ahuman provides feedback and instructions back and forth with a model to achievebest results) showed drastic improvement compared to GPT-4 with automaticprompting strategies. Moreover, we observe that participants tend to requestimprovements, add more context, or give specific instructions as conversationalprompts, which goes beyond typical and generic prompting strategies. Our studysuggests that, at its current state, GPT-4 with conversational prompting hasgreat potential for ASE tasks, but fully automated prompt engineering with nohuman in the loop requires more study and improvement.

Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering Tasks