English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Large Language Models (LLMs), such as GPT and BERT, have demonstratedremarkable capabilities in addressing neural language process tasks. Recently,the release of ChatGPT has garnered significant attention due to its ability toanalyze, comprehend, and synthesize information from user inputs. Therefore,these LLMs were adopted by researchers in many different domains. In the realmof code analysis, researchers have applied LLMs to tasks like code review andcode generation. However, we observed that the strengths and limitations ofadopting these LLMs to the code analysis have not been investigated. In thispaper, we delve into LLMs' capabilities in security-oriented program analysis,considering perspectives from both attackers and security analysts. We focus ontwo representative LLMs, ChatGPT and CodeBert, and evaluate their performancein solving typical analytic tasks with varying levels of difficulty. Given thedifferent natures of ChatGPT and CodeBERT, we conduct a qualitative analysis ofthe model's output for ChatGPT and a quantitative analysis for CodeBERT,respectively. For ChatGPT, we present a case study involving severalsecurity-oriented program analysis tasks while deliberately introducingchallenges to assess its responses. On the other hand, for CodeBERT, wesystematically analyze and classify the features in code, quantitativelyevaluating the impact of these features on the model's performance. Our studydemonstrates the LLM's efficiency in learning high-level semantics from code,positioning ChatGPT as a potential asset in security-oriented contexts.However, it is essential to acknowledge certain limitations, such as the heavyreliance on well-defined variable and function names, making them unable tolearn from anonymized code. We hope that our findings and analysis will offervaluable insights for future researchers in this domain.

The Effectiveness of Large Language Models (ChatGPT and CodeBERT) for Security-Oriented Code Analysis