English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Analysis of pre-trained code models also has revealed that they caneffectively learn program syntax. However, these works are limited in analyzingcode syntax and their distance-based approaches are not accurate due to thecurse of high dimensionality. Furthermore, the study of the learnt programsemantics of these models is rarely discussed. To further understand the codefeatures learnt by these models, in this paper, we target two well-knownrepresentative code pre-trained models (i.e., CodeBERT and GraphCodeBERT) anddevise a set of probing tasks for the syntax and semantics analysis.Specifically, on one hand, we design two probing tasks (i.e., syntax pair nodeprediction and token tagging prediction) to manipulate AST for theunderstanding of learnt program syntax. On the other hand, we design two tasks(i.e., semantic relationship prediction and semantic propagationprediction(inGraph) ) on the constructed control flow graph (CFG), datadependency graph (DDG) and control dependency graph (CDG) for the learntprogram semantic analysis. In addition, to understand which kind of programsemantics these pre-trained models can comprehend well, we conduct thestatistical analysis for attention weights learnt by different heads andlayers. Through extensive analysis in terms of program syntax and semantics, wehave the following findings: 1) Both CodeBERT and GraphCodeBERT can learn theprogram syntax well. 2) Both CodeBERT and GraphCodeBERT can learn programsemantics to different extents. GraphCodeBERT is superior to CodeBERT inlearning program control flow and data dependency information but has a similarcapability to CodeBERT in learning control dependency information. 3) BothCodeBERT and GraphCodeBERT can capture program semantics in the final layer ofrepresentation, but different attention heads and layers exhibit differentroles in learning program semantics.

Are Code Pre-trained Models Powerful to Learn Code Syntax and Semantics?