English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, fine-tuning pre-trained code models such as CodeBERT on downstreamtasks has achieved great success in many software testing and analysis tasks.While effective and prevalent, fine-tuning the pre-trained parameters incurs alarge computational cost. In this paper, we conduct an extensive experimentalstudy to explore what happens to layer-wise pre-trained representations andtheir encoded code knowledge during fine-tuning. We then propose efficientalternatives to fine-tune the large pre-trained code model based on the abovefindings. Our experimental study shows that (1) lexical, syntactic andstructural properties of source code are encoded in the lower, intermediate,and higher layers, respectively, while the semantic property spans across theentire model. (2) The process of fine-tuning preserves most of the codeproperties. Specifically, the basic code properties captured by lower andintermediate layers are still preserved during fine-tuning. Furthermore, wefind that only the representations of the top two layers change most duringfine-tuning for various downstream tasks. (3) Based on the above findings, wepropose Telly to efficiently fine-tune pre-trained code models via layerfreezing. The extensive experimental results on five various downstream tasksdemonstrate that training parameters and the corresponding time cost aregreatly reduced, while performances are similar or better. Replication packageincluding source code, datasets, and online Appendix is available at:\url{https://github.com/DeepSoftwareAnalytics/Telly}.

Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond