English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Pre-trained language models (PLMs) have become a prevalent technique in deeplearning for code, utilizing a two-stage pre-training and fine-tuning procedureto acquire general knowledge about code and specialize in a variety ofdownstream tasks. However, the dynamic nature of software codebases poses achallenge to the effectiveness and robustness of PLMs. In particular,world-realistic scenarios potentially lead to significant differences betweenthe distribution of the pre-training and test data, i.e., distribution shift,resulting in a degradation of the PLM's performance on downstream tasks. Inthis paper, we stress the need for adapting PLMs of code to software data whosedistribution changes over time, a crucial problem that has been overlooked inprevious works. The motivation of this work is to consider the PLM in anon-stationary environment, where fine-tuning data evolves over time accordingto a software evolution scenario. Specifically, we design a scenario where themodel needs to learn from a stream of programs containing new, unseen APIs overtime. We study two widely used PLM architectures, i.e., a GPT2 decoder and aRoBERTa encoder, on two downstream tasks, API call and API usage prediction. Wedemonstrate that the most commonly used fine-tuning technique from prior workis not robust enough to handle the dynamic nature of APIs, leading to the lossof previously acquired knowledge i.e., catastrophic forgetting. To addressthese issues, we implement five continual learning approaches, includingreplay-based and regularization-based methods. Our findings demonstrate thatutilizing these straightforward methods effectively mitigates catastrophicforgetting in PLMs across both downstream tasks while achieving comparable orsuperior performance.

On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code