English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Pre-trained models of source code have recently been successfully applied toa wide variety of Software Engineering tasks; they have also seen somepractical adoption in practice, e.g. for code completion. Yet, we still knowvery little about what these pre-trained models learn about source code. Inthis article, we use probing--simple diagnostic tasks that do not further trainthe models--to discover to what extent pre-trained models learn about specificaspects of source code. We use an extensible framework to define 15 probingtasks that exercise surface, syntactic, structural and semantic characteristicsof source code. We probe 8 pre-trained source code models, as well as a naturallanguage model (BERT) as our baseline. We find that models that incorporatesome structural information (such as GraphCodeBERT) have a betterrepresentation of source code characteristics. Surprisingly, we find that forsome probing tasks, BERT is competitive with the source code models, indicatingthat there are ample opportunities to improve source-code specific pre-trainingon the respective code characteristics. We encourage other researchers toevaluate their models with our probing task suite, so that they may peer intothe hidden layers of the models and identify what intrinsic codecharacteristics are encoded.

INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers