Codebert: a Pre-Trained Model for Programming and Natural Languages
Total Page：16
File Type：pdf, Size：1020Kb
CodeBERT: A Pre-Trained Model for Programming and Natural Languages Zhangyin Feng1,∗ Daya Guo2,∗ Duyu Tang3, Nan Duan3, Xiaocheng Feng1 Ming Gong4, Linjun Shou4, Bing Qin1, Ting Liu1, Daxin Jiang4, Ming Zhou3 1 Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China 2 The School of Data and Computer Science, Sun Yat-sen University, China 3 Microsoft Research Asia, Beijing, China 4 Microsoft Search Technology Center Asia, Beijing, China fzyfeng,xcfeng,qinb,[email protected] [email protected] fdutang,nanduan,migon,lisho,djiang,[email protected] Abstract and RoBERTa (Liu et al., 2019) have dramati- cally improved the state-of-the-art on a variety of We present CodeBERT, a bimodal pre-trained natural language processing (NLP) tasks. These model for programming language (PL) and pre-trained models learn effective contextual repre- natural language (NL). CodeBERT learns sentations from massive unlabeled text optimized general-purpose representations that support by self-supervised objectives, such as masked downstream NL-PL applications such as nat- ural language code search, code documen- language modeling, which predicts the original tation generation, etc. We develop Code- masked word from an artificially masked input BERT with Transformer-based neural architec- sequence. The success of pre-trained models in ture, and train it with a hybrid objective func- NLP also drives a surge of multi-modal pre-trained tion that incorporates the pre-training task of models, such as ViLBERT (Lu et al., 2019) for replaced token detection, which is to detect language-image and VideoBERT (Sun et al., 2019) plausible alternatives sampled from generators. for language-video, which are learned from bi- This enables us to utilize both “bimodal” data modal bi- of NL-PL pairs and “unimodal” data, where data such as language-image pairs with the former provides input tokens for model modal self-supervised objectives. training while the latter helps to learn bet- In this work, we present CodeBERT, a bimodal ter generators. We evaluate CodeBERT on pre-trained model for natural language (NL) and two NL-PL applications by fine-tuning model programming language (PL) like Python, Java, parameters. Results show that CodeBERT JavaScript, etc. CodeBERT captures the seman- achieves state-of-the-art performance on both tic connection between natural language and pro- natural language code search and code docu- mentation generation. Furthermore, to inves- gramming language, and produces general-purpose tigate what type of knowledge is learned in representations that can broadly support NL-PL CodeBERT, we construct a dataset for NL-PL understanding tasks (e.g. natural language code probing, and evaluate in a zero-shot setting search) and generation tasks (e.g. code documen- where parameters of pre-trained models are tation generation). It is developed with the multi- arXiv:2002.08155v4 [cs.CL] 18 Sep 2020 fixed. Results show that CodeBERT performs layer Transformer (Vaswani et al., 2017), which is better than previous pre-trained models on NL- adopted in a majority of large pre-trained models. PL probing.1 In order to make use of both bimodal instances of NL-PL pairs and large amount of available uni- 1 Introduction modal codes, we train CodeBERT with a hybrid Large pre-trained models such as ELMo (Peters objective function, including standard masked lan- et al., 2018), GPT (Radford et al., 2018), BERT guage modeling (Devlin et al., 2018) and replaced (Devlin et al., 2018), XLNet (Yang et al., 2019) token detection (Clark et al., 2020), where uni- modal codes help to learn better generators for ∗Work done while this author was an intern at Microsoft producing better alternative tokens for the latter Research Asia. objective. 1 All the codes and data are available at https:// github.com/microsoft/CodeBERT We train CodeBERT from Github code reposito- ries in 6 programming languages, where bimodal annotation. Dominant learning objectives are lan- datapoints are codes that pair with function-level guage modeling and its variations. For example, natural language documentations (Husain et al., in GPT (Radford et al., 2018), the learning objec- 2019). Training is conducted in a setting similar tive is language modeling, namely predicting the to that of multilingual BERT (Pires et al., 2019), next word wk given the preceding context words in which case one pre-trained model is learned for fw1; w2; :::; wk−1g. As the ultimate goal of pre- 6 programming languages with no explicit mark- training is not to train a good language model, it is ers used to denote the input programming lan- desirable to consider both preceding and following guage. We evaluate CodeBERT on two down- contexts to learn better general-purpose contextual stream NL-PL tasks, including natural language representations. This leads us to the masked lan- code search and code documentation generation. guage modeling objective used in BERT (Devlin Results show that fine-tuning the parameters of et al., 2018), which learns to predict the masked CodeBERT achieves state-of-the-art performance words of a randomly masked word sequence given on both tasks. To further investigate what type of surrounding contexts. Masked language modeling knowledge is learned in CodeBERT, we construct is also used as one of the two learning objectives a dataset for NL-PL probing, and test CodeBERT for training CodeBERT. in a zero-shot scenario, i.e. without fine-tuning the parameters of CodeBERT. We find that CodeBERT 2.2 Multi-Modal Pre-Trained Models consistently outperforms RoBERTa, a purely natu- ral language-based pre-trained model. The contri- The remarkable success of the pre-trained model butions of this work are as follows: in NLP has driven the development of multi-modal pre-trained model that learns implicit alignment • CodeBERT is the first large NL-PL pre- between inputs of different modalities. These mod- trained model for multiple programming lan- els are typically learned from bimodal data, such guages. as pairs of language-image or pairs of language- video. For example, ViLBERT (Lu et al., 2019) • Empirical results show that CodeBERT is ef- learns from image caption data, where the model fective in both code search and code-to-text learns by reconstructing categories of masked im- generation tasks. age region or masked words given the observed inputs, and meanwhile predicting whether the cap- • We further created a dataset which is the first tion describes the image content or not. Simi- one to investigate the probing ability of the larly, VideoBERT (Sun et al., 2019) learns from code-based pre-trained models. language-video data and is trained by video and 2 Background text masked token prediction. Our work belongs to this line of research as we regard NL and PL 2.1 Pre-Trained Models in NLP as different modalities. Our method differs from Large pre-trained models (Peters et al., 2018; Rad- previous works in that the fuels for model train- ford et al., 2018; Devlin et al., 2018; Yang et al., ing include not only bimodal data of NL-PL pairs, 2019; Liu et al., 2019; Raffel et al., 2019) have but larger amounts of unimodal data such as codes brought dramatic empirical improvements on al- without paired documentations. most every NLP task in the past few years. Suc- A concurrent work (Kanade et al., 2019) uses cessful approaches train deep neural networks on masked language modeling and next sentence pre- large-scale plain texts with self-supervised learning diction as the objective to train a BERT model on objectives. One of the most representative neural Python source codes, where a sentence is a log- architectures is the Transformer (Vaswani et al., ical code line as defined by the Python standard. 2017), which is also the one used in this work. It In terms of the pre-training process, CodeBERT contains multiple self-attention layers, and can be differs from their work in that (1) CodeBERT is conventionally learned with gradient decent in an trained in a cross-modal style and leverages both end-to-end manner as every component is differen- bimodal NL-PL data and unimodal PL/NL data, (2) tiable. The terminology “self-supervised” means CodeBERT is pre-trained over six programming that supervisions used for pre-training are auto- languages, and (3) CodeBERT is trained with a matically collected from raw data without manual new learning objective based on replaced token detection. TRAINING DATA bimodal DATA unimodal CODES GO 319,256 726,768 3 CodeBERT JAVA 500,754 1,569,889 JAVASCRIPT 143,252 1,857,835 We describe the details about CodeBERT in this PHP 662,907 977,821 section, including the model architecture, the input PYTHON 458,219 1,156,085 RUBY 52,905 164,048 and output representations, the objectives and data ALL 2,137,293 6,452,446 used for training CodeBERT, and how to fine-tune CodeBERT when it is applied to downstream tasks. Table 1: Statistics of the dataset used for training Code- BERT. 3.1 Model Architecture We follow BERT (Devlin et al., 2018) and provided by Husain et al.(2019), which includes RoBERTa (Liu et al., 2019), and use multi-layer 2.1M bimodal datapoints and 6.4M unimodal codes bidirectional Transformer (Vaswani et al., 2017) as across six programming languages (Python, Java, the model architecture of CodeBERT. We will not JavaScript, PHP, Ruby, and Go). Data statistics is review the ubiquitous Transformer architecture in shown in Table1. 2 detail. We develop CodeBERT by using exactly the The data comes from publicly available open- same model architecture as RoBERTa-base. The source non-fork GitHub repositories and are fil- total number of model parameters is 125M. tered with a set of constraints and rules.