DOCSLIB.ORG

An Empirical Cybersecurity Evaluation of Github Copilot's Code Contributions

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
An Empirical Cybersecurity Evaluation of Github Copilot's Code Contributions An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri Abstract—There is burgeoning interest in designing AI-based model currently available, it is important to understand: Are systems to assist humans in designing computing systems, Copilot’s suggestions commonly insecure? What is the including tools that automatically generate computer code. prevalence of insecure generated code? What factors of the The most notable of these comes in the form of the first self- described ‘AI pair programmer’, GitHub Copilot, a language “context” yield generated code that is more or less secure? model trained over open-source GitHub code. However, code We systematically experiment with Copilot to gain insights often contains bugs—and so, given the vast quantity of unvetted into these questions by designing scenarios for Copilot to code that Copilot has processed, it is certain that the language complete and by analyzing the produced code for security model will have learned from exploitable, buggy code. This weaknesses. As our corpus of well-defined weaknesses, we raises concerns on the security of Copilot’s code contributions. In this work, we systematically investigate the prevalence and check Copilot completions for a subset of MITRE’s “2021 conditions that can cause GitHub Copilot to recommend insecure CWE Top 25 Most Dangerous Software Weaknesses” [4], code. To perform this analysis we prompt Copilot to generate a list that is updated yearly to indicate the most dangerous code in scenarios relevant to high-risk CWEs (e.g. those from software weaknesses as measured over the previous two MITRE’s “Top 25” list). We explore Copilot’s performance on calendar years. The AI’s documentation recommends that three distinct code generation axes—examining how it performs given diversity of weaknesses, diversity of prompts, and diversity one uses “Copilot together with testing practices and security of domains. In total, we produce 89 different scenarios for tools, as well as your own judgment”. Our work attempts Copilot to complete, producing 1,692 programs. Of these, we to characterize the tendency of Copilot to produce insecure found approximately 40 % to be vulnerable. code, giving a gauge for the amount of scrutiny a human Index Terms—Cybersecurity, AI, code generation, CWE developer might need to do for security issues. We study Copilot’s behavior along three dimensions: (1) I. INTRODUCTION diversity of weakness, its propensity for generating code that With increasing pressure on software developers to produce is susceptible to each of weaknesses in the CWE top 25, given code quickly, there is considerable interest in tools and a scenario where such a vulnerability is possible; (2) diversity techniques for improving productivity. The most recent of prompt, its response to the context for a particular scenario entrant into this field is machine learning (ML)-based (SQL injection), and (3) diversity of domain, its response to code generation, in which large models originally designed the domain, i.e., programming language/paradigm. for natural language processing (NLP) are trained on vast For diversity of weakness, we construct three different quantities of code and attempt to provide sensible completions scenarios for each applicable top-25 CWE and use CodeQL [5] as programmers write code. In June 2021, GitHub released along with manual inspection to assess whether the Copilot [1], an “AI pair programmer” that generates code in suggestions returned are vulnerable to that CWE. Our goal a variety of languages given some context such as comments, here is to get a broad overview of the types of vulnerability function names, and surrounding code. Copilot is built on a Copilot is most likely to generate, and how often users might large language model that is trained on open-source code [2] encounter such insecure suggestions. Next, we investigate including “public code...with insecure coding patterns”, thus the effect different prompts have on how likely Copilot is to giving rise to the potential for “synthesize[d] code that arXiv:2108.09293v2 [cs.CR] 23 Aug 2021 return suggestions that are vulnerable to SQL injection. This contains these undesirable patterns” [1]. investigation allows us to better understand what patterns Although prior research has evaluated the functionality of programmers may wish to avoid when using Copilot, or ways code generated by language models [3], [2], there is no to help guide it to produce more secure code. systematic examination of the security of ML-generated code. Finally, we study the security of code generated by Copilot As GitHub Copilot is the largest and most capable such when it is used for a domain that was less frequently seen H. Pearce is with the Dept. of Electrical and Computer Engineering, New in its training data. Copilot’s marketing materials claim that York University, NY 11201 USA (e-mail: [email protected]) it speaks “all the languages one loves.” To test this claim, we B. Ahmad is with the Dept. of Electrical and Computer Engineering, New focus on Copilot’s behavior when tasked with a new domain York University, NY 11201 USA (e-mail: [email protected]) B. Tan is is with the Dept. of Electrical and Computer Engineering, New added to the MITRE CWEs in 2020—hardware-specific York University, NY 11201 USA (e-mail: [email protected]) CWEs [6]. As with the software CWEs, hardware designers B. Dolan-Gavitt is with the Dept. of Computer Science and Engineering, can be sure that their designs meet a certain baseline level New York University, NY 11201 USA (e-mail: [email protected]) R. Karri is with the Dept. of Electrical and Computer Engineering, New of security if their designs are free of hardware weaknesses. York University, NY 11201 USA (e-mail: [email protected]) We are interested in studying how Copilot performs when tasked with generating register-transfer level (RTL) code in was limited to one type of weakness (insecure crypto the hardware description language Verilog. parameters, namely short RSA key sizes and using AES in Our contributions include the following. We perform ECB mode). The authors note that “a larger study using the automatic and manual analysis of Copilot’s software and most common insecure code vulnerabilities” is needed, and hardware code completion behavior in response to “prompts” we supply such an analysis here. handcrafted to represent security-relevant scenarios and An important feature that Codex and Copilot inherit from characterize the impact that patterns in the context can have GPT-3 is that, given a prompt, they generate the most likely on the AI’s code generation and confidence. We discuss completion for that prompt based on what was seen during implications for software and hardware designers, especially training. In the context of code generation, this means that security novices, when using AI pair programming tools. the model will not necessarily generate the best code (by This work is accompanied by the release of our repository of whatever metric you choose—performance, security, etc.) but security-relevant scenarios (see the Appendix). rather the one that best matches the code that came before. As a result, the quality of the generated code can be strongly II. BACKGROUND AND RELATED WORK influenced by semantically irrelevant features of the prompt. A. Code Generation We explore the effect of different prompts in Section V-C. Software development involves the iterative refinement of a (plain language) specification into a software B. Evaluating Code Security implementation—developers write code, comments, and Numerous elements determine the quality of code. Code other supporting collateral as they work towards a functional generation literature emphasizes functional correctness, product. Early work proposed ML-based tools to support measured by compilation and checking against unit tests, developers through all stages of the software design life-cycle or using text similarity metrics to desired responses [2]. (e.g., predicting designer effort, extracting specifications [7]). Unlike metrics for functional correctness of generated code, With recent advancements in the domain of deep learning (DL) evaluating the security of code contributions made by Copilot and NLP, sophisticated models can perform sophisticated is an open problem. Aside from manual assessment by a interventions on a code base, such as automated program human security expert there are myriad tools and techniques repair [8]. In this work, we focus on Copilot as an “AI to perform security analyses of software [20]. Source code pair programmer” that offers a designer code completion analysis tools, known as Static Application Security Testing suggestions in “real-time” as they write code in a text editor. (SAST) Tools, are designed to analyze source code and/or There are many efforts to automatically translate compiled versions of code to find security flaws; typically specifications into computer code for natural language they specialize on identifying a specific vulnerability class. programming [9], through formal models for automatic code In this work, we gauge the security of Copilot’s contri- generation (e.g., [10], [11]) or via machine-learned NLP butions using a mix of automated analysis using GitHub’s [12]. DL architectures that demonstrate good fits for NLP CodeQL tool [5] (as it can scan for a wider range of security include LSTMs [13], RNNs [14], and Transformers [15] that weaknesses in code compared to other tools) alongside our have paved the way for models such as BERT [16], GPT- manual code inspection. CodeQL is open-source and supports 2 [17], and GPT-3 [18]. These models can perform language the analysis of software in languages such as Java, JavaScript, tasks such as translation and answering questions from the C++, C#, and Python. Through queries written in its QL query CoQA [19] dataset; after fine-tuning on specialized datasets, language, CodeQL can find issues in codebases based on a set the models can undertake tasks such as code completion [2].
Load more

Recommended publications
  • "This Book Was a Joy to Read. It Covered All Sorts of Techniques for Debugging, Including 'Defensive' Paradigms That Will Eliminate Bugs in the First Place
    Perl Debugged By Peter Scott, Ed Wright Publisher : Addison Wesley Pub Date : March 01, 2001 ISBN : 0-201-70054-9 Table of • Pages : 288 Contents "This book was a joy to read. It covered all sorts of techniques for debugging, including 'defensive' paradigms that will eliminate bugs in the first place. As coach of the USA Programming Team, I find the most difficult thing to teach is debugging. This is the first text I've even heard of that attacks the problem. It does a fine job. Please encourage these guys to write more." -Rob Kolstad Perl Debugged provides the expertise and solutions developers require for coding better, faster, and more reliably in Perl. Focusing on debugging, the most vexing aspect of programming in Perl, this example-rich reference and how-to guide minimizes development, troubleshooting, and maintenance time resulting in the creation of elegant and error-free Perl code. Designed for the novice to intermediate software developer, Perl Debugged will save the programmer time and frustration in debugging Perl programs. Based on the authors' extensive experience with the language, this book guides developers through the entire programming process, tackling the benefits, plights, and pitfalls of Perl programming. Beginning with a guided tour of the Perl documentation, the book progresses to debugging, testing, and performance issues, and also devotes a chapter to CGI programming in Perl. Throughout the book, the authors espouse defensible paradigms for improving the accuracy and performance of Perl code. In addition, Perl Debugged includes Scott and Wright's "Perls of Wisdom" which summarize key ideas from each of the chapters, and an appendix containing a comprehensive listing of Perl debugger commands.
    [Show full text]
  • A Style Guide
    How to Program Racket: a Style Guide Version 6.11 Matthias Felleisen, Matthew Flatt, Robby Findler, Jay McCarthy October 30, 2017 Since 1995 the number of “repository contributors” has grown from a small handful to three dozen and more. This growth implies a lot of learning and the introduction of inconsistencies of programming styles. This document is an attempt leverage the former and to start reducing the latter. Doing so will help us, the developers, and our users, who use the open source code in our repository as an implicit guide to Racket programming. To help manage the growth our code and showcase good Racket style, we need guidelines that shape the contributions to the code base. These guidelines should achieve some level of consistency across the different portions of the code base so that everyone who opens files can easily find their way around. This document spells out the guidelines. They cover a range of topics, from basic work (commit) habits to small syntactic ideas like indentation and naming. Many pieces of the code base don’t live up to the guidelines yet. Here is how we get started. When you start a new file, stick to the guidelines. If you need to edit a file, you will need to spend some time understanding its workings. If doing so takes quite a while due to inconsistencies with the guidelines, please take the time to fix (portions of) the file. After all, if the inconsistencies throw you off for that much time, others are likely to have the same problems.
    [Show full text]
  • Coding Style Guidelines
    Coding Style Guidelines Coding Style Guidelines Introduction This document was created to provide Xilinx users with a guideline for producing fast, reliable, and reusable HDL code. Table of Contents Top-Down Design ⎯ Section 1 ................................................................13-4 Behavioral and Structural Code...................................................................13-4 Declarations, Instantiations, and Mappings.................................................13-5 Comments ...................................................................................................13-6 Indentation...................................................................................................13-9 Naming Conventions ...................................................................................13-10 Signals and Variables ⎯ Section 2..........................................................13-13 Signals.........................................................................................................13-13 Casting.........................................................................................................13-13 Inverted Signals...........................................................................................13-13 Rule for Signals ...........................................................................................13-14 Rules for Variables and Variable Use .........................................................13-15 Packages ⎯ Section 3 ..............................................................................13-17
    [Show full text]
  • Coding Style
    Coding Style by G.A. (For internal Use Only. Do not distribute) April 23, 2018 1 Organizational 1.1 Codes and Automation Codes to which these rules apply are the following. DMRG++, PsimagLite, Lanczos++, FreeFermions, GpusDoneRight, SPFv7, and MPS++. The script indentStrict.pl can be used to automate the application of these rules, as shown in Listing1, where the option -fix will fix most problems in place. Listing 1: Use of the indentStrict.pl script. PsimagLite/scripts/indentStrict.pl -file file.h [-fix] 1.2 Indentation Indent using “hard” tabs and Kernighan and Ritchie indentation. This is the indentation style used by the Linux Kernel. Note that the tab character is octal 011, decimal 9, hex. 09, or “nt” (horizontal tab). You can represent tabs with any number of spaces you want. Regardless, a tab is saved to disk as a single character: hex. 09, and *not* as 8 spaces. When a statement or other line of code is too long and follows on a separate line, do not add an indentation level with tab(s), but use spaces for aligment. For an example see Listing2, where tabs (indentation levels) are indicated by arrows. Listing 2: Proper use of continuation lines. class␣A␣{ −−−−−−−!void␣thisIsAVeryVeryLongFunction(const␣VeryVeryLongType&␣x, −−−−−−−!␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣const␣VeryVeryLongType&␣y) −−−−−−−!{ −−−−−−−→−−−−−−−!//␣code␣here −−−−−−−!} I repeat the same thing more formally now: Indent following the Linux coding style, with exceptions due to the Linux style applying to C, but here we’re talking about code written in C++. 1. Classes are indented like this: 1 1.2 Indentation class A { }; That is, they are indented as for loops and not as functions.
    [Show full text]
  • The Pascal Programming Language
    The Pascal Programming Language http://pascal-central.com/ppl/chapter2.html The Pascal Programming Language Bill Catambay, Pascal Developer Chapter 2 The Pascal Programming Language by Bill Catambay Return to Table of Contents II. The Pascal Architecture Pascal is a strongly typed, block structured programming language. The "type" of a Pascal variable consists of its semantic nature and its range of values, and can be expressed by a type name, an explicit value range, or a combination thereof. The range of values for a type is defined by the language itself for built-in types, or by the programmer for programmer defined types. Programmer-defined types are unique data types defined within the Pascal TYPE declaration section, and can consist of enumerated types, arrays, records, pointers, sets, and more, as well as combinations thereof. When variables are declared as one type, the compiler can assume that the variable will be used as that type throughout the life of the variable (whether it is global to the program, or local to a function or procedure). This consistent usage of variables makes the code easier to maintain. The compiler detects type inconsistency errors at compile time, catching many errors and reducing the need to run the code through a debugger. Additionally, it allows an optimizer to make assumptions during compilation, thereby providing more efficient executables. As John Reagan, the architect of Compaq Pascal, writes, "it was easy to write Pascal programs that would generate better code than their C equivalents" because the compiler was able to optimize based on the strict typing.
    [Show full text]
  • Application Note C/C++ Coding Standard
    Application Note C/C++ Coding Standard Document Revision J April 2013 Copyright © Quantum Leaps, LLC www.quantum-leaps.com www.state-machine.com Table of Contents 1 Goals..................................................................................................................................................................... 1 2 General Rules....................................................................................................................................................... 1 3 C/C++ Layout........................................................................................................................................................ 2 3.1 Expressions................................................................................................................................................... 2 3.2 Indentation..................................................................................................................................................... 3 3.2.1 The if Statement.................................................................................................................................. 4 3.2.2 The for Statement............................................................................................................................... 4 3.2.3 The while Statement........................................................................................................................... 4 3.2.4 The do..while Statement.....................................................................................................................5
    [Show full text]
  • TT284 Web Technologies Block 2 Web Architecture an Introduction to Javascript
    TT284 Web technologies Block 2 Web Architecture An Introduction to JavaScript Prepared for the module team by Andres Baravalle, Doug Briggs, Michelle Hoyle and Neil Simpkins Introduction 3 Starting with JavaScript “Hello World” 3 Inserting JavaScript into Web Pages 3 Editing tools 4 Browser tools 5 Lexical structure of JavaScript 5 Semicolons 6 Whitespace 6 Case sensitivity 6 Comments 7 Literals 8 Variables and data types 8 Variables as space for data 9 Declaring variables and storing values 10 Expressions 11 Operators 11 Assignment operators 11 Arithmetic operators 12 Comparison operators 12 Logical operators 12 String operators 13 Operator precedence 14 Operators and type conversions 15 Debugging JavaScript 16 Statements 17 Copyright © 2012 The Open University TT284 Web technologies Conditional statements 18 Loop statements 19 Arrays 20 Working with arrays 21 Objects 22 Functions 23 Writing functions 24 Variable scope 25 Constructors 26 Methods 27 Coding guidelines 28 Why do we have coding guidelines? 28 Indentation style and layout 28 ‘Optional’ semicolons and ‘var’s 29 Consistent and mnemonic naming 29 Make appropriate comments 30 Reusable code 30 Other sources of information 31 Web resources 31 TT284 Block 2 An Introduction to JavaScript | 2 TT284 Web technologies Introduction This guide is intended as an aid for anyone unfamiliar with basic JavaScript programming. This guide doesn’t include information beyond that required to complete TT284 (there is very little on object orientated coding for example). To use this guide you must be fully familiar with HTML and use of a web browser. For many TT284 students this guide will be unnecessary or will serve perhaps as a refresher on programming after completing a module such as TU100, perhaps some time ago.
    [Show full text]
  • Library of Congress Cataloging-In-Publication Data
    Library of Congress Cataloging-in-Publication Data Carrano, Frank M. 39 Data abstraction and problem solving with C++: walls and mirrors / Frank M. Carrano, Janet J. Prichard.—3rd ed. p. cm. ISBN 0-201-74119-9 (alk. paper) 1. C++ (Computer program language) 2. Abstract data types. 3. Problem solving—Data Processing. I. Prichard, Janet J. II. Title. QA76.73.C153 C38 2001 005.7'3—dc21 2001027940 CIP Copyright © 2002 by Pearson Education, Inc. Readability. For a program to be easy to follow, it should have a good structure and design, a good choice of identifiers, good indentation and use of blank lines, and good documentation. You should avoid clever programming tricks that save a little computer time at the expense of much human time. You will see examples of these points in programs throughout the book. Choose identifiers that describe their purpose, that is, are self- documenting. Distinguish between keywords, such as int, and user- defined identifiers. This book uses the following conventions: • Keywords are lowercase and appear in boldface. Identifier style • Names of standard functions are lowercase. • User-defined identifiers use both upper- and lowercase letters, as follows: • Class names are nouns, with each word in the identifier capi- talized. • Function names within a class are verbs, with the first letter lowercase and subsequent internal words capitalized. • Variables begin with a lowercase letter, with subsequent words in the identifier capitalized. • Data types declared in a typedef statement and names of structures and enumerations each begin with an uppercase letter. • Named constants and enumerators are entirely uppercase and use underscores to separate words.
    [Show full text]
  • A Brief Emacs Walkthrough
    Lab Handout: Debugging A Brief Emacs Walkthrough Handout 0. Objectives & Topics This document is designed to give an overview of some of the more advanced features of Emacs, including: 1. A review of basic commands (save, close, open, etc.) 2. Movement commands 3. Handy Tools 4. Configuration Files A thorough understanding of the commands in this walkthrough should allow you to create, compile, and run your programs without ever needing to close Emacs. This is by no means a comprehensive look at Emacs, but it should be enough to get you started with some of the more advanced features of the program. For more information, checkout the online Emacs manual: http://www.gnu.org/software/emacs/manual/. 1. The Basics Autocompleting Commands Before you start reading anything else, there’s one tangentially related tip that’s incredibly useful for navigating both UNIX and Emacs: tab completion. Tab completion is just a fancy name for auto completing prompts by typing in a partial name, hitting tab, and letting the computer do the rest of the work for you. For example, if there were three folders in the current directory named “HelloWorld.c”, “HelloWorld.h”, and “SomeFolder”, then you could type ‘cd So’ and hit tab. This would fill in the rest of the command for you since there’s only one possible folder that begins with ‘So’. Now consider typing “emacs He” and hitting tab. This will autocomplete the file name until it’s ambiguous. That is, because there are two files that start with “He”, UNIX will fill in as much of the prompt as it can (“emacs HelloWorld.”) leaving you to type either “c” or “h” to complete the name the rest of the way.
    [Show full text]
  • Programming Domains Programming Domains (2)
    Topics • Big Picture Programming Languages • Where Scheme, C, C++, Java fit in History • Fortran • Algol 60 CSE 413, Autumn 2007 11-02-2007 Donald Knuth: » Programming is the art of telling another human being what one wants the computer to do. 1 2 Programming Domains Programming Domains (2) • Scientific Applications: • Parallel Programming: » Using The Computer As A Large Calculator » Parallel And Distributed Systems » FORTRAN, Mathematica » Ada, CSP, Modula • Artificial Intelligence: • Business Applications: » uses symbolic rather than numeric computations » Data Processing And Business Procedures » lists as main data structure, flexibility (code = data) » COBOL, Some PL/I, Spreadsheets » Lisp 1959, Prolog 1970s • Systems Programming: • Scripting Languages: » Building Operating Systems And Utilities » A list of commands to be executed » C, C++ » UNIX shell programming, awk, tcl, perl 3 4 Programming Domains (3) C History • Education: • Developed in early 1970s » Languages Designed Just To Facilitate Teaching • Purpose: implementing the Unix operating » Pascal, BASIC, Logo system • The C Programming Language, aka `K&R‘, for its authors Brian Kernighan & Dennis Ritchie, is canonical reference (1978) • Evolution: ANSI, C99 5 6 1 Lisp & Scheme History Lisp: • McCarthy (1958/1960) for symbolic processing • Based on Chruch’s Lambda Calculus • 2 main Dialects: Scheme & Common Lisp • Aside: Assembly language macros for the IBM 704 : car (Contents of Address Register) and cdr (Contents of Decrement Register) Scheme: • Developed by Guy Steele and Gerald Sussman in the 1970s • Smaller & cleaner than Lisp • Static scoping, tail recursion implemented efficiently 7 8 Before High Level Languages • Early computers (40’s) programmed directly using numeric codes Fortran • Then move to assembly language (requires an assembler) • Assembly language – slightly more friendly version of machine code.
    [Show full text]
  • Princeton University COS 217: Introduction to Programming Systems Emacs Tutorial and Reference
    Princeton University COS 217: Introduction to Programming Systems Emacs Tutorial and Reference Part 1: Tutorial This tutorial describes how to use a minimal subset of the emacs editor. See the emacs reference in Part 2 of this document for more information. Also see the GNU Emacs Tutorial document at http://www.gnu.org/software/emacs/tour. Throughout the tutorial text in boldface indicates hands-on activities. The tutorial assumes that you've copied the file /u/cos217/.emacs to your home directory, as described in the A Minimal COS 217 Computing Environment handout from our first precept. It also assumes that you've copied files named hello.c, circle.c, and testintmath.c into your working directory. (Those files contain C programs that we'll describe in upcoming precepts.) Those files are available in the directory /u/cos217/emacstestfiles. You can issue this command to copy them to your working directory: cp /u/cos217/emacstestfiles/* . Background The emacs editor was created in the mid 1970s by Richard Stallman. Originally it was a set of "editing macros" for an editor that now is extinct. The emacs editor is popular, for a few reasons. It is: • Free. It's a component of the GNU tool set from the Free Software Foundation. • Highly customizable. Emacs is written in the LISP programming language, and is easy to customize via that language. • Integrated with other GNU software. In particular, emacs is integrated with the Bash history mechanism. Essentially you can think of the Bash history list as a "file"; you can use Emacs commands to scroll through and edit that file, and thereby easily reissue previous commands or variants thereof.
    [Show full text]
  • Code Style Guide
    CS 105 & CS106 Introduction to Computer Programming 1 & 2 Code Style Guide Last revised: September 30, 2019 Introduction The code you submit for labs and assignments is made more readable by paying attention to style. This includes using semicolons, declaring variables, whitespace, placement of comments, and choice of variable and function names. None of these affect the way the program executes, but they do affect the readability of your code. Just like English, computer code is a language. The main goal of all languages is to facilitate communication. When writing code, you are not only communicating with a computer, but also with yourself (often your “future” self) and other people reading your code. In industry, those other people are typically other programmers. In this course, those other people will be your instructor and your TAs who are leading labs and marking your work. This is why it is important to consider the needs of human readers and present code in a way that most clearly communicates what the program does. This is just like how we use indentations, commas, and periods to signify the start of a new paragraph, a pause in a sentence, or the end of a sentence. Here is an example of the last two paragraphs without any common English language style rules (capital letters, commas, periods, breaks for paragraphs, etc.). Try reading it: just like english computer code is a language the main goal of all languages is to facilitate communication when writing code you are not only communicating with a computer but also with yourself
    [Show full text]