Characterizing The Influence of Continuous Integration Empirical Results from 250+ Open Source and Proprietary Projects Akond Rahman, Amritanshu Agrawal, Rahul Krishna, and Alexander Sobran* North Carolina State University, IBM Corporation* [aarahman,aagrawa8,rkrish11]@ncsu.edu,[email protected]* ABSTRACT 1 INTRODUCTION Continuous integration (CI) tools integrate code changes by auto- Continuous integration (CI) tools integrate code changes by auto- matically compiling, building, and executing test cases upon submis- matically compiling, building, and executing test cases upon sub- sion of code changes. Use of CI tools is getting increasingly popular, mission of code changes [9]. In recent years, usage of CI tools have yet how proprietary projects reap the benefits of CI remains un- become increasingly popular both for open source software (OSS) known. To investigate the influence of CI on software development, projects [3][12][25] as well as for proprietary projects [24]. we analyze 150 open source software (OSS) projects, and 123 propri- Our industrial partner adopted CI to improve their software de- etary projects. For OSS projects, we observe the expected benefits velopment process. Our industrial partner’s expectation was that after CI adoption, e.g., improvements in bug and issue resolution. similar to OSS projects [27][25], CI would positively influence However, for the proprietary projects, we cannot make similar ob- resolution of bugs and issues for projects owned by our industrial servations. Our findings indicate that only adoption of CI might partner. Our industrial partner also expected collaboration to in- not be enough to the improve software development process. CI crease upon adoption of CI. Being one of the primary Extreme can be effective for software development if practitioners use CI’s Programming (XP) practices [2], CI is expected to benefit collabo- feedback mechanism efficiently, by applying the practice of making ration amongst team members [23]. frequent commits. For our set of proprietary projects we observe We conduct an empirical study to investigate if our industrial practitioners commit less frequently, and hence not use CI effec- partner’s expectations were fulfilled. Such an empirical study can be tively for obtaining feedback on the submitted code changes. Based beneficial in the following ways: (i) to quantify if CI benefits projects on our findings we recommend industry practitioners to adopt the with respect to bug and issue resolution, along with collaboration; best practices of CI to reap the benefits of CI tools for example, and (ii) to derive lessons that industry practitioners should keep making frequent commits. in mind when using CI. We conduct an empirical study with 150 OSS and 123 proprietary projects to quantify the influence of CI on CCS CONCEPTS bug resolution, collaboration, and issue resolution. We answer the following research questions: • Software and its engineering → Agile software develop- ment; RQ1: Does adoption of continuous integration influence com- KEYWORDS mit patterns? Commit frequency and sizes significantly increases for OSS projects after CI adoption but not for our set of proprietary Continuous Integration, DevOps, GitHub, Mining Software Reposi- projects. tories, Software Development Practice RQ2: How does adoption of continuous integration influence collaboration amongst team members? After adopting CI, col- ACM Reference format: laboration significantly increases for both, OSS and our setof Akond Rahman, Amritanshu Agrawal, Rahul Krishna, and Alexander So- proprietary projects. The increase in collaboration is more observ- bran*. 2018. Characterizing The Influence of Continuous Integration. In Proceedings of Proceedings of the 4th ACM SIGSOFT International Workshop able for OSS projects than the proprietary projects. RQ3: How does adoption of continuous integration influence arXiv:1711.03933v4 [cs.SE] 14 Sep 2018 on Software Analytics, Lake Buena Vista, FL, USA, November 5, 2018 (SWAN ’18), 7 pages. bug and issue resolution? Significantly more bugs and issues https://doi.org/10.1145/3278142.3278149 are resolved after adoption of CI for OSS projects, but not for our set of proprietary projects. Permission to make digital or hard copies of all or part of this work for personal or In summary, we observe usage of CI to be beneficial for OSS classroom use is granted without fee provided that copies are not made or distributed projects but not for our set of proprietary projects. For proprietary for profit or commercial advantage and that copies bear this notice and the full citation projects, we acknowledge that there may be benefits to CI which are on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, not captured by our study, for example, cultural benefits in adopting to post on servers or to redistribute to lists, requires prior specific permission and/or a CI tools. Findings from our paper can help industry practitioners fee. Request permissions from [email protected]. SWAN ’18, November 5, 2018, Lake Buena Vista, FL, USA revise their expectations about the benefits of CI. Our paper may © 2018 Association for Computing Machinery. also help to identify possible strategies to fully reap the benefits of ACM ISBN 978-1-4503-6056-2/18/11...$15.00 CI. https://doi.org/10.1145/3278142.3278149 SWAN ’18, November 5, 2018, Lake Buena Vista, FL, USA A. Rahman, A. Agrawal, R. Krishna, and A. Sobran Travis CI Jenkins Version Control Systems Build Results as Feedback Item #1 Build & Test Execu/on Item #2 CI Tools Programmers Item #3 Build Results Figure 1: An example work-flow of the continuous integration (CI) process. 2 BACKGROUND and identified three trade-offs to adopt CI: assurance, flexibility, We first provide a brief background on CI, then describe prior and security. Rahman et al. [18] observed that adoption of CI is not research work related to CI. wide-spread amongst practitioners. They investigated which diffu- sion of innovation (DOI) factors influence adoption of CI tools, and 2.1 About Continuous Integration (CI) reported four factors: relative advantages, compatibility, complexity, and education. CI is identified as one of the primary practices to implement XP[2]. Usage: Beller et al. [3] collected and analyzed Java and Ruby- According to Duvall et al. [9], CI originated from the imperatives based projects from Github, and synthesized the nature of build and of agility, in order to respond to customer requests quickly. When test attributes exhibited amongst OSS projects that use CI. Vasilescu building the source code, CI tools can execute unit and integration et al. [25] analyzed OSS GitHub projects that use Travis CI, and tests to ensure quality of the integrated source code. If the tests do reported that adoption of CI increases productivity for OSS projects. not pass, CI tools can be customized to give feedback on the submit- Zhao et al. [27] mined OSS GitHub projects, and investigated if ted code changes. Even though the concept of CI was introduced in software development practices such as commit frequency, commit 2006, initial usage of CI was not popular amongst practitioners [8]. size, and pull request handling, changes after adoption of CI. However, since 2011, with the advent of CI tools such as Travis The above-mentioned findings highlight the community’s inter- CI [6], usage of CI has increased in recent years [12]. est in how CI is being used in software projects. From the above- When a software team adopts CI, the team has to follow a set of mentioned prior work, we can list the following as exemplars of practices [9]. According to the CI methodology all programmers the expected benefits of adopting CI: have to check-in their code daily, which are integrated daily [9]. Unlike, traditional methodologies such as waterfall, in CI, program- • Zhao et al. [27] reported that for OSS GitHub projects, the number mers get instant feedback on their code via build results. To im- of closed issues increases after the adoption of CI tools. plement CI, the team must maintain its source code in a version • Vasilescu et al. [25] reported that for OSS GitHub projects, num- control system (VCS), and integrate the VCS with the CI tool so that ber of bugs do not increase after adoption of CI. builds are triggered upon submission of each commit [9]. Figure1 Note that all of these findings are derived from OSS projects. provides an example on how a typical CI process works. Program- With respect to the development process, structure, and complexity, mer make commits in a repository maintained by a VCS such as, proprietary projects are different from OSS projects [17][20], which GitHub, and these commits trigger CI jobs on a CI tool such as motivates us to pursue our research study. Hence, for the rest of Travis CI which executes, builds, tests, and produces build results. this paper, we will compare the influence of adopting CI within These build results are provided to the programmers as a feedback OSS and our set of proprietary projects. We consider the following either through e-mails, or phone alerts [9] on their submitted code attributes of software development: bug resolution, collaboration changes. Based on the build results, programmers make necessary amongst team members, commit patterns, and issue resolution. changes to their code, and repeats the CI process again. 3 METHODOLOGY 2.2 Related Work In this section, we describe our methodology to filter datasets, Our paper is closely related to prior research that have investigated followed by metrics and statistical measurements that we use to usage of CI tools. We briefly describe these prior work as following. answer our research questions. Adoption: Hilton et al. [12] mined OSS projects hosted on Github. They observed that most popular projects use CI, and re- 3.1 Filtering ported that the median time of CI adoption is one year.