Trade-Os in Continuous Integration: Assurance, Security, and Flexibility Michael Hilton Nicholas Nelson Timothy Tunnell Oregon State University, USA Oregon State University, USA University of Illinois, USA mhilton@cmu:edu nelsonni@oregonstate:edu tunnell2@illinois:edu Darko Marinov Danny Dig University of Illinois, USA Oregon State University, USA marinov@illinois:edu digd@oregonstate:edu 1 INTRODUCTION Continuous integration (CI) systems automate the compilation, ABSTRACT building, and testing of software. CI usage is widespread throughout Continuous integration (CI) systems automate the compilation, the software development industry. For example, the “State of Agile” building, and testing of software. Despite CI being a widely used industry survey [51], with 3,880 participants, found half of the activity in software engineering, we do not know what motivates respondents use CI. The “State of DevOps” report [33], a survey developers to use CI, and what barriers and unmet needs they face. of over 4,600 technical professionals from around the world, nds Without such knowledge, developers make easily avoidable errors, CI to be an indicator of “high performing IT organizations”. We tool builders invest in the wrong direction, and researchers miss previously reported [19] that 40% of the 34,000 most popular open- opportunities for improving the practice of CI. source projects on GitHub use CI, and the most popular projects We present a qualitative study of the barriers and needs devel- are more likely to use CI (70% of the top 500 projects). opers face when using CI. We conduct semi-structured interviews Despite the widespread adoption of CI, there are still many unan- with developers from dierent industries and development scales. swered questions about CI. In one study, Vasilescu et al. [50] show We triangulate our ndings by running two surveys. We nd that that CI correlates with positive quality outcomes. In our previous developers face trade-os between speed and certainty (Assurance), work [19], we examine the usage of CI among open-source projects between better access and information security (Security), and be- on GitHub, and show that projects that use CI release more fre- tween more conguration options and greater ease of use (Flexi- quently than projects that do not. However, these studies do not bility). We present implications of these trade-os for developers, present what barriers and needs developers face when using CI, or tool builders, and researchers. what trade-os developers must make when using CI. To ll in the gaps in knowledge about developers’ use of CI, CCS CONCEPTS we ask the following questions: What needs do developers have • Software and its engineering → Agile software develop- that are unmet by their current CI system(s)? What problems have ment; Software testing and debugging; developers experienced when conguring and using CI system(s)? How do developers feel about using CI? Without answers to these KEYWORDS questions, developers can potentially nd CI more obstructive than helpful, tool builders can implement unneeded features, and re- Continuous Integration, Automated Testing searchers may not be aware of areas of CI usage that require further ACM Reference Format: examination and solutions that can further empower practitioners. Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny To answer these questions, we employ complementary estab- Dig. 2017. Trade-Os in Continuous Integration: Assurance, Security, and lished research methodologies. Our primary methodology is inter- Flexibility. In Proceedings of 2017 11th Joint Meeting of the European Software views with 16 software developers from 14 dierent companies of Engineering Conference and the ACM SIGSOFT Symposium on the Foun- all sizes. To triangulate [15] our ndings, we deploy two surveys. dations of Software Engineering, Paderborn, Germany, September 4–8, 2017 The Focused Survey samples 51 developers at Pivotal1. The Broad (ESEC/FSE’17), 11 pages. Survey samples 523 participants, of which 95% are from industry, https://doi:org/10:1145/3106237:3106270 and 70% have seven or more years of software development experi- ence. The interviews provide the content for the surveys, and the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Focused Survey provides depth, while the Broad Survey provides for prot or commercial advantage and that copies bear this notice and the full citation breadth. Analyzing all this data, we answer four research questions: on the rst page. Copyrights for components of this work owned by others than ACM RQ1: What barriers do developers face when using CI? (see §4.1) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a RQ2: What unmet needs do developers have with CI tools? (see §4.2) fee. Request permissions from [email protected]. RQ3: Why do developers use CI? (see §4.3) ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany RQ4: What benets do developers experience using CI? (see §4.4) © 2017 Association for Computing Machinery. ACM ISBN 978-1-4503-5105-8/17/09...$15.00 https://doi:org/10:1145/3106237:3106270 1pivotal.io 197 ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany M. Hilton, N. Nelson, T. Tunnell, D. Marinov, and D. Dig Based on our ndings, we identify three trade-os developers We asked our interview participants to describe their CI usage face when using CI. Other researchers [32, 34, 53] have identied pipeline. While not all pipelines are the same, they generally share similar trade-os in dierent domains. We name these trade-os some common elements. Assurance, Security, and Flexibility. Changesets are a group of changes that a developer makes to the Assurance describes the trade-o between increasing the added code. They may be a single commit, or a group of commits, but they value that extra testing provides, and the extra cost of perform- should be a complete change, so that after the changeset is applied, ing that testing. Rothermel et al. [34] identify this trade-o as a it should not break the program. motivation for test prioritization. When a CI system observes a change made by developers, this Security describes the trade-o between increased security mea- triggers a CI event. How and when the CI is triggered is based on sures, and the ability to access and modify the CI system as needed. how the CI is congured. One common way to trigger CI is when a Post and Kagan [32] found a third of knowledge workers report commit is pushed to a repository. security restrictions hinder their ability to perform their jobs. We For the CI to test the code without concern for previous data or observe this issue also applies to CI users. external systems, it is important that CI runs in a clean environment. Flexibility describes the trade-o that occurs when developers The automated build script should be able to start with a clean want systems that are both powerful and highly congurable, yet environment and build the product from scratch before executing at the same time, they want those systems to be simple and easy to tests. Many developers use containers (e.g., Docker2) to implement use. Xu et al. [53] identify the costs of over-congurable systems clean environments for builds. and found that these systems severely hinder usability. We also An important step in the CI pipeline is conrming that the observe the tension from this trade-o among developers using CI. changeset was integrated correctly into the application. One com- In the context of these three trade-os, we present implications mon method is a regression test suite, including unit tests and for three audiences: developers, tool builders, and researchers. For integration tests. The CI system can also perform other analyses, example, developers face dicult choices about how much testing such as linting or evaluating test coverage. is enough, and how to choose the right tests to run. Tool builders The last step is to deploy the artifact. We found some develop- should create UIs for CI users to congure their CI systems, but ers consider deployment to be a part of CI, and others consider these UIs should serialize congurations out to text les so that continuous deployment (CD) to be a separate process. they can be kept in version control. Researchers have much to bring to the CI community, such as helping with fault localization and 3 METHODOLOGY test parallelization when using CI, and examining the security Inspired by established guidelines [24, 28, 31, 41, 48], the primary challenges developers face when using CI. methodologies we employ in this work are interviews with soft- This paper makes the following contributions: ware developers and two surveys of software developers to trian- (1) We conduct exploratory semi-structured interviews with 16 gulate [15] our ndings. developers, then triangulate these ndings with a Focused Survey Interviews are a qualitative method and are eective at discover- of 51 developers at Pivotal and a Broad Survey of 523 developers ing the knowledge and experiences of the participants. However, from all over the world. they often have a limited sample size [41]. Surveys are a quantita- (2) We provide an empirically justied set of developers’ motiva- tive technique that summarizes information over a larger sample tions for using CI. size and thus provides broader results. Together, they provide a (3) We expose gaps between developers’ needs and existing tooling much clearer picture than either can provide alone. for CI. We rst use interviews to elicit developers experiences and ex- (4) We present actionable implications that developers, tool builders, pectations when working with CI, and we build a taxonomy of and researchers can build on.