Continuous Integration, Delivery and Deployment: a Systematic Review on Approaches, Tools, Challenges and Practices”, IEEE Access, 2017
Total Page：16
File Type：pdf, Size：1020Kb
M. Shahin, M. Ali Babar, and L. Zhu, ―Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices”, IEEE Access, 2017. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices Mojtaba Shahina, Muhammad Ali Babara, Liming Zhub a CREST – The Centre for Research on Engineering Software Technologies, The University of Adelaide, Australia b Data61, Commonwealth Scientific and Industrial Research Organisation, Sydney, NSW 2015, Australia [email protected], [email protected], [email protected] Abstract—Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software I. INTRODUCTION development industry practices that enable organizations to With increasing competition in software market, frequently and reliably release new features and products. organizations pay significant attention and allocate resources With the increasing interest in and literature on continuous to develop and deliver high-quality software at much practices, it is important to systematically review and accelerated pace [1]. Continuous Integration (CI), Continuous synthesize the approaches, tools, challenges, and practices DElivery (CDE), and Continuous Deployment (CD), called reported for adopting and implementing continuous practices. continuous practices for this study, are some of the practices aimed at helping organisations to accelerate their Objective: This research aimed at systematically reviewing development and delivery of software features without the state of the art of continuous practices to classify compromising quality [2]. Whilst CI advocates integrating approaches and tools, identify challenges and practices in this work-in-progress multiple times per day, CDE and CD are regard, and identify the gaps for future research. about ability to quickly and reliably release values to Method: We used systematic literature review (SLR) method customers by bringing automation support as much as for reviewing the peer-reviewed papers on continuous possible [3, 4]. practices published between 2004 and 1st June 2016. We Continuous practices are expected to provide several applied thematic analysis method for analysing the data benefits such as: (1) getting more and quick feedback from extracted from reviewing 69 papers selected using predefined the software development process and customers; (2) having criteria. frequent and reliable releases, which lead to improved customer satisfaction and product quality; (3) through CD, Results: We have identified thirty approaches and associated the connection between development and operations teams is tools, which facilitate the implementation of continuous strengthened and manual tasks can be eliminated [5, 6]. A practices in the following ways: (1) ―reducing build and test growing number of industrial cases indicate that the time in continuous integration (CI)‖; (2) ―increasing visibility continuous practices are making inroad in software and awareness on build and test results in CI‖; (3) development industrial practices across various domains and ―supporting (semi-) automated continuous testing‖; (4) sizes of organizations [5, 7, 8]. At the same time, adopting ―detecting violations, flaws and faults in CI‖; (5) ―addressing continuous practices is not a trivial task since organizational security and scalability issues in deployment pipeline‖, and processes, practices, and tool may not be ready to support the (6) ―improving dependability and reliability of deployment highly complex and challenging nature of these practices. process‖. We have also determined a list of critical factors Due to the growing importance of continuous practices, an such as ―testing (effort and time)‖, ―team awareness and increasing amount of literature describing approaches, tools, transparency‖, ―good design principles‖, ―customer‖, ―highly practices, and challenges has been published through diverse skilled and motivated team‖, ―application domain‖, and venues. An evidence for this trend is the existence of five ―appropriate infrastructure‖ that should be carefully secondary studies on CI, rapid release, CDE and CD [9-13]. considered when introducing continuous practices in a given These practices are highly correlated and intertwined, in organization. The majority of the reviewed papers were which distinguishing these practices are sometimes hard and validation (34.7%) and evaluation (36.2%) research types. their meanings highly depends on how a given organization This review also reveals that continuous practices have been interprets and employs them [14]. Whilst CI is considered the successfully applied to both greenfield and maintenance first step towards adopting CDE practice [15], truly projects. implementing CDE practice is necessary to support Conclusion: Continuous practices have become an important automatically and continuously deploying software to area of software engineering research and practice. Whilst the production or customer environments (i.e., CD practice). We reported approaches, tools, and practices are addressing a noticed that there was no dedicated effort to systematically wide range of challenges, there are several challenges and analyze and rigorously synthesize the literature on continuous gaps which require future research work for: improving the practices in an integrated manner. By integrated manner we capturing and reporting of contextual information in the mean simultaneously investigating approaches, tools, studies reporting different aspects of continuous practices; challenges, and practices of CI, CDE, and CD, which aims to gaining a deep understanding of how software-intensive explore and understand the relationship between them and systems should be (re-) architected to support continuous what steps should be followed to successfully and smoothly practices; addressing the lack of knowledge and tools for move from one practice to another. This study aimed at filling engineering processes of designing and running secure that gap by conducting a Systematic Literature Review (SLR) deployment pipelines. of the approaches, tools, challenges and practices for adopting and implementing continuous practices. Index Terms— continuous integration, continuous delivery, continuous deployment, continuous software engineering, systematic literature review, empirical software engineering 1 FIGURE 1. The relationship between continuous integration, delivery and deployment [16, 17] This SLR provides an in-depth understanding of the Continuous Integration (CI) is a widely established challenges of adopting continuous practices and the strategies development practice in software development industry [4], (e.g., tools) used to address the challenges. Such an in which members of a team integrate and merge understanding is expected to help identify the areas where development work (e.g., code) frequently, for example methodological and tool support to be improved. This multiple times per day. CI enables software companies to increases the efficacy of continuous practices for different have shorter and frequent release cycle, improve software types of organizations and software-intensive applications. quality, and increase their teams‘ productivity [4]. This Moreover, the findings are expected to be used as guidelines practice includes automated software building and testing [5]. for practitioners to become more aware of the approaches, Continuous DElivery (CDE) is aimed at ensuring an tools, challenges and implement appropriate practices that application is always at production-ready state after suit their industrial arrangements. For this review, we have successfully passing automated tests and quality checks [19, systematically identified and rigorously reviewed 69 relevant 20]. CDE employs a set of practices e.g., CI, and deployment papers and synthesized the data extracted from those papers automation to deliver software automatically to a production- in order to answer a set of research questions that motivated like environment [15]. According to [6, 21], this practice this review. The significant contributions of this work are: offers several benefits such as reduced deployment risk, 1. A classification of the reported approaches, associated lower costs and getting user feedback faster. Figure 1 tools, and challenges and practices of continuous practices indicates that having continuous delivery practice requires in an easily accessible format. continuous integration practice. 2. A list of critical factors that should be carefully considered Continuous Deployment (CD) practice goes a step further when implementing continuous practices in both software and automatically and continuously deploys the application to development and customer organizations. production or customer environments [19, 22]. There is 3. An evidence-based guide to select appropriate approaches, robust debate in academic and industrial circles about tools and practices based on the required suitability for defining and distinguishing between continuous deployment different contexts. and continuous delivery [4, 19, 20]. What differentiates 4. A list of researchable issues to direct the future research continuous deployment from continuous delivery is a efforts for advancing the state-of-the-art of continuous production environment (i.e., actual customers): the goal of practices. continuous deployment practice is to automatically and The rest of the paper is organized