Continuous Experimentation for Software Developers

Total Page:16

File Type:pdf, Size:1020Kb

Continuous Experimentation for Software Developers Zurich Open Repository and Archive University of Zurich Main Library Strickhofstrasse 39 CH-8057 Zurich www.zora.uzh.ch Year: 2019 Continuous experimentation for software developers Schermann, Gerald Posted at the Zurich Open Repository and Archive, University of Zurich ZORA URL: https://doi.org/10.5167/uzh-172623 Dissertation Published Version Originally published at: Schermann, Gerald. Continuous experimentation for software developers. 2019, University of Zurich, Faculty of Economics. Department of Informatics Continuous Experimentation for Software Developers Dissertation submitted to the Faculty of Business, Economics and Informatics of the University of Zurich to obtain the degree of Doktor der Wissenschaften, Dr. sc. (corresponds to Doctor of Science, PhD) presented by Gerald Schermann from Austria approved in July 2019 at the request of Prof. Dr. Harald C. Gall Dr. Philipp Leitner Prof. Dr. Elisabetta Di Nitto The Faculty of Business, Economics and Informatics of the University of Zurich hereby authorizes the printing of this dissertation, without indicating an opinion of the views expressed in the work. Zurich, July 17, 2019 Chairman of the Doctoral Board: Prof. Dr. Thomas Fritz Acknowledgments First and foremost, I thank my advisors Philipp Leitner and Harald Gall. Not only their continuous guidance and support but also their expertise was invaluable during the course of my doctoral studies. Thank you for giving me the opportunity to pursue a PhD paired with the freedom to work on topics that interest me the most. I thank Elisabetta Di Nitto for serving on my PhD committee as an external examiner, dedicating her valuable time for evaluating my work, and the great feedback that I received. Special thanks go to Erik Wittern and Fábio Oliveira who gave me the opportunity for doing an internship at IBM Research in New York. Thanks for the great discussions and valuable feedback. I especially thank my current and former colleagues from the Software Evo- lution and Architecture Lab (SEAL), without them this adventure would have been only half the fun: Carol Alexandru, Martin Brandtner, Adelina Ciurumelea, Thomas Fritz, Christian Inzinger, André Meyer, Sebastiano Panichella, Sebas- tian Proksch, Manuela Züger, and in particular Jürgen Cito, Giovanni Grano, Katja Kevic, Christoph Laaber, Sebastian Müller, and Carmine Vassallo. Being it endless discussions during coffee breaks or lunches, playing Uno and Explod- ing Kittens, or all the other fun we had in our offices, at retreats, or on conference trips, all these little fragments made the time of my PhD memorable. Finally, I want to thank my family for their unconditional support and encouragement during my studies. Gerald Schermann Zürich, July 2019 Abstract For staying competitive in highly-contested and fast-growing markets such as the Web companies need to continuously adapt their software. Releasing incremental changes fast, while at the same time guaranteeing high quality, requires release processes that are strongly based on tools to automate software build, test, and deployment. While previous methods of releasing changes hardly involved evi- dence to support decisions (e.g., do users appreciate my new feature?), nowadays, sophisticated telemetry solutions keep track of releases and captured live produc- tion data has become the basis for data-driven decision making. High automation bundled with telemetry promotes the advent of continuous experimentation prac- tices (e.g., canary releases, or A/B testing) that guide development activities based on data collected on a fraction of the user population on a new experimen- tal version of the software in the production environment. However, adopting continuous experimentation to move towards data-driven decision making is not a straightforward process, it involves setting up a complex experimentation infrastructure and requires methods and tools to cover the entire life cycle of experiments, from their design to the assessment of their outcome. In the context of this thesis, we address challenges surfacing within experiment life cycle phases with the goal to devise research approaches to support the thesis statement: “A detailed understanding of the characteristics of continuous experiments enables building a conceptual framework for planning, executing, and analyzing experiments”. To pay attention to the trend towards decentralized microservice teams independently running experiments, our approaches are tailored to software developers and release engineers and designed to foster the parallel execution of experiments with as little overhead as possible, to identify iv optimal plans to collect required sample sizes for sound statistical interpretation, and to provide means for experiment health assessment. Informed by the findings from an empirical study on the state of practice, we characterized experimentation practices into regression-driven experiments (e.g., canary releases) and business-driven experiments (e.g., A/B testing), and derived a conceptual framework for experimentation. This framework built the basis for three research approaches and prototypes as concrete instantiations that have been extensively validated through numerical experimentation: Fenrir targets the planning phase of experiments and scheduling in particular. We formulate scheduling as an optimization problem with the aim of fostering the parallel execution of experiments, while at the same time ensuring that enough data is collected for every experiment and the collected data is not skewed by overlapping experiments. Fenrir outperforms other approaches not only in the quality of the schedules identified but also in terms of execution time. Bifrost supports the execution phase and involves the automated, data- driven execution of multi-phased experiments (e.g., an A/B test follows a canary release). Experiments are specified in a domain-specific language and our concept of conditional chaining allows triggering automated actions such as rollbacks in case of spotted irregularities. Bifrost supports running more than a hundred experiments in parallel without introducing a significant performance degradation. Finally, we investigated approaches and devised a research prototype for experiment health assessment with the goal of raising the developer’s awareness about (topological) changes in the context of experiments. We characterized change types that surface within the evolution of microservice-based applications and developed and evaluated multiple heuristics to rank identified changes according to their potential impact on the application’s health state. Overall, we demonstrated that our framework enables planning, executing, and analyzing large-scale continuous experiments. There are multiple opportunities for future work to extend our framework and approaches including smarter experimentation platforms that dynamically decide how experimentation logic is executed, visualization extensions to IDEs (integrated developer environments), and providing means for experiment verification based on statistical models. Zusammenfassung Um Wettbewerbsfähigkeit in hart umkämpften und schnell wachsenden Märkten wie dem Web zu gewährleisten, ist es für Unternehmen essentiell, fortlaufend ihre Software anzupassen. Schnelle und kontinuierliche Releases von inkrementellen Änderungen bei gleichzeitiger Gewährleistung einer hohen Qualität, erfordern Releaseprozesse, die stark auf Werkzeuge zur Automatisierung von Software Builds, Tests und Deployments basieren. Während bisherige Releaseverfahren kaum Echtzeitinformationen zur Entscheidungsfindung herangezogen haben (bei- spielsweise, schätzen Benutzer mein neues Feature?), überwachen heutzutage ausgefeilte Telemetrielösungen Software Releases und die dabei gesammelten Echtzeitinformationen bilden die Basis für datengetriebene Entscheidungsfindung. Telemetrie gebündelt mit einem hohen Grad an Automatisierung fördert das Aufkommen von Continuous Experimentation Praktiken (z.B. Canary Relea- ses oder A/B-Tests), welche Entwicklungstätigkeiten auf der Grundlage von Informationen lenken, die für einen Bruchteil der Benutzerpopulation an einer neuen experimentellen Version der Software direkt in der Produktivumgebung gesammelt werden. Die Einführung von Continuous Experimentation mit dem Ziel der datengetriebenen Entscheidungsfindung ist jedoch kein simpler Prozess. Die Einführung erfordert nicht nur den Aufbau einer komplexen Infrastruktur, sondern auch Methoden und Werkzeuge, die den gesamten Lebenszyklus von Continuous Experiments abdecken, von der Planung bis zur Ergebnisevaluation. Im Rahmen dieser Dissertation beschäftigen wir uns mit Herausforderun- gen, welche in den verschiedenen Phasen der Lebenszyklen von Experimenten auftreten, mit dem Ziel, Forschungsansätze zu entwickeln, die die These dieser vi Dissertation unterstützen: “Ein umfassendes Verständnis der Charakteristiken von Continuous Experiments ermöglicht die Umsetzung eines konzeptuellen Fra- meworks für das Planen, Durchführen und Analysieren von Experimenten”. Um dem Trend zu dezentralisierten Microservice-Teams Rechnung zu tragen, die unabhängig voneinander Experimente durchführen, sind unsere Ansätze auf Soft- wareentwickler und Release Engineers zugeschnitten. Darüber hinaus sollen diese Forschungsansätze die parallele Durchführung von Experimenten mit möglichst geringen Leistungseinbussen unterstützen, Wege identifizieren um erforderliche Stichprobengrössen für robuste statistische Auswertung zu gewährleisten und Möglichkeiten für eine aussagekräftige
Recommended publications
  • A Multi-Method Empirical Study on Continuous Experimentation
    We're Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation Gerald Schermanna,∗, J¨urgenCitoa, Philipp Leitnera, Uwe Zdunb, Harald C. Galla aDepartment of Informatics, University of Zurich, Switzerland fschermann, cito, leitner, gallg@ifi.uzh.ch bUniversity of Vienna, Austria [email protected] Abstract Context: Continuous experimentation guides development activities based on data collected on a subset of online users on a new experimental version of the software. It includes practices such as canary releases, gradual rollouts, dark launches, or A/B testing. Objective: Unfortunately, our knowledge of continuous experimentation is currently primarily based on well-known and outspoken industrial leaders. To assess the actual state of practice in continuous experimentation, we conducted a mixed-method empirical study. Method: In our empirical study consisting of four steps, we interviewed 31 developers or release engineers, and performed a survey that attracted 187 complete responses. We analyzed the resulting data using statistical analysis and open cod- ing. Results: Our results lead to several conclusions: (1) from a software architecture perspective, continuous experimen- tation is especially enabled by architectures that foster independently deployable services, such as microservices-based architectures; (2) from a developer perspective, experiments require extensive monitoring and analytics to discover run- time problems, consequently leading to developer on call policies and influencing the role and skill sets required by developers; and (3) from a process perspective, many organizations conduct experiments based on intuition rather than clear guidelines and robust statistics. Conclusion: Our findings show that more principled and structured approaches for release decision making are needed, striving for highly automated, systematic, and data- and hypothesis-driven deployment and experimentation.
    [Show full text]
  • Branching Strategies for Developing New Features Within the Context of Continuous Delivery
    2nd Workshop on Continuous Software Engineering Branching strategies for developing new features within the context of Continuous Delivery Konrad Schneid1 Abstract: This paper evaluates based on current literature, whether the versioning strategies “branch by feature” and “develop on mainline” can be used for developing new software features in connection with Continuous Delivery. The strategies will be introduced and possible applications for Continuous Delivery will be demonstrated and rated. A solution recommendation is finally given. It becomes evident that develop on mainline is the more recommendable method in form of “features toggles” or in case of bigger changes in form of “branch by abstraction” within the context of Continuous Delivery. Keywords: Continuous Delivery, Branch by Feature, Develop on Mainline, Feature Toggles, Branch by Abstraction 1 Problem formulation Different attempts exist around developing new software features. In practice, several developers are involved in programming software. They work in parallel on developing the software. In order to perform the parallel development of different software versions, version control systems (VCS) like Git are used [CS14, p.27]. VCS offer various versioning strategies for developing new features. The development of different features should be separated to provide these independently to the production system. It must be guaranteed that features which are still in development do not disturb the productive operation, but can still be developed. Two appropriate strategies are branch by feature and develop on mainline [HF10, Chap.14]. This paper researches whether and how these strategies can be used within the context of Continuous Delivery (CD). One important aspect of CD is Continuous Integration (CI) [HF10, p.24].
    [Show full text]
  • Open Source Version Control Software
    FACULTY OF INFORMATION TECHNOLOGY AND ELECTRICAL ENGINEERING Vili Seppänen ENHANCING CONTINUOUS INTEGRATION PROCESSES IN AGILE DEVELOPMENT USING PROJECT MANAGEMENT TOOLS Master’s Thesis Degree Programme in Computer Science and Engineering 08 2018 Seppänen V. (2018) Enhancing continuous integration processes in agile development using project management tools. University of Oulu, Department of Computer Science and Engineering. Master’s Thesis, 58 p. ABSTRACT This thesis presents a concept to enhance the continuous integration processes in agile development, by utilising the combination of project management tools and user stories. First, the reader is introduced to the fields of continuous integration, agile, feature toggles and version control to provide a good basic understanding of the context. Related work on release engineering is presented from the perspective of release planning and constructing. Problems with current, commonly used continuous integration processes are identified and analysed, and then solutions for these problems are designed and proposed. This includes listing the requirements for implementing the solutions, describing the designs of the solutions and discussing the expected benefits of the solutions. These solutions form the concept to enhance the continuous integration processes in agile development. This new concept is evaluated in a user-study among seasoned IT professionals. The study includes applicable elements from Expectation Disconfirmation Theory to examine the validity and the likely future adoption of the proposed approach in the information technology sector. The evaluation results suggest that the solution, when implemented in a production environment, will be easy to use; have the capability, functions and features needed to accomplish its tasks; will be reliable and beneficial for its purpose and would be likely to improve the overall performance of the software project.
    [Show full text]
  • Development and Evolution of Agile Changes in a World of Change
    Development and Evolution of Agile Changes in a World of Change Thomas J Marlowe Vassilka Kirova Garett Chang Department of Mathematics & Bell Labs Consulting Highstep Technologies Computer Science vassilka.kirova@bell-labs- [email protected] Seton Hall University consulting.com [email protected] Omer Hashmi Stephen P. Masticola Vice-President, Agile Brains Consulting SPM Consulting [email protected] [email protected] Abstract1,2 often been misinterpreted as lack of discipline or need of documentation, and has led to many pitfalls and disappointments Agile software development is an approach first codified in the in the implementation of agile [82]. Agile Manifesto in 2001. This was a statement of core values that became associated with a set of principles and practices. A primary motivation for preferring agile processes to waterfall, Key ideas include early and constant customer involvement, spiral, and other software engineering approaches [91] is that self-organizing teams that embrace change, rapid delivery of developing software cannot be treated as a theoretically well- value, short timeboxed iterations coordinated by a shared list of defined process in practice. Schwaber [100] differentiates items—a product backlog and driven by user stories and use between “theoretical” processes (i.e., those which can be cases, clean code, test-driven development, and continuous specified in a white-box manner from well-understood a priori integration. The values, principles, and practices have permeated parameters) and “empirical” processes (i.e., those which are not the technical and business world, translated and modified to fit predictable from theory.) Empirical processes cannot be many domains, affecting both production and management.
    [Show full text]
  • ALM Rangers Home Page – Branching Strategies – Foreword
    Visual Studio ALM Rangers Home Page – http://aka.ms/vsarmsdn Branching Strategies – Foreword The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, you should not interpret this to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Microsoft grants you a license to this document under the terms of the Creative Commons Attribution 3.0 License. All other rights are reserved. 2014 Microsoft Corporation. Microsoft, Active Directory, Excel, Internet Explorer, SQL Server, Visual Studio, and Windows are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners. Page 2 of 75 Branching Strategies – Foreword Table of Contents Foreword ............................................................................................................................................................................................................................................ 4 Introduction .....................................................................................................................................................................................................................................
    [Show full text]
  • Investigating Modern Release Engineering Practices
    Investigating Modern Release Engineering Practices Md Tajmilur Rahman A Thesis in The Department of Computer Science & Software Engineering Presented in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy (Software Engineering) at Concordia University Montreal,´ Quebec,´ Canada July 2017 c Md Tajmilur Rahman, 2017 CONCORDIA UNIVERSITY School of Graduate Studies This is to certify that the thesis prepared By: Md Tajmilur Rahman Entitled: Investigating Modern Release Engineering Practices and submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Software Engineering) complies with the regulations of this University and meets the accepted standards with respect to originality and quality. Signed by the final examining committee: Chair Dr. Waiz Ahmed External Examiner Dr. Shane McIntosh External to Program Dr. Abdelwahab Hamou-Lhadj Examiner Dr. Nikolas Tsantalis Examiner Dr. Juergen Rilling Thesis Supervisor Dr. Peter Rigby Approved by Dr. Sudhir Mudur, Chair of Department August 28th, 2017 Date of Defence Dr. Amir Asif, Dean, Faculty of Engineering and Computer Science Abstract Investigating Modern Release Engineering Practices Md Tajmilur Rahman, Ph.D. Concordia University, 2017 Modern release engineering has moved from longer release cycles and separate development and release teams to a continuous and integrated process. However, release engineering practices include not only integration, build and test execution but also a better management of features. The goal of this research is to investigate the modern release engineering practices which cover four milestones in the field of release engineering, i. understanding rapid release by measuring the time and effort involved in release cycles, ii. feature management based on feature toggles iii.
    [Show full text]
  • Feature Toggles: Practitioner Practices and a Case Study
    Feature Toggles: Practitioner Practices and a Case Study Md Tajmilur Rahmany, Louis-Philippe Querely, Peter C. Rigbyy, Bram Adamsx y Concordia University, x Polytechnique Montreal Montreal, QC, Canada {mdt_rahm, l_querel}@encs.concordia.ca [email protected], [email protected] ABSTRACT continuous delivery requires streamlining all release engineer- Continuous delivery and rapid releases have led to innovative ing activities [2]. techniques for integrating new features and bug fixes into One of the most unpredictable release engineering activi- a new release faster. To reduce the probability of integra- ties is the integration process [6]. During integration, new tion conflicts, major software companies, including Google, features and bug fixes are combined with the latest code Facebook and Netflix, use feature toggles to incrementally from other teams (e.g., by merging git branches [8]) to create integrate and test new features instead of integrating the a coherent new release. Integration is unpredictable because feature only when it's ready. Even after release, feature the combination of many changes late in the develop cycle toggles allow operations managers to quickly disable a new can lead to instability in the software [30]. Stabilization of feature that is behaving erratically or to enable certain fea- such changes can take substantial time, leading to even larger tures only for certain groups of customers. Since literature gaps between the latest development and the release stabi- on feature toggles is surprisingly slim, this paper tries to lization branch. These incompatibilities (\merge conflicts") understand the prevalence and impact of feature toggles. only surface at integration or testing time, which is close to First, we conducted a quantitative analysis of feature toggle release time, and hence introduces delays.
    [Show full text]
  • Synthesizing Continuous Deployment Practices Used in Software Development
    Synthesizing Continuous Deployment Practices Used in Software Development Akond Ashfaque Ur Rahman, Eric Helms, Laurie Williams, and Chris Parnin Department of Computer Science, North Carolina State University, Raleigh, North Carolina, USA {aarahman, edhelms, lawilli3, cjparnin}@ncsu.edu Abstract – Continuous deployment speeds up the process of software practitioners have identified the ‘lack of existing agile methods, such as Scrum, and Extreme understanding’ of continuous deployment as an adoption Programming (XP) through the automatic deployment of challenge [9]. software changes to end-users upon passing of automated tests. One way to understand continuous deployment is to learn Continuous deployment has become an emerging software about the practices and techniques used by software engineering process amongst numerous software companies, such as Facebook, Github, Netflix, and Rally Software. A systematic companies to implement continuous deployment. Some of the analysis of software practices used in continuous deployment can software companies have used Internet artifacts, such as blog facilitate a better understanding of continuous deployment as a posts, slide and audio presentations, to share their experiences software engineering process. Such analysis can also help and techniques of using continuous deployment. However, software practitioners in having a shared vocabulary of practices searching the Internet to collect and contrast all relevant and in choosing the software practices that they can use to artifacts might require significant time and effort [28]. implement continuous deployment. The goal of this paper is to Software practitioners can benefit from a study that aid software practitioners in implementing continuous systematically analyzes the software practices used by deployment through a systematic analysis of software practices software companies to implement continuous deployment.
    [Show full text]