Feature Toggles: Practitioner Practices and a Case Study Md Tajmilur Rahmany, Louis-Philippe Querely, Peter C. Rigbyy, Bram Adamsx y Concordia University, x Polytechnique Montreal Montreal, QC, Canada {mdt_rahm, l_querel}@encs.concordia.ca [email protected], [email protected] ABSTRACT continuous delivery requires streamlining all release engineer- Continuous delivery and rapid releases have led to innovative ing activities [2]. techniques for integrating new features and bug fixes into One of the most unpredictable release engineering activi- a new release faster. To reduce the probability of integra- ties is the integration process [6]. During integration, new tion conflicts, major software companies, including Google, features and bug fixes are combined with the latest code Facebook and Netflix, use feature toggles to incrementally from other teams (e.g., by merging git branches [8]) to create integrate and test new features instead of integrating the a coherent new release. Integration is unpredictable because feature only when it's ready. Even after release, feature the combination of many changes late in the develop cycle toggles allow operations managers to quickly disable a new can lead to instability in the software [30]. Stabilization of feature that is behaving erratically or to enable certain fea- such changes can take substantial time, leading to even larger tures only for certain groups of customers. Since literature gaps between the latest development and the release stabi- on feature toggles is surprisingly slim, this paper tries to lization branch. These incompatibilities (\merge conflicts") understand the prevalence and impact of feature toggles. only surface at integration or testing time, which is close to First, we conducted a quantitative analysis of feature toggle release time, and hence introduces delays. Such delays are usage across 39 releases of Google Chrome (spanning five incompatible with continuous delivery. years of release history). Then, we studied the technical debt Although substantial research has been done on predict- involved with feature toggles by mining a spreadsheet used ing painful merges and quantifying the effort involved in a by Google developers for feature toggle maintenance. Finally, merge [9, 12, 14, 41], release engineers of Facebook, Google we performed thematic analysis of videos and blog posts of and Netflix in their talks at the 2nd International Workshop release engineers at major software companies in order to on Release Engineering (RELENG) [1] instead declared that further understand the strengths and drawbacks of feature they simply try to merge more rapidly, effectively integrating toggles in practice. We also validated our findings with four partial (work-in-progress) versions of a feature into the code Google developers. We find that toggles can reconcile rapid base instead of waiting for the complete feature. When we releases with long-term feature development and allow flex- asked these workshop participants, \if you are integrating in- ible control over which features to deploy. However they complete features, how do you prevent them from interfering with other features?", they unequivocally replied \by using also introduce technical debt and additional maintenance for 1 developers. feature toggles" . A feature toggle is an age-old and conceptually simple concept [4, 33]. It basically is a variable used in a con- 1. INTRODUCTION ditional statement to guard code blocks, with the aim of In recent years, one of the top priorities of many companies either enabling or disabling the feature code in those blocks and organizations has been to re-engineer their release pro- for testing or release. For example, in Figure 1c, the if cess in order to achieve continuous delivery of new features statement checks the value of the Google Chrome toggle or at least more timely releases [4, 27]. While traditional kDisableFlashFullscreen3d. If the toggle is present in the releases would take months, modern software companies Chrome configuration, the return statement ensures that the have managed to reduce their apps' \cycle time" to 6 weeks remainder of the method is not executed, effectively disabling (Google Chrome [39] and Mozilla Firefox [40]), 2 weeks (Face- the 3D flash feature. In contrast to traditional compile-time book Mobile app [36]) or even daily (Facebook web app [36], feature toggles [7, 15, 16] that exclude features from an ap- Netflix [37] and IMVU [20]). Successful migration towards plication's binary altogether, modern feature toggles allow features to be switched on or off without recompilation, i.e., at run-time or at least at startup-time. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Despite the conceptual simplicity of feature toggles, they for profit or commercial advantage and that copies bear this notice and the full citation also contain risk. Each toggle basically \comments out" large on the first page. Copyrights for components of this work owned by others than ACM blocks of code (features) that are not yet ready for testing, must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a release or should be used only by a small number of users. fee. Request permissions from [email protected]. This leads to partially dead code and hence technical debt, MSR’16, May 14-15, 2016, Austin, TX, USA unless features are made permanent (by removing their if c 2016 ACM. ISBN 978-1-4503-4186-8/16/05. $15.00 1 DOI: http://dx.doi.org/10.1145/2901739.2901745 Sometimes also called \flippers", \switches" or \gates". statements) as soon as they are stable. Furthermore, as the number of feature toggles grows, there is a combinatorial explosion of possible feature sets that need to be tested, (a) as an organization could pick any combination of features for their next release. Large companies such as Facebook, Yahoo, Google and Adobe [18] are cognizant of some of these problems, but similar to small and medium sized software companies, it is unclear to them what is the long-term impact of feature toggles and what best practices exist. (b) Although toggles are used extensively in industry, we are unaware of any empirical study that examines how feature toggles actually are being used in the software industry. (c) Hence, we conducted a mixed-methods study to provide a better understanding of feature toggles and their benefits Figure 1: Examples of Chrome 23, showing (a) the names and risks. of known switch files, (b) example toggles inside con- In the first stage, we measure the following basic parame- tent_switches.cc, and (c) code that is covered by the toggle ters in an exploratory study of toggle use on Chrome: kDisableFlashFullscreen3d. 1. Adoption: How many toggles exist? from the move towards continuous delivery of software sys- 2. Usage: How are toggles used? tems [27]. With increasingly shorter release cycles, features 3. Development Stage: Are toggles modified during devel- are developed behind release toggles such that they can easily opment or release stabilization? be disabled if the feature is not yet ready for release. This flexibility allows release engineers to disable a feature that is 4. Lifetime: How long do toggles remain in the system? blocking a release simply by disabling the feature toggle. In addition, Bass et al. [4] promote the use of release toggles to In the second stage, we examine toggle maintenance and decouple the roll-out of a new feature to data centers from debt by studying the impact of a toggle maintenance cam- the actual release of the feature to (subsets of) users. In this paign launched by Chrome developers. In the third stage, we paper, we focus on both kinds of feature toggles. complement and generalize our Chrome results by examining From a practical perspective, Fowler states that feature the talks and writing of prominent release engineers of large toggles require the following [21]: successful companies including Twitter, Facebook, Google, etc. 1. A configuration file with all feature toggles and their The paper is structured around these research stages. In state (value). section 2, we define feature toggles and give examples of 2. A mechanism to switch the state of the toggles (for actual toggles on the Google Chrome projects. In section 3, example, at program startup or while the application we describe our mixed methods research approach and the is running), effectively enabling or disabling a feature. three data-sources that we use in our study. In section 4, we conduct an exploratory case study to measure the adoption, 3. Conditional blocks that guard all entry points to a usage, development stage, and lifetime of toggles. We also feature such that changing a toggle's value can enable discuss the lifecycle of toggles on Chrome. In section 5, we or disable the feature's code. describe toggle debt and the toggle maintenance campaign and the types of toggles used on Chrome. In section 6, we To illustrate Fowler's requirements, we use Google Chrome's use the talks and blog posts of developers on other large toggle implementation. Chrome is a popular, large open successful projects to generalize our understanding of toggle source project that has been using feature toggles for more use. In section 7, we discuss threats to validity. In section 8, than five years and that we use as case study in this pa- we conclude the paper, tying together our findings from our per. Instead of having a single configuration file for toggles, various data sources, and we discuss future work. Chrome has multiple \switch files" representing toggleable sets of related features, see Figure 1a. For example, there 2. TOGGLE BACKGROUND AND EXAM- are switch files for features related to how content is dis- played in Chrome (content_switches.cc) or to GPU testing PLE (test_switches.cc).