Components of a Modern Quality Approach To Software Development Dolores Zage, Wayne Zage Ball State University Sponsor: iconectiv Final Report October 2015 1 Table of Contents Section 1: Software Development Process and Quality 4 1.1 New Versus Traditional Process Overview 4 1.2 DevOps 5 1.2.1 Distinguishing Characteristics of High-Performing DevOps Development Cultures 5 1.2.2 Zeroturnaround Survey 5 1.2.2.1 Survey: Quality of Development 6 1.2.2.2 Survey Results: Predictability 6 1.2.2.3 Survey Results: Tool Type Usage 7 1.2.3 DevOps Process Maturity Model 8 1.2.4 DevOps Workflows 9 1.3 Code Reviews 10 1.3.1 Using Checklists 11 1.3.1.1 Sample Checklist Items 11 1.3.2 Checklists for Security 12 1.3.3 Monitoring the Code Review Process 12 1.3.4 Evolvability Defects 13 1.3.5 Other Guidelines 13 1.3.6 High Risk Code and High Risk Changes 14 1.4 Testing 14 1.4.1 Number of Test Cases 15 1.5 Agile Process and QA 16 1.5.1 Agile Quality Assessment (QA) on Scrum 17 1.6 Product Backlog 19 Section 2: Software Product Quality and its Measurement 19 2.1 Goal Question Metric Model 19 2.2 Generic Software Quality Models 20 2.3 Comparing Traditional and Agile Metrics 22 2.4 NESMA Agile Metrics 23 2.4.1 Metrics for Planning and Forecasting 23 2.4.2 Dividing the Work into Manageable Pieces 24 2.4.3 Metrics for Monitoring and Control 24 2.4.4 Metrics for Improvement (Product Quality and Process Improvement) 25 2.5 Another State-of-the-Practice Survey on Agile Metrics 26 2.6 Metric Trends are Important 27 2.7 Defect Measurement 28 2.8 Defects and Complexity Linked 32 2.9 Performance, a Factor in Quality 35 2.10 Security, a Factor in Quality 37 2.10.1 Security Standards 37 2.10.2 Shift of Security Responsibilities within Development 39 2 2.10.3 List of Current Practices 40 2.10.4 Risk of Third Party Applications 40 2.10.5 Rate of Repairs 41 2.10.6 Other Code Security Strategies 41 2.10.7 Design Vulnerabilities 42 Section 3: Assessment of Development Methods and Project Data 43 3.1 The Namcook Analytics Software Risk Master (SRM) tool 43 3.2 Crosstalk Table 46 3.2.1 Ranges of Software Development Quality 49 3.3 Scoring Method of Methods, Tools and Practices in Software Development 50 3.4 DevOps Self-Assessment by IBM 50 3.5 Thirty Software Engineering Issues that have Stayed Constant for Thirty Years 51 3.6 Quality and Defect Removal 52 3.6.1 Error-Prone Modules (EPM) 52 3.6.2 Inspection Metrics 53 3.6.3 General Terms of Software Failure and Software Success 53 Section 4: Conclusions and Project Take-Aways 53 4.1 Process 53 4.2 Product Measurements 55 Acknowledgements 56 Appendix A – Namcook Analytics Estimation Report 56 Appendix B – Sample DevOps Self-Assessment 65 References 68 3 Section 1: Software Development Process and Quality 1.1 New Versus Traditional Process Overview At first enterprises used Agile development techniques for pilot projects developed by small teams. Realizing the benefits of shorter delivery and release phases and responsiveness to change while still delivering quality software, enterprises searched for methods to mimic similar benefits for their larger development efforts by scaling Agile. Thus, many frameworks and methods were developed to satisfy this need. Scaled Agile Framework (SAFe) is one of the most implemented scaled Agile frameworks. Most of the pieces of the framework are borrowed, existing Agile methods packaged and organized in a different way to accommodate the scale. Integrated within SAFe and other enterprise development scaled methodologies are Agile methods, such as Scrum and other techniques to foster the delivery of software. Why evaluate the process? Developing good software is difficult and a good process or method can make this difficult task a little easier and perhaps more predictable. In the past, researchers performed an analysis of software standards and determined that standards heavily focus on processes rather than products [PFLE]. They characterized software standards as prescribing “the recipe, the utensils, and the cooking techniques, and then assume that the pudding will taste good.” This corresponds with Deming who argued that, “The quality of a product is directly related to the quality of the process used to create it” [DEMI]. Watts Humphrey, the creator of the CMM, believes that high quality software can only be produced by a high quality process. Most would agree that the probability of producing high quality software is greater if the process is also of high quality. All recognize that possessing a good process in isolation is not enough. The process has to be filled with skilled, motivated people. Can a process promote motivation? Agile methods are people-oriented rather than process-oriented. Agile methods assert that no process will ever make up the skill of the development team and, therefore, the role of a process is to support the development team in their work. Another movement, DevOps, encourages collaboration through integrating development and operations. Figure 1 compares the old and new way of delivering software. The catalyst for many of the enumerated changes is teamwork and cooperation. Everyone should be participating and all share in responsibility and accountability. In true Agile, teams have the autonomy of choosing development tools. In the new world, disconnects should be removed and the development tools and processes should be chosen so that people can receive feedback quickly and make necessary changes. Successful integration of DevOps and Agile development will play a key role in the delivery of rapid, continuous, high quality software. Institutions that can accomplish this merger at the enterprise scale will outpace those struggling to adapt. Figure 1: Old and New Way of Developing Software For this reason, identifying the characteristics of high-performing Agile and DevOps cultures is important to assist in outlining a new transformational technology. 4 1.2 DevOps DevOps is the fusion of “Development” and “Operations” functions to reduce risk, liability, and time-to- market while increasing operational awareness. Over the past decade, it is one of the newest and largest movements in Information Technology. DevOps evolution stemmed from many of the previous ideas in software development such as automation tooling, culture shifts, and iterative development models such as Agile. During the fusion process, DevOps was not provided with a central set of operational guidelines. Once an organization decided to use DevOps for its software development, it had free reign on deciding how to implement DevOps which produced its own challenges. Even Agile, whose many attributes were adopted by the DevOps movement, falls into the same predicament. In 2006, Gilb and Brody wrote an article suggesting the same lack of quantification for Agile methods, and this is a major weakness [GILB]. There is insufficient focus on quantified performance levels such as metrics evaluating required qualities, resource savings and workload capacities of the developed software. This omission of not being able to measure the change to DevOps does not suggest that DevOps does not prescribe monitoring and measuring. The purpose of monitoring and measuring within DevOps is to compare the current state of a project to the same project from some time in the near past, providing an answer to current project progress. However, this quantification does not help to answer the question about benchmarking the overall DevOps implementation. 1.2.1 Distinguishing Characteristics of High-Performing DevOps Development Cultures A possible description of a high-performing DevOps environment is that it produces good quality systems on time. It is important to identify the characteristics of such high-performing cultures so that these practices are emulated and metrics can be identified that quantify these typical characteristics and track successful trends. Below are seven key points of a high-performing DevOps culture [DYNA]: 1. Deploy daily – decouple code deployment from feature releases 2. Handle all non-functional requirements (performance, security) at every stage 3. Exploit automated testing to catch errors early and quickly - 82% of high-performance DevOps organizations use automation [PUPP]. 4. Employ strict version control – version control in operations has the strongest correlation with high performing DevOps organizations [PUPP]. 5. Implement end-to-end performance monitoring and metrics 6. Perform peer code review 7. Allocate more cycle time to reduction of technical debt. Other attributes of high-performing DevOps environments are that operations are involved early on in the development process so that plans for any necessary changes to the production environment can be formulated, and a continuous feedback loop between development, testing and operations is present. Just as development has its daily stand-up meeting, development, QA and operations should meet frequently to jointly analyze issues in production. 1.2.2 Zeroturnaround Survey Many of the above trends are highlighted in a survey conducted by Zeroturnaround of 1,006 developers on the practices developers use during software development. The goal of this survey was to prove or disprove the effectiveness of the best quality practices including the methodologies, tools and company size and industry within the context of these practices [KABA]. Noted in the report was that the survey respondents possessed a disproportionate bias towards Java and were employed in Software/Technology 5 companies. Zeroturnaround also divided the respondents into three categories depending on their responses into three groups. The top 10% were identified as rock stars, the middle 80% as average and the bottom 10% as laggards. For this report, Zeroturnaround concentrated on two aspects of software development, namely, the quality of the software and the predictability of delivery.