TEXT 1 OF 31 development ONLY Industrial-Scale Agile from Craft to Engineering Essence is IVAR JACOBSON, IAN SPENCE, AND ED SEIDEWITZ instrumental in moving software n his paper titled “Industrial Scale Agile,”9 Roly Stimson development characterizes industrial-scale agile as: toward a true “Agile at any scale.” engineering “Agile as the rule, not the exception.” discipline “Agile sustainably, forever,” not just as an unrepeatable I“one-off.” This means being able to sustainably apply agile strategies appropriately to anything and everything that can benefit from them. This includes: 3 Being able to do “agile at scale” as and when appropriate. 3 Doing small-scale agile as and when possible/ appropriate. 3 Evolving the entire application landscape and not just individual applications. Although it is important, and a necessary precursor to industrial-scale agile, scaling agile is not the challenge here. Rather, it’s about how to achieve sustainability of the following: 3 The way of working in the face of ever-changing teams. acmqueue | september-october 2016 1 development 2 OF 31 3 The systems in the face of rapid change. 3 The application landscape as a whole. 3 Individuals and their careers, and the development organization as a whole. 3 Long-term investment in IT. There are many, many ways to illustrate how fragile IT investments can be. You just have to look at the way that, even after huge investments in education and coaching, many organizations are struggling to broaden their agile adoption to the whole of their organization—or at the way other organizations are struggling to maintain the momentum of their agile adoptions as their teams change and their systems mature. Another frequent example of unsustainability is in the way that many companies are facing an uncontrolled explosion in the number of applications they have to support and the overall cost of ownership of IT as a whole. So industrial-scale agile requires much more than just being able to scale agile. It also means taking a disciplined approach to ensuring that IT investments result in sustainable benefits for both the producing organization and its customers. This involves adopting a different approach to many aspects of agility. We need to look beyond small-scale agile, beyond independent competitive islands of agile excellence, beyond individual craftsmanship and heroic teams, and beyond the short-term instant gratification that seems to be the focus of many well-intentioned but self-centered agile teams. It is this adoption of a more holistic approach that we call moving from craft to engineering. (We have tried to keep this article short, acmqueue | september-october 2016 2 development 3 OF 31 but for readers new to this space, we recommend “A New Software Engineering”3 for more background.) FROM CRAFT TO ENGINEERING The move toward agility has led to many benefits for the software industry. It has broken the tyranny of the prescriptive waterfall approach to software engineering, an approach that was causing more and more large project failures, and it has allowed software developers to keep up with the ever-increasing demand for more innovative IT solutions. It has enabled many companies to do great things but in many cases has led to a culture of entitlement, heroic programming, and short-term thinking that threatens the sustainability of the parent companies and the IT solutions on which they depend. Little or no thought is put into maintainability, the heroes become potential single points of failure, and the cost of keeping the lights on just keeps growing and growing. What is needed is a way to maintain the values of agility while making software development more an engineering discipline than a craft—a new form of agile software engineering fit for the Internet Age. What are craft and engineering? The term craft is usually applied to people occupied in small-scale production of bespoke goods and trades where skills are passed in person from master to apprentice. Engineering, on the other hand, is defined by Wikipedia (https://en.wikipedia.org/wiki/Engineering) as “the application of mathematics, empirical evidence and acmqueue | september-october 2016 3 development 4 OF 31 scientific, economic, social, and practical knowledge in order to invent, innovate, design, build, maintain, research, and improve structures, machines, tools, systems, components, materials, and processes.” There have been many discussions about whether or not the term engineering should be applied to software development and whether or not software engineers are actually engineers. With the rise of cloud computing, big data, and the Internet of things, however, it is clear that there are many types of software and many aspects of software development that would benefit from an engineering approach. In her seminal paper, “Prospects for an Engineering Discipline of Software,”7 in 1990, Mary Shaw suggested that a definition of software engineering would include these clauses: “Creating cost-effective solutions…to practical problems…by applying scientific knowledge… building things…in the service of mankind.” She also said about software work that “most tasks are routine and not innovative,” but it “is treated more often as original than routine,” implying that there is a lot of potential for improving quality and shortening time to market “if we captured and organized what we already know” by codifying our knowledge, possibly even automating it. Her observations are still highly relevant; at the GoTo Amsterdam 2015 conference on software development, she talked about the progress that has been made toward establishing a software engineering discipline. According to Shaw, the characteristics of engineering are as follows: 3 Limited time, knowledge, and resources force decisions on tradeoffs. acmqueue | september-october 2016 4 development 5 OF 31 3 The best-codified knowledge, preferentially science, shapes design decisions. 3 Reference materials make knowledge and experience available. 3 Analysis of design predicts properties of implementation. Although software development shares many of the characteristics of an engineering discipline, we are not there yet. The rise of agile is not a problem unless this is where we stop. Why more engineering? Why is it important to move from craft to engineering? Doing so will help us cope with the ever-increasing challenges of a more automated, more interconnected world—where small improvements in software performance can make the difference between profit and loss; where a reputation for robustness, scalability, and security can add millions to the share price; and where software is more and more the public face of the business. The codified knowledge and professionalism of an engineering discipline are necessary for: 3 Sustaining and growing delivery capability through changes in technologies, teams, and suppliers. 3 Predictably scaling operations from early prototypes to global rollouts. 3 Taking control of investments and knowing when to pivot to solutions more likely to deliver favorable returns. 3 Systematically growing the levels of reuse and interoperability of solution components and systems. 3 Producing long-lived solutions with affordable costs of ownership. acmqueue | september-october 2016 5 development 6 OF 31 Perhaps not everybody needs to move from craft to engineering. As Mary Shaw says, “The greatest need for engineering discipline exists for software systems that are fully automated and are operating unattended and where the consequences of failure are catastrophic. Examples are telecom equipment, nuclear safety devices, medical implants, self-driving cars, and stock-trading programs. The need for engineering in software development depends upon how serious the consequences are when things go wrong and whether human beings can take action in time to minimize the consequences.” There is also a strong need for engineering systems used for e-commerce, finance, electronic medical records, and even human resources. The consequences of failures in such systems may not include the immediate loss of life, but they can still be “catastrophic” to either the businesses or the individuals affected. Thus, for many organizations and software systems craft is not enough. The good news is that there is a way forward that maintains the values of agile while making software development more of an engineering discipline than a craft. It involves: Engineering of software. This means the holistic engineering of all software to improve the application landscape as a whole, as well as the individual point solutions. Practices are needed that help teams engineer their software for capturing requirements and for developing software designed for engineering great products. It also means encouraging innovation in the large as well as the small—innovation of new business and new product acmqueue | september-october 2016 6 development 7 OF 31 opportunities as well as innovation that addresses the total cost of ownership impacting the whole organization, rather than just individual users and applications. Engineering of methods. Methods should be engineered to support the full range of development challenges faced today and in the future. The emerging best practice should be captured and codified in a way that makes it easy to communicate and share among teams, and enables each team to compose the method they need from this growing set of reusable, proven practices. Furthermore, moving from craft to engineering provides a robust platform for encouraging, establishing, and sustaining true organizational agility. Engineering of software How would software be developed if the craft were already a real engineering discipline? As in other engineering disciplines, it would be engineered by using commonly accepted, consistent practices that would be supported by models and analysis based on a common ground of foundational knowledge. In the past, such an engineering mindset has been misinterpreted as meaning “big upfront design,” with everything downstream of this being akin to manufacturing rather than engineering. Upfront blueprinting is, indeed, often necessary for the engineering of physical artifacts such as buildings, bridges, and cars.