DOI:10.1145/2677034
Article development led by queue.acm.org What happened to the promise of rigorous, disciplined, professional practices for software development?
BY IVAR JACOBSON AND ED SEIDEWITZ A New Software Engineering
WHAT HAPPENED TO software engineering? What happened to the promise of rigorous, disciplined, professional practices for software development, like those observed in other engineering disciplines? What has been adopted under the rubric of “software engineering” is a set of practices largely adapted from other engineering dis- to make the software work—which ciplines: project management, design they do, regardless of what the design and blueprinting, process control, “blueprints” say should be done. and so forth. The basic analogy was Not surprisingly, this has led to a lot to treat software as a manufactured of dissatisfaction. product, with all the real “engineer- Today’s software craftsmanship ing” going on upstream of that—in movement is a direct reaction to the requirements analysis, design and engineering approach. Focusing on modeling, among others. the craft of software development, this Doing the job this way in other en- movement questions whether it even gineering disciplines makes sense be- makes sense to engineer software. Is cause the up-front work is based on a this the more sensible view? strong foundational understanding, Since it is the code that has to be so the results can be trusted. Software made to work in the end anyway, it engineering has had no such basis, so does seem sensible to focus on craft- “big up-front design” often just has not ing quality code from the beginning. paid off. Indeed, the ethos of software Coding, as a craft discipline, can then engineering has tended to devalue cod- build on the experience of software ers (if not explicitly, then implicitly “masters,” leading the community to through controlling practices). Coders, build better and better code. In addi- though, are the ones who actually have tion, many of the technical practices of
DECEMBER 2014 | VOL. 57 | NO. 12 | COMMUNICATIONS OF THE ACM 36 practice
agile development have made it possi- whole point of an engineering theory is ble to create high-quality software sys- to support practitioners, this is essen- tems of significant size using a craft ap- tially what was missing from previous proach—negating a major impetus for incarnations of software engineering. all the up-front activities of software How does the software community engineering in the first place. Today’s software go about this task of “refounding” soft- In the end, however, a craft disci- craftsmanship ware engineering? pline can take you only so far. From an- The SEMAT (Software Engineering cient times through the Middle Ages, movement is Method and Theory) initiative is an in- skilled artisans and craftsmen created a direct reaction ternational effort dedicated to answer- many marvelous structures, from the ing this question (http://www.semat. pyramids to gothic cathedrals. Unfor- to the engineering org). As the name indicates, SEMAT is tunately, these structures were incred- focusing both on supporting the craft ibly expensive and time consuming to approach. (methods) and building foundational build—and they sometimes collapsed Focusing on understanding (theory). in disastrous ways for reasons that This is still a work in progress, but were often not well understood. the craft of software the essence of a new software engineer- Modern structures such as skyscrap- development, ing is becoming clear. The remainder ers became possible only with the devel- of this article explores what this es- opment of a true engineering approach. this movement sence is and what its implications are Modern construction engineering has questions for the future of the discipline. a firm foundation in material science and the theory of structures, and con- whether it even Engineering Is Craft struction engineers use this theoreti- makes sense to Supported by Theory cal foundation as the basis of a careful, A method (equivalently, methodology disciplined approach to designing the engineer software. or process) is a description of a way of structures they are to build. working to carry out an endeavor, such Of course, such structures still Is this the more as developing software. Ultimately, all sometimes fail. When they do, how- sensible view? methods are derived from experience ever, a thorough analysis is again done with what does and does not work in to determine whether the failure was carrying out the subject endeavor. This caused by malfeasance or a shortcom- experience is distilled, first into rules ing in the underlying theory used in of thumb and then into guidelines and, the original design. Then, in the latter when there is consensus, eventually case, new understanding can be incor- into standards. porated into the foundational practice In a craft discipline, masters, who and future theory. have the long experience necessary, Construction engineering serves as largely develop methods. In older an example of how a true engineering times, the methods of a master were discipline combines craftsmanship closely guarded and passed down only with an applied theoretical founda- to trusted apprentices. In today’s world, tion. The understanding captured in however, various approaches based on such an accepted foundation is used the work of master craftsmen are often to educate entrants into the discipline. widely published and promoted. It then provides them with a basis for As a craft develops into an engineer- methodically analyzing and address- ing discipline, it is important to recog- ing engineering problems, even when nize commonality between the meth- those problems are outside the experi- ods of various masters, based on the ence of the engineers. underlying shared experience of the From this point of view, today’s soft- endeavor being carried out. This com- ware engineering is not really an engi- mon understanding is then captured neering discipline at all. in a theory that can be used as a basis What is needed instead is a new for all the different methods to be ap- software engineering built on the expe- plied to the endeavor. rience of software craftsmen, captur- In this sense, theory is not the bad ing their understanding in a founda- word it is sometimes treated as in our tion that can then be used to educate culture (“Oh, that’s just a theory”). and support a new generation of prac- As noted earlier, having a theoretical titioners. Because craftsmanship is re- foundation is, in fact, the key that al- ally all about the practitioner, and the lows for disciplined engineering analy-
37 COMMUNICATIONS OF THE ACM | NOVEMBER 2014 | VOL. 57 | NO. 11 practice sis. This is true of material science for ods and the teams of practitioners that by producing software in small incre- construction engineering, electromag- really use them. ments, obtaining feedback in rapid netic theory for electrical engineering, iterations, and continually adjusting aerodynamics for aeronautical engi- Agility Is for Methods, as necessary. neering, and so forth. Not Just Software Agile software-development teams Of course, the interplay between The current movement to promote take charge of their own way of work- the historical development of an engi- agility in software development com- ing. Such a team applies the methods neering discipline and its associated plements the recognition of software it feels it needs for the project at hand theory is generally more complicated craftsmanship. As the name sug- as they are needed, adapting the devel- than this simple explanation implies. gests, agile software development opment process throughout a project. Engineering experience is distilled is about promoting flexibility and In effect, an agile team needs to evolve into theory, which then promotes adaptability in the face of inevitably and improve its methods in as agile a better engineering, and back again. changing requirements. This is done fashion as it develops its software. Nevertheless, the important point to realize here is this: traditional soft- Figure 1. The kernel alphas. ware engineering did not have such an underlying theory. One might suggest computer sci- < provide ence provides the underlying theory Opportunity Stakeholders consume > for software engineering—and this focuses use and was, perhaps, the original expectation Customer < helps to address when software engineering was fi rst > conceived. In reality, however, com- < demand > support Software puter science has remained a largely Requirements System Solution constrains > < fulfills < produces Scopes and set up to
academic discipline, focused on the address > science of computing in general but mostly separated from the creation of software-engineering methods in updates and changes > < performs and plans industry. While “formal methods” Work Team from computer science provide the promise of doing some useful theo- < guides Endeavor < applies retical analysis of software, practitio- Way of ners have largely shunned such meth- Working ods (except in a few specialized areas such as methods for precise numeri- cal computation). As a result, there have often been cycles of dueling methodologies for Figure 2. Tracking progress with alphas. software “engineering,” without a true foundational theory to unite Opportunity them. In the end, many of these meth- ods did not even address the true needs of the skilled craft practitio- ners of the industry. Way of Working Stakeholders So, how to proceed? The creation of a complete, new the- ory of software engineering will take some time. Rather than starting with an academic approach, we can begin, as already mentioned, by capturing the commonality among the methods that have proven successful in the craft of Team Requirements software development. This, in turn, requires a common way of describing, understanding, and combining vari- ous software-development techniques, instead of setting them up in competi- tion with each other. Work Software System To see how this might be accom- plished, let’s take a closer look at meth-
NOVEMBER 2014 | VOL. 57 | NO. 11 | COMMUNICATIONS OF THE ACM 38 practice
Figure 3. Alphas made tangible with cards.
Opportunity Stakeholders Requirements Software va System
Identified Involved Addressed Demonstrable