DOI:10.1145/2677034

Article development led by queue.acm.org What happened to the promise of rigorous, disciplined, professional practices for development?

BY IVAR JACOBSON AND ED SEIDEWITZ A New Software Engineering

WHAT HAPPENED TO software engineering? What happened to the promise of rigorous, disciplined, professional practices for software development, like those observed in other engineering disciplines? What has been adopted under the rubric of “software engineering” is a set of practices largely adapted from other engineering dis- to make the software work—which ciplines: , design they do, regardless of what the design and blueprinting, process control, “blueprints” say should be done. and so forth. The basic analogy was Not surprisingly, this has led to a lot to treat software as a manufactured of dissatisfaction. product, with all the real “engineer- Today’s software craftsmanship ing” going on upstream of that—in movement is a direct reaction to the requirements analysis, design and engineering approach. Focusing on modeling, among others. the craft of software development, this Doing the job this way in other en- movement questions whether it even gineering disciplines makes sense be- makes sense to engineer software. Is cause the up-front work is based on a this the more sensible view? strong foundational understanding, Since it is the code that has to be so the results can be trusted. Software made to work in the end anyway, it engineering has had no such basis, so does seem sensible to focus on craft- “big up-front design” often just has not ing quality code from the beginning. paid off. Indeed, the ethos of software Coding, as a craft discipline, can then engineering has tended to devalue cod- build on the experience of software ers (if not explicitly, then implicitly “masters,” leading the community to through controlling practices). Coders, build better and better code. In addi- though, are the ones who actually have tion, many of the technical practices of

DECEMBER 2014 | VOL. 57 | NO. 12 | COMMUNICATIONS OF THE ACM 36 practice

agile development have made it possi- whole point of an engineering theory is ble to create high-quality software sys- to support practitioners, this is essen- tems of significant size using a craft ap- tially what was missing from previous proach—negating a major impetus for incarnations of software engineering. all the up-front activities of software How does the software community engineering in the first place. Today’s software go about this task of “refounding” soft- In the end, however, a craft disci- craftsmanship ware engineering? pline can take you only so far. From an- The SEMAT (Software Engineering cient times through the Middle Ages, movement is Method and Theory) initiative is an in- skilled artisans and craftsmen created a direct reaction ternational effort dedicated to answer- many marvelous structures, from the ing this question (http://www.semat. pyramids to gothic cathedrals. Unfor- to the engineering org). As the name indicates, SEMAT is tunately, these structures were incred- focusing both on supporting the craft ibly expensive and time consuming to approach. (methods) and building foundational build—and they sometimes collapsed Focusing on understanding (theory). in disastrous ways for reasons that This is still a work in progress, but were often not well understood. the craft of software the essence of a new - Modern structures such as skyscrap- development, ing is becoming clear. The remainder ers became possible only with the devel- of this article explores what this es- opment of a true engineering approach. this movement sence is and what its implications are Modern construction engineering has questions for the future of the discipline. a firm foundation in material science and the theory of structures, and con- whether it even Engineering Is Craft struction engineers use this theoreti- makes sense to Supported by Theory cal foundation as the basis of a careful, A method (equivalently, methodology disciplined approach to designing the engineer software. or process) is a description of a way of structures they are to build. working to carry out an endeavor, such Of course, such structures still Is this the more as developing software. Ultimately, all sometimes fail. When they do, how- sensible view? methods are derived from experience ever, a thorough analysis is again done with what does and does not work in to determine whether the failure was carrying out the subject endeavor. This caused by malfeasance or a shortcom- experience is distilled, first into rules ing in the underlying theory used in of thumb and then into guidelines and, the original design. Then, in the latter when there is consensus, eventually case, new understanding can be incor- into standards. porated into the foundational practice In a craft discipline, masters, who and future theory. have the long experience necessary, Construction engineering serves as largely develop methods. In older an example of how a true engineering times, the methods of a master were discipline combines craftsmanship closely guarded and passed down only with an applied theoretical founda- to trusted apprentices. In today’s world, tion. The understanding captured in however, various approaches based on such an accepted foundation is used the work of master craftsmen are often to educate entrants into the discipline. widely published and promoted. It then provides them with a basis for As a craft develops into an engineer- methodically analyzing and address- ing discipline, it is important to recog- ing engineering problems, even when nize commonality between the meth- those problems are outside the experi- ods of various masters, based on the ence of the engineers. underlying shared experience of the From this point of view, today’s soft- endeavor being carried out. This com- ware engineering is not really an engi- mon understanding is then captured neering discipline at all. in a theory that can be used as a basis What is needed instead is a new for all the different methods to be ap- software engineering built on the expe- plied to the endeavor. rience of software craftsmen, captur- In this sense, theory is not the bad ing their understanding in a founda- word it is sometimes treated as in our tion that can then be used to educate culture (“Oh, that’s just a theory”). and support a new generation of prac- As noted earlier, having a theoretical titioners. Because craftsmanship is re- foundation is, in fact, the key that al- ally all about the practitioner, and the lows for disciplined engineering analy-

37 COMMUNICATIONS OF THE ACM | NOVEMBER 2014 | VOL. 57 | NO. 11 practice sis. This is true of material science for ods and the teams of practitioners that by producing software in small incre- construction engineering, electromag- really use them. ments, obtaining feedback in rapid netic theory for electrical engineering, iterations, and continually adjusting aerodynamics for aeronautical engi- Agility Is for Methods, as necessary. neering, and so forth. Not Just Software Agile software-development teams Of course, the interplay between The current movement to promote take charge of their own way of work- the historical development of an engi- agility in software development com- ing. Such a team applies the methods neering discipline and its associated plements the recognition of software it feels it needs for the project at hand theory is generally more complicated craftsmanship. As the name sug- as they are needed, adapting the devel- than this simple explanation implies. gests, agile software development opment process throughout a project. Engineering experience is distilled is about promoting flexibility and In effect, an agile team needs to evolve into theory, which then promotes adaptability in the face of inevitably and improve its methods in as agile a better engineering, and back again. changing requirements. This is done fashion as it develops its software. Nevertheless, the important point to realize here is this: traditional soft- Figure 1. The kernel alphas. ware engineering did not have such an underlying theory. One might suggest computer sci- < provide ence provides the underlying theory Opportunity Stakeholders consume > for software engineering—and this focuses use and was, perhaps, the original expectation Customer < helps to address when software engineering was fi rst > conceived. In reality, however, com- < demand > support Software puter science has remained a largely Requirements System Solution constrains > < fulfills < produces Scopes and set up to

academic discipline, focused on the address > science of computing in general but mostly separated from the creation of software-engineering methods in updates and changes > < performs and plans industry. While “formal methods” Work Team from computer science provide the promise of doing some useful theo- < guides Endeavor < applies retical analysis of software, practitio- Way of ners have largely shunned such meth- Working ods (except in a few specialized areas such as methods for precise numeri- cal computation). As a result, there have often been cycles of dueling methodologies for Figure 2. Tracking progress with alphas. software “engineering,” without a true foundational theory to unite Opportunity them. In the end, many of these meth- ods did not even address the true needs of the skilled craft practitio- ners of the industry. Way of Working Stakeholders So, how to proceed? The creation of a complete, new the- ory of software engineering will take some time. Rather than starting with an academic approach, we can begin, as already mentioned, by capturing the commonality among the methods that have proven successful in the craft of Team Requirements software development. This, in turn, requires a common way of describing, understanding, and combining vari- ous software-development techniques, instead of setting them up in competi- tion with each other. Work Software System To see how this might be accom- plished, let’s take a closer look at meth-

NOVEMBER 2014 | VOL. 57 | NO. 11 | COMMUNICATIONS OF THE ACM 38 practice

Figure 3. Alphas made tangible with cards.

Opportunity Stakeholders Requirements Software va System

Identified Involved Addressed Demonstrable

Opportunity identified that could be Stakeholder representatives carry Enough requirements are implemented Key architecture characteristics addressed by a software-based solution out responsibilities for the system to be acceptable demonstrated Stakeholder representatives Relevant stakeholders agree A stakeholder wishes to make provide feedback and take part Stakeholders agree the system architecture is appropriate an investment in better understanding in decisions in a timely way is worth making operational potential value Critical interface and system Stakeholder representatives configurations exercised Other stakeholders who share promptly communicate to opportunity identified stakeholder group

1 / 6 3 / 6 5 / 6 2 / 6

Work Team Way of Working

Under Control Performing In Use

Work going well, risks being Team working efficiently Some members of the team managed and effectively are using the way of working Unplanned work and re-work Adapts to changing context Use of practices and tools under control Produce high quality output regularly inspected Work items completed within estimates Minimal backtracking and re-work Practices and tools being adapted and supported by team Measures tracked Waste continually eliminated Procedures in place to handle feedback

4 / 6 4 / 5 3 / 6

A lack of agility in methods is a than process control. As a result, ag- introduces practices such as main- central failure of traditional software ile development focuses on support- taining a backlog, daily scrums, and engineering. ing the practitioner in building qual- sprints. Scrum is not really a complete Software is, by its very nature, mal- ity software, rather than requiring the method but a composite practice built leable and (physically) easy to change. practitioner to support the process. from a number of other practices de- A complicated software system, how- So, how do you introduce agility into signed to work together. Scrum, how- ever, can exhibit a kind of intellectual software-engineering methods? By ever, can be used as a process frame- rigidity in which it is difficult to make looking at the basic things that practi- work combined with practices from, changes correctly, with each change in- tioners actually do—their practices. say, , to form troducing as many or more errors as it the method used by an agile team. resolves. In the face of this, the response Methods Are Made from Practices That exemplifies the power of ex- of traditional software engineering was A method may appear monolithic, but plicitly considering how methods are to adopt process-control and project- any method may be analyzed as being made up of practices. Teams can pull management techniques such as those composed of a number of practices. A together the practices that best fit the used to handle similar problems with practice is a repeatable approach to do- development task at hand and the complicated hardware systems. ing something with a specific purpose skills of the team members involved. From an agile viewpoint, however, in mind. Practices are the things that Further, when necessary, a team can the application of hardware-engineer- practitioners actually do. evolve its method in not only small ing techniques was a mistake. Agile For example, the agile method of steps, but also more radical and bigger techniques, instead, take advantage of Extreme Programming is described as steps such as replacing an old practice the changeable nature of software, us- having 12 practices, including pair pro- with a better practice (without having ing quick feedback cycles allowed by gramming, test-driven development, to change any other practices). continuous integration and integrated and continuous integration. The agile Note how the focus is on teams testing to manage complexity, rather framework Scrum, on the other hand, and the practitioners in teams, rather

39 COMMUNICATIONS OF THE ACM | NOVEMBER 2014 | VOL. 57 | NO. 11 practice than “method engineers,” who create ing introduced for the kernel. In the Of course, this ability to use the methods for other people to carry out. end, a new term was adopted without kernel to describe practices is exactly Creating their own way of working is any of the old baggage.) The seven what is needed as a foundation for a new responsibility for a lot of teams, dimensions are: opportunity, stake- true software-engineering methods. however, and it is also necessary to sup- holders, requirements, software sys- port a team’s ability to do this across tem, work, team, and way of working. Practices Built on the Kernel projects. It is also useful, therefore, to These alphas relate to each other as Enable Agile Methods provide for groups interested in creat- shown in Figure 1. A practice can be expressed in terms of ing and extending practices, outside of Each alpha has a specific set of the kernel by: any specific project, so they can then be states that codify points along the ! Identifying the areas in which it ad- used as appropriate by project teams. dimension of progress represented vances the endeavor. This can be seen as a separation of by the alpha. Each of the states has a ! Describing the activities used to concerns: practices can be created and checklist to help practitioners moni- achieve this advancement and the grown within an organization, or even tor the current state of their endeavor work products produced. by cross-organization industry groups along a certain alpha and to under- ! Describing the specific competen- (such as is effectively the case with Ex- stand the state they need to move to- cies needed to carry out these activities. treme Programming and Scrum); prac- ward next. The idea is to provide an A practice can also extend the ker- titioners on project teams can then intuitive tool for practitioners to rea- nel with additional states, checklists, adopt, adapt, and apply these practices son about the progress and health of or even new alphas. as appropriate. their endeavors in a common, meth- The critical point is that the kernel What assurance do project teams od-independent way. provides a common framework for have that disparately created practices One way to visualize the seven- describing all practices and allowing can actually be smoothly combined dimensional space of alphas is using them to be combined into methods. to produce effective methods? This the spider chart1 shown in Figure 2. Bringing a set of practices into this is where a new software-engineering In this chart, the gray area represents common system allows gaps and over- foundation is needed, independent how far an endeavor has progressed, laps to be more easily identified. The of practices and methods but able to while the white area shows what still gaps can then be filled with additional provide a common underpinning for needs to be completed before the en- practices and the overlaps resolved by them. deavor is done. A quick look at such connecting the overlapping practices a diagram provides a good idea of together appropriately. The Kernel Is the Foundation where a project is at any point in time. For example, consider two prac- for Practices and Methods The alphas can be made even more tices: one about using a backlog to The first tangible result of the SEMAT tangible by putting each of the alpha manage the work to be carried out by initiative is what is known as the ker- states on a card, along with the state a team (advancing the work alpha); the nel for software engineering. This checklist in an abbreviated form other about defining requirements us- kernel can be thought of as the mini- (see Figure 3). The deck of all such ing user stories (advancing the require- mal set of things that are universal to cards can then fit easily into a per- ments alpha). The backlog practice all software-development endeavors. son’s pocket. Although more detailed does not prescribe what the items on The kernel consists of three parts: guidelines are available, these cards the backlog must be, while the user- ! A means for measuring the prog- contain key reminders that can be story practice does not prescribe how ress and health of an endeavor. used by development teams in their the team should manage the imple- ! A categorization of the activities daily work, much like an engineer’s mentation of those stories. The two necessary to advance the progress of handbook in other disciplines. practices are thus complementary and an endeavor. A more complete discussion of the can be used together—but, when so ! A set of competencies necessary kernel and its application is available combined, they overlap. The two prac- to carry out such activities. in previous work.2,3 The kernel itself tices can be connected in a smooth and Of particular importance is having is formally defined as part of the Es- intuitive way within an overall method a common means for understanding sence specification that has been by identifying backlog items from the how an endeavor is progressing. The standardized through the Object one with user stories from the other, SEMAT kernel defines seven dimen- Management Group.6 In addition to so that user stories become the items sions for measuring this progress, the full kernel, the Essence standard managed on the backlog. known as alphas. (The term alpha also defines a language that can be Note, in particular, how the com- was originally an acronym for ab- used both to represent the kernel and mon framework of the kernel provides stract-level progress health attribute to describe practices and methods in a predictive capability. A construction but is now simply used as the word terms of the kernel. Importantly, this engineer can use material science for a progress and health dimension language is intended to be usable by and the theory of structures to under- as defined in the kernel. Many other practitioners, not just method engi- stand at an early stage whether a pro- existing terms were considered, but neers; for basic uses, it can be learned posed building is likely to stand or fall. all had connotations that clashed in just a couple of hours (the alpha Similarly, using the kernel, a software with the essentially new concept be- state cards are an example of this). developer can understand whether a

NOVEMBER 2014 | VOL. 57 | NO. 11 | COMMUNICATIONS OF THE ACM 40 practice

proposed method is well constructed, the development of a true engineer- coming part of this “refounding” of and, if there are gaps or overlaps in its ing discipline as a necessary evolu- software engineering—or, perhaps, re- practices, how to resolve those. tion from their (just recently hard- ally founding it for the first time. Further, through the separation won!) craft discipline. of concerns discussed earlier, an or- In regard to the second point, in his ganization or community can build foreword to The Essence of Software En- Related articles up a library of practices and even ba- gineering: Applying the SEMAT Kernel,3 on queue.acm.org sic methods that a new project team Robert Martin, one of the SEMAT sig- The Essence of Software Engineering: may draw on to form its initial way natories, describes a classic pendulum The SEMAT Kernel of working. Each team can then con- swing away from software engineering Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, tinue to agilely adapt and evolve its toward software craftsmanship. Mar- Ian Spence and Svante Lidman own methods within the common tin’s assessment is correct, but it is http://queue.acm.org/detail.cfm?id=2389616 Essence framework.4 important to note that this proverbial First, Do No Harm: A Hippocratic Oath Ultimately, the goal will be, as an pendulum should not simply swing for Software Developers Phillip A. Laplante industry, to provide for the standard- back in the direction it came. To the http://queue.acm.org/detail.cfm?id=1016991 ization of particularly useful and suc- contrary, while swing it must, it now Software Development with Code Maps cessful practices, while enhancing, needs to swing in almost a 90-degree Robert DeLine, Gina Venolia and Kael Rowan not limiting, the agility of teams in different direction from which it came, http://queue.acm.org/detail.cfm?id=1831329 applying and adapting those prac- in order to move toward a new disci- tices, as well as building new ones pline of true software engineering. as necessary. And that, finally, is the There is, perhaps, hardly a better im- References 1. Graziotin, D. and Abrahamsson, P. A Web-based path toward a true discipline of soft- age for a paradigm shift than that. In the modeling tool for the SEMAT Essence theory of ware engineering. end, the new paradigm of software en- software engineering. J. Open Research Software 1, 1 (2013), e4; http://dx.doi.org/10.5334/jors.ad. gineering, while building on the current 2. Jacobson, I., Ng, P-W., McMahon, P., Spence, I. and Conclusion paradigm of software craftsmanship, Lidman, S. The Essence of software engineering: The SEMAT kernel. ACM Queue 10, 10 (2012); http://queue. The term paradigm shift may be a bit must move beyond it, but it will also be acm.org/detail.cfm?id=2389616. overused these days; nevertheless, a shift away from the old paradigm of 3. Jacobson, I., Ng, P.-W., McMahon, P. E., Spence, I. and Lidman, S. The Essence of Software Engineering: the kernel-based Essence approach to traditional software engineering. And, Applying the SEMAT Kernel. Addison-Wesley, Reading, software engineering can quite reason- like all paradigm shifts before, this one PA, 2013. 4. Jacobson, I., Spence, I. and Ng, P.-W. Agile and SEMAT— ably be considered to be such a shift. It will take considerable time and effort Perfect partners. Comm. ACM 6, 11 (Nov. 2013); http:// cacm.acm.org/magazines/2013/11/169027-agile-and- truly represents a profound change of before it is complete—at which point, semat/abstract. viewpoint for the software-engineering as the new paradigm, everyone will con- 5. Kuhn, T. The Structure of Scientific Revolutions. University of Chicago Press, 1962. community. sider its benefits obvious. 6. . Essence—Kernel and When Thomas Kuhn introduced Even as it stands today, though, us- language for software engineering methods, 2014; the concept of a paradigm shift in ing Essence can provide a team with http://www.omg.org/spec/Essence. his influential book, The Structure of some key benefits. Essence helps Scientific Revolutions,5 he stressed the teams to be agile when working with Ivar Jacobson is the founder and chairman of Ivar Jacobson International. He is a father of components difficulty (Kuhn even claimed impos- methods and to measure progress in and architecture, use cases, aspect-oriented sibility) of translating the language terms of real outcomes and results of software development, modern business engineering, the Unified , and the Rational Unified and theory of one paradigm into an- interest to stakeholders. These prog- Process. other. The software-development ress measurements are not only on one community has actually seen such dimension, but along the seven dimen- Ed Seidewitz is the former CTO, Americas, for Ivar Jacobson International and is currently chair of the shifts before, in which those steeped sions of the kernel alphas, all of which ongoing Essence Revision Task Force. With Ivar Jacobson in the old paradigm have trouble need to move along at some pace to re- International, he has led agile system architecture and development engagements in both the commercial even understanding what the new duce risks and achieve results. and government sectors and participated in practice paradigm is all about. The move to Further, Essence can allow orga- development. object orientation was one such shift, nizations to simplify governance of Copyright held by owners/authors. Publication rights as, in many ways, is the current shift methods, using a pool of practices licensed to ACM. $15.00, to agile methods. that may be adopted and adapted by In this regard, Essence can, in- project teams. Having Essence as a deed, be considered a paradigm shift common foundation for this also al- in two ways. First, those steeped in lows practitioners to learn from one the “old school” of software engi- another more readily. neering have to start thinking about The real shift, however, will only the true engineering of software spe- come as teams truly realize the ben- cifically, rather than just applying efits of Essence today and as SEMAT practices largely adapted from other builds on Essence to complete the engineering disciplines. Second, new software engineering paradigm. those in the software craftsmanship A community of practitioners is now and agile communities need to see contributing their experience and be-

41 COMMUNICATIONS OF THE ACM | NOVEMBER 2014 | VOL. 57 | NO. 11