The

Research Crisis

ROBERTL. GLASS,Computing Trends

With the advantage of computing and software research are ’ using only one of them. Even though more than 25 years’ hindsight, certainly glad we’ve been able to put we now understand how terribly limit- this twenty-first century author the “research crisis” behind us. ’ ing that approach was, it is easy to see, What was this crisis? It was the in retrospect, why computer scientists looks askance at the “misis”in realization that occurred, right around , could not see the error in their ways. the turn of the century, that research Before I continue, I think it is software practice and expresses in computing and software - as it was worth spending a moment or two deep concern fir a Cris in then focused - was all too often both elaborating on the term research crisis. arrogant and narrow. ’ There is a fine irony to it, and the software research. It was arrogant because many corn- story makes good telling.

puting researchers of that era were ~ To understand this research crisis, doing research in a topic they thought it is important that we confront and they understood, but didn’t. It is deal with a prior “software crisis.” amazing that, in retrospect, those ’ It has been about 50 years now computer scientists simply didn’t since the software community coined know what they didn’t know. , the term sojhuare crisis. It was invented It was narrow because, of all the at a conference predominantly attend- possible research models twentieth , ed by theorists rather than practition- century computer scientists might ers, but still, at the outset, almost

42 07407459/94/$04 W 0 1994 IEEE NOVEMBER 1994 everyone agreed there was some validi- ing Office on the problems of building resentment by software practitioners ty to the term. What did “software cri- software, as support for their claims of of software theorists and software the- sis” mean? It is hard to believe, now, xisis. ory. Most people in a profession would but what it meant way back then was In most cases, this GAO study was be offended, of course, by the notion that the practice of software was in cri- the only real, nonanecdotal data cited, that their projects and products were sis - that it was characterized by pro- and almost every researcher cited it. It constantly characterized as “behind jects that were “always over budget, seemed to say that most software pro- schedule, over budget, and unreliable.” behind schedule, and unreliable.” jects failed, and that most money spent Ironically, computing researchers With the advantage of hindsight, on software was wasted. never understood how offensive their we can look back on the last century as Fortunately, cooler heads began to software-crisis campaign had been to a time of wonder. Of all the dramatic prevail. One researcher, then a voice practitioners. It is fair to say, in the changes that took place in the world, crying in the wilderness, brought an enlightened year 2020, that those old computing and software gave rise to end to the use of the resentments still linger, and Dartlv because of the name “computing era” that we term software crisis. II now recognize as the proper character- Bruce Blum, arguably CRISIS, INDEED! them, new theory still ization of the last half of the twentieth one of the few research- has a difficult time pene- century. Oh, there were software pro- ers of the time who actu- THE TWENTIETH trating current practice. ject failures, of course. Some of them ally read the study (most But back to the 1990s. were even catastrophic - runaway researchers apparently New government studies described the worst of them. But their copied the data from A TIME OF and open-minded com- existence provided only anecdotal evi- some other researcher’s puting researchers both dence, certainly not solid data, to sup- paper without going IGNoRANCEfbegan pointing out that port the claims of crisis. In fact, the back to the source), dis- BUT ALSO THE serious problems existed issue of solid data became the shoal on covered the data was in the 1990s model of which the claims of crisis foundered, as being misunderstood DAWN OF THE software research. In the we will see later in this story. and misused.’ The study, GOLDEN AGE early 1990s, for example, Software crisis indeed! We may an analysis of govern- a couple of government look back on that earlier time as archa- ment software projects, OF PRACTICE* studies concluded that ic and ignorant, perhaps, but certainly examined only -projects research was ignoring not as a crisis. It was the beginning, in that were in trouble practice almost entirely fact, of the Golden Age of Computing when it was conducted. Given that, the and, in addition, ignoring the notion Practice that persists today. study’s conclusion that most such pro- of practical application. There was a Nevertheless, the notion of soft- jects failed, and most money spent on flurry of opposition to those reports, ware crisis reached a fever pitch of them was wasted, was interesting - as researchers mired in the old ways intensity in the 1980s and 1990s. “troubled projects frequently fail” - resisted change (the very thing they Researchers began nearly every paper but hardly a basis for blackening the accused practitioners of!), but eventu- on software engineering by invoking reputation of all software practice. ally most computing researchers 1 the software crisis as a reason for lis- Even after the Blum revelation, a began to realize that theirs was truly a tening to whatever new theoretical few computing researchers continued, troubled endeavor. A flurry of criti- notion they were advocating. Re- for their own self-serving reasons, to cisms stirred the pot: Software-engi-

~ searchers at the time didn’t realize that use the GAO data and cry “software neering research was described as what many of them were doing would crisis,” knowing that the data was increasing in quantity but not in quali- later be characterized, derisively, as being misused. But eventually common ty, laclung in evaluation, “becoming advocacy research. But the software crisis sense and ethics prevailed, and the less credible,” having a “gaping hole” was in fact the platform on which most notion of a software crisis slowly died in its “generally accepted methods,” of this advocacy research was founded. away. Furthering its demise was and in need of a “paradigm shift ... Michiel van Genuchten’s data, which from purely theoretical and building- presented findings from other re- oriented to experimental ....” 3 SOFTWARE CRISIS DISCREDITED searchers to the effect that typical soft- Later, in an exchange of letters to ware overruns were 33-36 percent over the editor, both a letter-writer and the At the time, researchers frequently budget and 22 percent behind schedule, authors of the article addressed by the cited both the anecdotal evidence clearly a problem but hardly a crisis. 2 letter-writer agreed that, “software referred to earlier and a government One lingering aftereffect of this so- research is in a sad state,” and that, study from the US General Account- called crisis, however, was a deep “Without a scientific method, tech-

IEEE SOFTWARE 43 nobabble, lemmingineering, and fads gant was that the typical researcher sure and analyze, validate the model, run rampant.”4 had never worked in software practice repeat. At the time, it seemed natural to and had no basis for assuming that his + The analytical method. Propose a shift the crisis from practice to re- or her idea would really work there. formal theory or set of axioms, devel- search. This was not done in a spirit of Most researchers had a mental model op a theory, derive results, and if pos- meanness, but rather in the original of software practice as an enterprise in sible compare with empirical observa- fundraising spirit of the term. Re- crisis, one that did a bad job of what- tions. searchers proclaimed a “research cri- ever it undertook. There seemed to be Then, a few years later, another sis” to obtain funds for newer and bet- an underlying assumption in most paper also addressed the issue,’ break- ter ways of doing research, just as their research that any change was better ing research down into four phases: forerunners had used the practitioner than the status quo. + The informationalphase. Gather or “software crisis” as a way of obtaining The problem got so bad, in fact, aggregate information via reflection, funding to investigate the problems that the software consortia and institu- literature survey, people/organization- that crisis implied. For quite a long tions of the time were sometimes char- a1 survey, or poll. time, cries of “research crisis” were as acterized as “arrogant and ignorant.” + The propositional phase. Propose prevalent in the computing literature This charge was unfair to many of and/or build a hypothesis, method or as mentions of its predecessor had them, of course, but there was enough algorithm, model, theory, or solution. been. As I mentioned at the beginning truth to it that the notion stuck. It is + The analytical phase. Analyze and of this essay, that notion, like the earli- difficult for a research-and-develop- explore a proposal, leading to a de- er crisis that preceded it, has died out. ment institution to be effective when monstration and/or the formulation of But it was a long time in D its --people are seen as ar- a principle or theory. the dying. rogant and ignorant. + The evaluative phase. Evaluate a THE ONLY That problem nearly des- proposal or analpc finding by means troyed the effectiveness of experimentation (controlled) or FOCUS ON RESEARCH RESEARCH of those institutions be- observation (uncontrolled, such as a MODEL IN fore they managed to over- case study or protocol analysis), per- Back to those twin come it. But that is an- haps leading to a substantiated model, notions of arrogant and USE WAS AN other story. principle, or theory. narrow that character- ANALYTICAL What about the word Once those papers were absorbed ized twentieth century narrow? Basically, there by the field, the conclusion was computing and software METHOD was only one research inevitable. Almost no computing research. What was that DEVOID OF model used in most twen- research to that time had used the sci- all about? tieth century computing. entific method (It begins with Let’s start with arro- EMPIRICAL I have already character- “observe the world.” No one was do- gant. Remember the EVALUATION. ized that model as advo- ing even that first step, let alone for- term advocacy research? cacy research, but that is mulating and validating hypotheses); Well, in the early - per- not entirely fair. Still, it there was just as little use of the engi- haps even primitive - model of com- wasn’t until the early 1990s that one neering-research method (almost no puting research employed up to the software researcher published materi- one was “observing existing solu- end of the twentieth century, much al on possible research models,6 iden- tions”); and there was a fringe group research consisted of a model that tifymg these: using the empirical method, but judg- some characterized as “conceive an + The scientific method. Observe the ing by the academic-tenure success of idea, analyze the idea, advocate the idea.” world, propose a model or theory of many of its proponents, their research At the conclusion of many research behavior, measure and analyze, vali- was not admired by their more tradi- papers was a discussion of the implica- date hypotheses of the model or theo- tional colleagues. tions for practice, which usually con- ry, and if possible repeat. Furthermore, almost no computing cluded with the claim that the idea + The engineering method. Observe research to that time had an evaluative should be transferred to practice as existing solutions, propose better solu- phase. Information may have been quickly as possible. Or words to that tions, build or develop, measure and gathered, propositions may have been effect. Colin Potts referred to this as analyze, repeat until no further im- made, and analysis may have been con- the “research-then-eansfer” approach, provements are possible. ducted. But, typically, the research contrasting it with what he called the + The empirical method. Propose a ended there. “industry-as-laboratory” approach.5 model, develop statistical or other In other words, the only research What made advocacy research arro- methods, apply to case studies, mea- model commonly in use in that unen-

44 NOVEMBER 1994 lightened era was the analytical tained any hypothesis testing, and less analytical in nature. method, the one we have characterized than 20 percent contained quantitative But why did computing research unfavorably as advocacy research. evaluation. The paper concluded, get stuck there? Here, the explanation Most research involved proposing for- “Computer scientists may produce too is somewhat more complicated. There mal methods for building software (the many designs and not enough quanti- are several reasons: propositional phase), fairly deep and tative results,” and, “The + Mathematicians at often mathematical analysis of those balance between theory many academic institu- methods (the analyucal phase), deriva- and experiment in Com- THE RESEARCH tions long ago divided tion of a theory of the applicability of puter Science seems themselves into two those methods (more analytical phase), skewed.” It was a combi- CRISIS BEGAN warring camps: those and no, repeat, no “compare with nation of the govern- TO ABATE WHEN who did applied re- empirical observations” (the evaluative ment research commit- search, studying ways phase). Probably the first paper to tees emphasizing reality- GOVERNMENT of using mathematics in publicly notice this deficiency was focused research and COMMITTEES other fields, and those Norman Fenton’s. 8 articles like Tichy’s posi- who did pure research, Perhaps the analytical method tion paper, which showed BEGAN TO studying mathematics might have been somewhat more how prevalent the prob- for the sake of its own acceptable had it employed the last lem was, that finally EMPHASIZE A improvement. Both dis- part of its definition (its evaluative caused the research com- REAL-WORLD ciplines are vitally need- phase, “if possible compare with munity to address its ed, of course, but as empirical observations”), but it rarely Droblems. This. in turn. FOCUS. often happens when did. It was as if there were a disdain for helped the research cri- humans and politics get anything connected with practice. sis begin to abate and, eventually, go into the act, these two groups began to Establishing pilot studies to try out away. dislike and, eventually, to disdain one ideas in a realistic setting and evaluate another. Johnny-come-lately comput- their success, incredibly enough, was ing people, born into this conflict, nat- not done. Researchers seemed to SOFTWARE RESEARCH‘S urally chose up sides. Most of them believe that was a task that practice MATHEMATICS CONNECTION favored the “pure” side, perhaps should do, much like the old “exercise because that was the side that often left for the student.” One leading com- Why did computing and software had more academic “respectability” puting-research journal of the early research start - and continue - to and political power. Whereas applied 1990s devoted a special issue to use such a narrow approach? With mathematicians and computing researchers giving practitioners advice respect to how it got started, the an- researchers might have broadened on how to evaluate new research ideas, swer is easy: their research approach, pure mathe- as if to say, “That’s your job, not Computing and software at most maticians could not and did not need ours.” No one seemed to notice the academic institutions were spawned to, and “pure” computer scientists sim- irony. in a mathematics department. Mathe- ply emulated that outlook. It was the publication of Walter matics is a peculiar discipline. There + The artifacts of computing and Tichy’s position statement on experi- are no physical artifacts for mathe- software are expensive and time-con- mental software engineering research maticians to study, as there are in suming to build (much more so than, that finally provided the solid data to the more scientific disciplines, for for example, new mathematical con- demonstrate how widespread the example, and thus mathematical re- cepts). Software research that involved research crisis was.9 In that article, search takes on an entirely different concept evaluation in a somewhat Tichy studied several of the leading flavor from other academic research. realistic setting would be very expen- computing and software-research jour- Of the research models mentioned sive in terms of both time and money. nals of the time, characterizing the above, the only one that makes much The money to do so was typically papers contained therein as to how sense is the last one. Mathematics, in unavailable in research settings, and much they involved other words, is rarely into scientific most researchers were not motivated + theory, or engineering or empirical research. to try to solve this problem. + design, Analytical research is the only ap- There is an irony to that reason, of + quantitative evaluation, and proach that it tends to employ. It is no course. At the same time that software + hypothesis testing. accident that the origins of computing practitioners were taking the position In what the author called “alarm- research were the same as the mathe- that it was too expensive to try out new ing” findings, few if any papers con- matical research that gave it birth: ideas from research, researchers were

IEEE SOFTWARE 45 taking the position that it was too of what the research field needed to often exchanged. Perhaps best of all, expensive to test the value of their new achieve. That model was a consortium practitioners and researchers tend to ideas with respect to practice. Software involving academe (the University of even like and respect each other! progress was in fact stuck in place by a Maryland Computer Science Depart- funding problem that almost no one ment); industry (Computer Sciences Good research results make it into prac- even recognized! Corporation); and government (NASA- tice. There is, of course, the lingering + During the early days of comput- Goddard, the sponsoring body), which mistrust of theory that built up during ing and software theory and practice, together had formed the Software the era of the “software crisis,” but the field moved forward so rapidly that Engineering Laboratory. The SEL most of those effects can be overcome almost all new ideas were good ideas. does engineering and empirical re- when a researcher is working along- For the field to wait until these new search using all the research phases, side a practitioner, being open to ad- ideas were evaluated would have dra- including - especially - evaluation. justing and improving ideas in matically slowed down that early Researchers the world over began order to make them useful in practice. progress in the field. Progress began to to study that model, and emulate it in Both researchers and practitioners slow in the 1970s, and continued to do other settings and for other application have come to understand the time and so in the 1980s, despite a growth spurt domains (the SEL was focused on money cost of the learning curve, real- triggered by the advent of the ubiqui- flight dynamics problems). It took izing that the adoption of any new tous microcomputer. Yet once years, of course, but that is how the idea has an initial price to be paid progress slowed, no one stepped back research crisis disappeared. before any payoff is achieved. Mod- to reanalyze the field’s approaches What happens in computing and ifying or getting rid of the schedule- to it to see that a new model was now software research and practice in the driven approach to building soft- needed. Long past the year 2020? It took a long ware has freed practitioners to try new time that it made sense, time to get here, but we ideas, to undertake the risks necessary researchers continued to NOW, IN THE have been able to achieve to making progress. The era of boast- expect practice to em- three things: ful claims of “breakthroughs” has brace their new ideas long since disappeared, replaced by with open arms. Software practice and re- a healthy understanding of the real- search work together. Re- ities of technology transfer. Slowly but THE RESEARCH searchers have given up surely, ways of building software are 2020 VISION AND PRACTICE on the “arrogant and nar- improving. We still have a lot to learn, row” approaches to soft- of course, but at least we understand But that was then and COMMUNITIES ware research, and have what that really means now. this is now. As I men- I come to realize that the tioned at the outset, we only way to tell if their Bad research ideas get discarded fairly in computing and soft- IEACHEVEN OTHER.RESPECT new ideas have value is quickly. Now that research ideas must ware research are cer- to try them out in a meet the test of practical usage, we no tainly glad we’ve been practical setting. There longer have the situation that pre- able to put the research crisis behind is a real “development” focus as well vailed in the twentieth century when a us. as a research one in most “research- new research idea would be conceiv- How did the change happen? The and-development” organizations and ed, analyzed by researcher after re- first necessary phase, of course, was projects, both in academe and in searcher, advocated thoroughly, and acknowledging that there was a prob- industry. The “industry-as-labora- never used in practice. The early feed- lem. Some of the papers mentioned tory” research concept advocated so back that researchers now get as to the earlier in our story were instrumental long ago5 has finally come to fruition! value of their new ideas is invalu- in making that happen; the gradual In fact, researchers and practition- able. Research, as a result, tends to accumulation of enough researchers ers tend to move back and forth freely move forward to embrace and study expressing the same view began to between their previously isolated turfs. new ideas, not get stuck regurgitating swing the field toward less arrogant Top practitioners move to the world old ones. For example, nearly 50 years and narrow, more realistic approaches. of academe and help ensure that ago, the research topic of formal veri- Progress was slow, of course, as it research and pedagogy there reflect fication (also called proof of correct- always is when fundamental human reality. Top researchers move to the ness) was conceived. Its advocates per- viewpoints must be changed. It helped world of industry to try out their ideas sisted for over 30 years in pushing that that there had been in existence for in a setting that can be truly evaluative. particular wheelbarrow uphill, paying nearly two decades an excellent model Sabbaticals and leaves of absence are no attention either to the disinterest

46 NOVEMBER 1994 Free report from Peter Coad reveals amazing industry breakthrough! “Object modeling and C++ programming, side-by- side, always up-to-date.” Big CASE tool vendors caught with their pants down! hat if you could have your OONOOD and disclaimers of practitioners who today, of course, but at least we now model and all of your C++ code continu- saw more cost than benefit in its use have a mechanism in place for shed- wously up-to-date, all the time, throughout your or to the warnings of fellow research- ding bad ideas. development effort? ers, published every five years or so in Consider the possibilities the literature, and angrily refuted each In one window, you see an object model, with automatic, semi-automatic, and manual time they appeared! (For example, t is often true that we humans make layout modes plus corn lete view manage- Fenton proclaimed “There is no hard Imore progress out of our failures ment. Side-bi-side, in tEe other window, you see fully-parsed C++ code. You edit one evidence to show that: than our successes. Perhaps, in spite of window or the other. Press a key. Both + formal methods have been used the pain of the research crisis, in the windows agree with each other. Together. cost-effectively on a realistic, safety- long term something good has come Su pose that you are working on a project wit% some existing code. (That’sno surprise, critical system development, of it. Certainly we are glad now that who’d consider developin in C++ without + the use of formal methods can we understand the value of practice some off-the-shelf classe8) You read the code deliver reliability more cost-effectively and theory moving forward hand in in. Hit a button. And seconds later, you see an object model, automatically laid out and than, say, traditional structured meth- hand. Both researchers and practition- ready for you to study side-by-side with the ods with enhanced testing, ers, working together, can see a future C++ code itself. Together. + either developers or users can in which the wisdom of each group is Or suppose you are building software with other people (that’sno sur rise either). You ever be trained in sufficient numbers understood and appreciated by the collaborate with others an$ develo software to make proper use of formal methods)? 10 other. And that, in human terms, may with a lot less hassle, because the Fully integrated configuration management feature It is still possible to pursue a re- be the biggest success of all, in the year helps you keep it all ...Together. 2020. search idea past its point of value even The name of this product? It’searned the name.. . Together/C++ REFERENCES: conhnuowly uptoclate Object modeling and C++ programming 1. B.I. Blum, “Some Very Famous Staustics,” The &&are Practztzoner, March 1991. 2. M. van Genuchten, “Why is Software Late! An Empirical Study of Reasons for Delay in Software Development,” ZEEE Trans. Software Eng., June 1991, pp. 582-590. Key features: 3. W. F. Tichy, N. Haberman, and L. Prechelt, “Future Directions in Software Engineering, Continuously up-to-date object modeling & C+t Summary of the 1992 Dagstuhl Workshop,” &@are Eng. Notes, Jan. 1993. programming Automatic semi-automatic, and manual layout of 4. E.V. LaBudde, “Why is Requirements Engineering Underused?” ZEEE Sojiware, Mar. 1994, pp.6-9; object models response by P. Hsia, A. Davis, and D. Kung. Object modeling view management, including view 5. C. Potts, “Software Engineering Research Revisited,” ZEEE Sojware, Sept. 1993, pp. 19-28. control by C++ construct, regular expression, II 6. W. R. Adrion, “Research Methodology in Software Engineering, Summary of the Dagstuhl proximity, layer, or directory Fully flexible documentation generation, version Workshop on Future Directions in Software Engineering,” Sojwure Eng. Notes, Jan. 1993. control, and SQL generation 7. R. L. Glass, “A Structure-Based Critique of Contemporary Computing Research,”?. Systems and Sojware, Jan. 1995, to appear. “State-of-the-art application development.” 8. N. Fenton, “How Effective Are Software Engineering Methods?”J. Systmsand Sojiware, Aug. 1993. -- ComputerworldlGermany 9. W. F. Tichy, et al., “Experimental Evaluation in Computer Science: A Quantitative Study,” (draft), “You’ve really hit the nail on the head when it 7. Systm and So&are, Jan. 1995, to appear. comes to reverse engineering existing C++ 10. N. Fenton, S. L. Pfleeger, and R.L. Glass, “Science and Substance: A Challenge to the code, No other tool comes close to the power and capability of Together/C++.” Software Engineering Community,” IEEE Sojware, July 1994, pp. 86-95. -- Russell Rudduck, Perot Systems Money-back gurantee. Purchase Together/ C++ and try it out risk-free for 30 days. If for an reason you aren’t satisfied, return it for a Robert L. Glass is publisher of The Software Practztzoner, editor of theJournal of fulT refund. (No hassles, no hard feelings Systems and Software, a regular columnist for Managing System Development, and a either. )We’rethat confident about Together/ C++. You see To ether/C++ has already someume mSihng professor of software engneering at Linkoping University. He helped software fevelopers deliver better IS interested in all facets of software engneering, especially in quality and mainte- systems, with success stories in tele- nance He has written 17 books and more than 30 papers on compuung and soft- communications, insurance and natural ware. resource managment. Glass received an MSc in mathemaucs from the University of Wisconsin at How to order. Order Together/C++ by Madison. He is a member of the IEEE, the IEEE Computer Society, and ACM purchase order, check, or credit card, or for more information, please contact: Obiect International. Inc Outside of Sorth Amenca. contact

[email protected] e-mad object4acm org c 1991 Oh KI &I I Inc Together. ,,“,‘i:~~~~~seag~~r,Int‘l. Inc Address questions about this article to Glass at Computing Trends, 350 Dalkeith Ave., Los Angeles, CA 11 90049. lEEEl194 I’

IEEE SOFTWARE