The Research Crisis ROBERTL. GLASS,Computing Trends With the advantage of computing and software research are ’ using only one of them. Even though more than 25 years’ hindsight, certainly glad we’ve been able to put we now understand how terribly limit- this twenty-first century author the “research crisis” behind us. ’ ing that approach was, it is easy to see, What was this crisis? It was the in retrospect, why computer scientists looks askance at the “misis”in realization that occurred, right around , could not see the error in their ways. the turn of the century, that research Before I continue, I think it is software practice and expresses in computing and software - as it was worth spending a moment or two deep concern fir a Cris in then focused - was all too often both elaborating on the term research crisis. arrogant and narrow. ’ There is a fine irony to it, and the software research. It was arrogant because many corn- story makes good telling. puting researchers of that era were ~ To understand this research crisis, doing research in a topic they thought it is important that we confront and they understood, but didn’t. It is deal with a prior “software crisis.” amazing that, in retrospect, those ’ It has been about 50 years now computer scientists simply didn’t since the software community coined know what they didn’t know. , the term sojhuare crisis. It was invented It was narrow because, of all the at a conference predominantly attend- possible research models twentieth , ed by theorists rather than practition- century computer scientists might ers, but still, at the outset, almost 42 07407459/94/$04 W 0 1994 IEEE NOVEMBER 1994 everyone agreed there was some validi- ing Office on the problems of building resentment by software practitioners ty to the term. What did “software cri- software, as support for their claims of of software theorists and software the- sis” mean? It is hard to believe, now, xisis. ory. Most people in a profession would but what it meant way back then was In most cases, this GAO study was be offended, of course, by the notion that the practice of software was in cri- the only real, nonanecdotal data cited, that their projects and products were sis - that it was characterized by pro- and almost every researcher cited it. It constantly characterized as “behind jects that were “always over budget, seemed to say that most software pro- schedule, over budget, and unreliable.” behind schedule, and unreliable.” jects failed, and that most money spent Ironically, computing researchers With the advantage of hindsight, on software was wasted. never understood how offensive their we can look back on the last century as Fortunately, cooler heads began to software-crisis campaign had been to a time of wonder. Of all the dramatic prevail. One researcher, then a voice practitioners. It is fair to say, in the changes that took place in the world, crying in the wilderness, brought an enlightened year 2020, that those old computing and software gave rise to end to the use of the resentments still linger, and Dartlv because of the name “computing era” that we term software crisis. II now recognize as the proper character- Bruce Blum, arguably CRISIS, INDEED! them, new theory still ization of the last half of the twentieth one of the few research- has a difficult time pene- century. Oh, there were software pro- ers of the time who actu- THE TWENTIETH trating current practice. ject failures, of course. Some of them ally read the study (most But back to the 1990s. were even catastrophic - runaway researchers apparently New government studies described the worst of them. But their copied the data from A TIME OF and open-minded com- existence provided only anecdotal evi- some other researcher’s puting researchers both dence, certainly not solid data, to sup- paper without going IGNoRANCEfbegan pointing out that port the claims of crisis. In fact, the back to the source), dis- BUT ALSO THE serious problems existed issue of solid data became the shoal on covered the data was in the 1990s model of which the claims of crisis foundered, as being misunderstood DAWN OF THE software research. In the we will see later in this story. and misused.’ The study, GOLDEN AGE early 1990s, for example, Software crisis indeed! We may an analysis of govern- a couple of government look back on that earlier time as archa- ment software projects, OF PRACTICE* studies concluded that ic and ignorant, perhaps, but certainly examined only -projects research was ignoring not as a crisis. It was the beginning, in that were in trouble practice almost entirely fact, of the Golden Age of Computing when it was conducted. Given that, the and, in addition, ignoring the notion Practice that persists today. study’s conclusion that most such pro- of practical application. There was a Nevertheless, the notion of soft- jects failed, and most money spent on flurry of opposition to those reports, ware crisis reached a fever pitch of them was wasted, was interesting - as researchers mired in the old ways intensity in the 1980s and 1990s. “troubled projects frequently fail” - resisted change (the very thing they Researchers began nearly every paper but hardly a basis for blackening the accused practitioners of!), but eventu- on software engineering by invoking reputation of all software practice. ally most computing researchers 1 the software crisis as a reason for lis- Even after the Blum revelation, a began to realize that theirs was truly a tening to whatever new theoretical few computing researchers continued, troubled endeavor. A flurry of criti- notion they were advocating. Re- for their own self-serving reasons, to cisms stirred the pot: Software-engi- ~ searchers at the time didn’t realize that use the GAO data and cry “software neering research was described as what many of them were doing would crisis,” knowing that the data was increasing in quantity but not in quali- later be characterized, derisively, as being misused. But eventually common ty, laclung in evaluation, “becoming advocacy research. But the software crisis sense and ethics prevailed, and the less credible,” having a “gaping hole” was in fact the platform on which most notion of a software crisis slowly died in its “generally accepted methods,” of this advocacy research was founded. away. Furthering its demise was and in need of a “paradigm shift ... Michiel van Genuchten’s data, which from purely theoretical and building- presented findings from other re- oriented to experimental ....” 3 SOFTWARE CRISIS DISCREDITED searchers to the effect that typical soft- Later, in an exchange of letters to ware overruns were 33-36 percent over the editor, both a letter-writer and the At the time, researchers frequently budget and 22 percent behind schedule, authors of the article addressed by the cited both the anecdotal evidence clearly a problem but hardly a crisis. 2 letter-writer agreed that, “software referred to earlier and a government One lingering aftereffect of this so- research is in a sad state,” and that, study from the US General Account- called crisis, however, was a deep “Without a scientific method, tech- IEEE SOFTWARE 43 nobabble, lemmingineering, and fads gant was that the typical researcher sure and analyze, validate the model, run rampant.”4 had never worked in software practice repeat. At the time, it seemed natural to and had no basis for assuming that his + The analytical method. Propose a shift the crisis from practice to re- or her idea would really work there. formal theory or set of axioms, devel- search. This was not done in a spirit of Most researchers had a mental model op a theory, derive results, and if pos- meanness, but rather in the original of software practice as an enterprise in sible compare with empirical observa- fundraising spirit of the term. Re- crisis, one that did a bad job of what- tions. searchers proclaimed a “research cri- ever it undertook. There seemed to be Then, a few years later, another sis” to obtain funds for newer and bet- an underlying assumption in most paper also addressed the issue,’ break- ter ways of doing research, just as their research that any change was better ing research down into four phases: forerunners had used the practitioner than the status quo. + The informationalphase. Gather or “software crisis” as a way of obtaining The problem got so bad, in fact, aggregate information via reflection, funding to investigate the problems that the software consortia and institu- literature survey, people/organization- that crisis implied. For quite a long tions of the time were sometimes char- a1 survey, or poll. time, cries of “research crisis” were as acterized as “arrogant and ignorant.” + The propositional phase. Propose prevalent in the computing literature This charge was unfair to many of and/or build a hypothesis, method or as mentions of its predecessor had them, of course, but there was enough algorithm, model, theory, or solution. been. As I mentioned at the beginning truth to it that the notion stuck. It is + The analytical phase. Analyze and of this essay, that notion, like the earli- difficult for a research-and-develop- explore a proposal, leading to a de- er crisis that preceded it, has died out. ment institution to be effective when monstration and/or the formulation of But it was a long time in D its --people are seen as ar- a principle or theory. the dying. rogant and ignorant. + The evaluative phase. Evaluate a THE ONLY That problem nearly des- proposal or analpc finding by means troyed the effectiveness of experimentation (controlled) or FOCUS ON RESEARCH RESEARCH of those institutions be- observation (uncontrolled, such as a MODEL IN fore they managed to over- case study or protocol analysis), per- Back to those twin come it.