wdv-notes Stand: 29.DEZ.1994 (2.) 329 – Business Must Learn from the Debacle. Wiss.Datenverarbeitung © 1994–1995 Edited by Karl-Heinz Dittberner FREIE UNIVERSITÄT BERLIN Theo Die kanadische Monatszeitschrift The Im folgenden sowie in den wdv-notes verbreitet werden, wenn dabei die folgen- Post, Winnipeg veröffentlicht Nr. 330 werden diese Artikel im Original den Spielregeln beachtet werden. in mehreren Artikeln in ihrer Januar-Aus- nachgedruckt. Der Dank dafür geht an Permission is hereby granted to copy gabe 1995 eine exzellente erste Zusam- Sylvia Douglas von The Computer Post, this article electronically or in any other menfassung des Debakels um den Defekt 301 – 68 Higgins Avenue, Winnipeg, Ma- form, provided it is reproduced without des Pentium-Mikroprozessors [1–2] des nitoba, Canada, Email: SDouglas@post. alteration, and you credit it to The Compu- Computergiganten Intel. mb.ca. Diese Artikel dürfen auch weiter- ter Post.

Intel’s top of the line Pentium™ microproc- This particular error in the Pentium was in essor chip has turned out to have a slight flaw the floating-point divide unit. Intel manage- Editorial: The Computer Post – Jan.95 in its character: when dividing certain rare ment was concerned enough about it that pairs of floating point numbers, it gives the they pulled together a special team to assess The Way it Will Be wrong answer. For anyone who owns a Pen- the implications. tium-based computer, or was thinking about This month you’ll find we’re reporting a In the words of Andrew Grove, Intel’s CEO, buying one, or is just feeling curious, here lot of background information on Intel’s posting later to the [3], “We were are... Pentium problems. In fact, we have never puzzled as to why neither we nor anyone else before devoted this much space to any sin- The Facts about Pentium had encountered this earlier. We started a gle news story. separate project, including mathematicians We’re not just picking on Intel because By SYLVIA DOUGLAS and scientists who work for us in areas other they’re big – although there is a certain The Computer Post than the Pentium processor group to exam- fascination in watching such a large corpo- ine the nature of the problem and its impact.” ration being forced by public pressure to Like an avalanche, it started small. One Inexplicably, though, the now-infamous reverse a strongly held position. What’s little whisper that something was wrong. Sud- FDIV bug didn’t make its way onto any lists of new and different about this drama, what denly, without any more warning than that, known bugs. Intel didn’t inform even their deserves serious study by everyone running everything was sliding out of control. Giant largest customers. a business in the second half of this decade, Intel, the company that makes chips for 80% is the role played by online communica- of the world’s PCs, found itself in the middle Instead, the special team did their analysis. tions. of the worst PR debacle in its corporate In Grove’s words, “This group concluded Let’s not exaggerate. The Internet didn’t history. after months of work that (1) an error is only do it all. Intel didn’t give in until the Pen- likely to occur at a frequency of the order of tium story became widespread enough to Five years from now, one way or another, once in nine billion random floating point this will be an excellent case study for busi- affect stock prices and threaten future sales. divides, and that (2) this many divides in all This still means CNN and USA Today, not a ness schools in their crisis management cour- the programs they evaluated (which included ses – as an example of how not to do it. It was newsgroup and a few Web sites – not yet. many scientific programs) would require At the rate people are connecting to the a minor technical problem, Intel management elapsed times of use that would be longer thought, just like many others they have dealt Internet now, within a few years the public than the mean time to failure of the physical at large is going to be looking online first for with over the years. Last June, just when Intel computer subsystems.” was in the midst of ramping up its Pentium their news and information. More than that, production to meet the challenge of the online will be where they can ask their own PowerPC chip (coming from Apple, IBM, and While those months passed by, many more questions, express their own opinions, and Motorola), some of its engineers noticed a scientists and engineers, persuaded by In- discover what other people think. So it is wrong answer in one of the multitude of tests tel’s advertising that the Pentium was now vital for business to understand the new they were running on the Pentium. able to give workstation performance at af- dynamics of an interactive customer base. fordable prices, adopted this platform for criti- Here is a case study of how it happened to cal calculations. The price of started one company. »You know postmodern – to come down, and businesses adopted the Intel ran into the phenomenon early be- platform for calculation-intensive spreadsheet cause of who they are, and who their cus- Well, this is PostReality.« work. After all, one of the advertised advan- tomers are. They sold the Pentium to scien- The Computer Post, Winnipeg. tages of the Pentium was its very fast floating tists and engineers who had been connected point arithmetic. to the Internet for some time and were comfortable with expressing themselves that All chips have errors. This is a fact of life in In what now turns out to be Intel’s one bit of way. the semiconductor business. Connecting mill- good news, the platform was also adopted by Already, a controversy on the Internet is ions of transistors is not something that can large numbers of consumers looking for a very, very public. Many business people are be done error-free the first time. The normal fast, capable multimedia computer to run all now connected, and a rapidly growing practice is to collect up a number of these the exciting new games and educational soft- number of private individuals. There were errors and their design fixes, and apply them ware coming to market. more than ten thousand messages posted to to a revised version of the wafer mask which The report produced by the Intel team has comp.sys.intel, and there is no way of deter- then becomes what the production lines are now been made public. We have included mining how many times that number of churning out. excerpts from this “White Paper” on page 12 people scanned the newsgroup for informa- tion. In software this is called a slipstream up- [Editors Note: See [4] ]; Intel will send a copy to anyone who asks for it. Journalists were certainly among them. grade; most software companies do it. In This past year, only the leading-edge re- hardware, it’s called a stepping. Chips can go Curiously enough, though, it was not Intel porters were using the Internet as an infor- through 5, 10 or even 20 steppings in the that made the first public announcement of mation-gathering tool; two years from now, course of their lifetime. Normal practice is to the bug. Dr. Thomas R. Nicely of Lynchburg they will all be doing it. It is a matter of keep at least major chip customers updated College, a professor of mathematics, was moments to follow up a Web page reference with a current list of outstanding bugs in each doing some work in number theory that in- in a newsgroup article and find a detailed chip. volved taking a lot of reciprocals. To double- source document, or to post a question and check his work, he did it twice – once in To be continued … © 1995 – The Computer Post, Winnipeg floating point, and once in integer arithmetic. Die 1. Edition erschien bereits am 23. Dezember 1994. Seite 1 wdv-notes 329 Back in June, Nicely noticed a small dis- had developed a software workaround that On December 20, Intel caved in to the crepancy between his two sets of results. Of could at least be compiled into programs pressure. They announced that they would course, it did not occur to him at first that this where source code is available [8]. They take an unspecified reserve against their was a problem with the Pentium chip. He didn’t meet each other in person – all the work fourth-quarter earnings, and “exchange the spent a lot of time eliminating other causes, was done via the Internet, and all results were current version of the Pentium processor for such as compiler artifacts and errors in the posted publicly for the world to see, criticize, an updated version, for any owner who re- PCI bus. and contribute to. quests it, free of charge anytime during the life of their computer.” On October 22, after running his test case Meanwhile, the implications of the FDIV on a number of 486 and Pentium , bug began to sink in. Vaughan Pratt of Stanford Most of the trouble could have been avoided he became certain the fault was in the Pen- University’s computer science department if, in early summer, Intel had not tried to tium CPU. That day, he contacted Intel tech started examining the Coe model, and did conceal the defect. They might have lost support. After eight days, they had no an- some analysis of Intel’s probability analysis. some workstation-market sales, but the swers for him, so on October 30, he sent He wrote, “Internet postings [are] the best workarounds would have been developed email messages to a number of people and medium for the sort of turnaround this time- and incorporated into compiler libraries. Even critical subject needs, with its additional ad- as late as October or November, if Intel had Question: What’s the defini- vantage of feedback about mistakes in my immediately given an offer of unconditional tion of the so-called “tertiary analysis – in effect real-time refereeing.” replacement – a position they have been backed into now, anyway – they could have sciences”? On November 28, Intel posted the state- come across as responsible, and concerned ment from Andy Grove quoted above into the Answer: These are sciences about the people who use their computers. using Intel computer plat- comp.sys.intel forum [3]. Having made the one statement, which confirmed that they Intel is far too large and cash-rich to go forms – especially Pentium had known about the bug since early sum- under because of one botched job of public PC’s – only, mostly running mer, they then lapsed into silence. They were relations. They will not even lose money in under Microsoft’s DOS & not prepared to be told, over and over again the coming quarter. But the lesson is going to Windows. They never use in hundreds of replies, that they were badly be expensive. Lets hope they learn the right more reliable platforms or missing the point. one. operating systems. A typical furious quote: “Having conclu- Read on UseNet [9]. sively demonstrated themselves utterly un- Sonstige Hinweise worthy of the public’s trust, they still seem Die weltweite öffentliche Diskussion der organizations he felt would have access to unable to comprehend what that means.” Folgen des Pentium-Defekts wird im Internet many Pentium systems, and asked them to The anger was fuelled by the growing ex- vorwiegend in der Newsgruppe comp.sys.intel check for the problem. pert consensus that the problem was worse geführt. Ohne die Existenz des wäre The speed with which events developed than Intel was claiming it to be. Pratt of der Pentium-Fehler für die Öffentlichkeit mit from that moment is a graphic illustration of Stanford wrote: “This bug is maximally insidi- Sicherheit niemals aufgeklärt worden. the nature of today’s, and tomorrow’s, inter- ous: it is about as large as it could possibly be connected world. Things are going to work without actually triggering warning bells when Literatur like this from now on. Other businesses of people review their columns of data. In this Alle im folgenden aufgezählten Materialien every variety, take note. way tiny errors of one part in a hundred sind auch elektronisch publiziert und stehen als thousand can over a long period of time Dateien (Files) auf dem Ftp-Server ftp.grumed.fu- Within three weeks, high-powered engi- sneak into the trillions of calculations done berlin.de im Verzeichnis PC zum Kopieren via neers like Tim Coe of Vitesse Semiconductor around the world and there is no practical way Internet (Anonymous Ftp) zur allgemeinen Ver- [5] and Mike Carleton of USC/ISI [6] had to spot them short of doing a massively de- fügung. Auf diesem Ftp-Server sind weitere reverse-engineered Intel’s floating point divi- Infodateien zum Pentium-Defekt archiviert. tailed audit of a kind that would have to be sion algorithm, and developed a mathemati- tailored to the Pentium FDIV bug and would cal model which successfully predicted [1] Dittberner, K.-H. (Ed.): „Intel Pentium – The be entirely unnecessary for a reliable floating Chip that Redefines Mathematics“. FU number pairs for which the FDIV instruction point unit.” Berlin (IfP): wdv-notes Nr. 307, 1994–1995. would fail. Within six weeks, Coe, together [2] Dittberner, K.-H.: Intel Pentium – Der ein- with Cleve Moler of Mathworks, Inc., Terje Meanwhile, on December 12 Intel lost any gebaute Divisionsfehler. FU Berlin (IfP): chance that this issue might slip by with little Mathisen of Norsk Hydro in Norway [7], and wdv-notes Nr. 327, 1994–1995. Peter Tang of Argonne National Laboratories mainstream attention, when IBM, angered by Intel’s failure to inform them, and possibly [3] Grove, Andy (Präsident der Intel Corp.): My Perspective on Pentium. – Publiziert in der Editorial: The Computer Post – Jan.95 continued: wanting to dissociate themselves from what UseNet-Gruppe comp.sys.intel, 27. Novem- looked like disaster in the making, announced ber 1994 – 19.31 GMT. – File: INTELS_ ask anyone in the world who’s had a specific that they were stopping shipment of all Pen- PRESIDENT.TXT. kind of experience to make contact by email. tium-based systems [10]. [4] Sharangpanl, H. P. and Barton, M. L.: Sta- Monthly periodicals like The Computer tistical Analysis of Floating Point Flaw in Post can’t expect to be first with a fast- Since then there have been major focus articles on the subject in the Wall Street the Pentium Processor (1994). Santa Clara breaking news story – that’s the job of (USA): Intel Corp. White Paper, 30. No- radio, TV, and the newswire services. (It Journal, Business Week, and the New York vember 1994. – File: INTELS_WHITE_ was only by luck that Intel’s about-face Times. The Gartner Group has recommended PAPER.PS.ZIP. announcement came in time for us to in- that major corporate clients hold off on pur- [5] Coe, Tim: A C-model of Intel’s Pentium clude it.) What we can do is provide some chases of Pentiums. The Pharmaceutical hardware divider that predicts many of the perspective, and enough information to let Association warned members not to use Pen- failing divides. – File: PBUG_MODEL.TXT. our readers understand what a news story tiums in calculations to be sent to the Food [6] Carlton, Mike (Ed.): The Pentium Divide means, and how it indicates trends. and Drug Administration. Lawsuits were Bug FAQ. – File: PBUG_FAQ.TXT. In our front page story, I wrote that five launched. [7] Mathisen, Terje: Test program for the Pen- years from now, business schools will be And, the ultimate humiliation: on the sports tium chip FDIV bug. – File: P87TEST.ZIP studying the Intel case. Business executives page of the New York Times, in a story about (komprimierte Datei). can’t afford to wait five years to grasp the the New Jersey Nets basketball team and [8] Dittberner, K.-H. (Ed.): Suggestions for Pen- implications of the coming social change. how their record does not match their ability, tium Bug Software Workarounds. – File: PBUG_WORKAROUND.TXT. In 1994 it was only a computer chip. Over there was the headline, “Mentally Speaking, the next few years, every business will have Nets are Pentiums.” [9] The UseNet Community: A Intel/Pentium to adapt to the fact that customers have Jokes List. – File: PBUG_JOKES.TXT. acquired a whole new dimension of power. Intel’s expensively promoted premium [10] IBM’s statement on their Pentium study. Sylvia Douglas brand name, being used as an insult on the Published: 12. December 1994. – File: sports page! PBUG_IBM_STUDY.TXT. Seite 2