American Scientist the Magazine of Sigma Xi, the Scientific Research Society

COMPUTING SCIENCE

Brian Hayes

A reprint from American Scientist the magazine of Sigma Xi, the Scientific Research Society

Volume 91, Number 6 November–December, 2003 pages 484–488

This reprint is provided for personal and noncommercial use. For any other use, please send a request to Permissions, American Scientist, P.O. Box 13975, Research Triangle Park, NC, 27709, U.S.A., or by electronic mail to [email protected]. © 2003 Brian Hayes. COMPUTING SCIENCE

A LUCID INTERVAL

Brian Hayes

ive a digital computer a problem in arith- glitches. Nevertheless, the error in computer Gmetic, and it will grind away methodical- multiplication is mildly disconcerting. We would ly, tirelessly, at gigahertz speed, until ulti- like to believe that the mathematical machines mately it produces the wrong answer. The cause that control so much of our lives could at least do of this sorry situation is not that software is full of elementary arithmetic correctly. bugs—although that is very likely true as well— One approach to dealing with such numerical nor is it that hardware is unreliable. The problem errors is a technique called interval arithmetic. It is simply that computers are discrete and finite does nothing directly to improve the accuracy of machines, and they cannot cope with some of the calculations, but for every number it provides a continuous and infinite aspects of mathematics. certificate of accuracy—or the lack of it. The result 1 Even an innocent-looking number like ⁄10 can of an ordinary (non-interval) computation is a sin- cause no end of trouble: In most cases, the com- gle number, a point on the real number line, puter cannot even read it in or print it out exactly, which lies at some unknown distance from the much less perform exact calculations with it. true answer. An interval computation yields a Errors caused by these limitations of digital ma- pair of numbers, an upper and a lower bound, chines are usually small and inconsequential, but which are guaranteed to enclose the exact answer. sometimes every bit counts. On February 25, 1991, Maybe you still don’t know the truth, but at least a Patriot missile battery assigned to protect a mil- you know how much you don’t know. itary installation at Dahrahn, Saudi Arabia, failed to intercept a Scud missile, and the malfunction Measuring Ignorance was blamed on an error in computer arithmetic. Suppose you are surveying a rectangular field. No The Patriot’s control system kept track of time by matter how carefully you read the measuring counting tenths of a second; to convert the count tape, you can never be certain of the exact dimen- 1 into full seconds, the computer multiplied by ⁄10. sions, but you can probably state with confidence Mathematically, the procedure is unassailable, but that the correct figures lie within certain bounds. computationally it was disastrous. Because the Perhaps you are sure the length is no less than 68 1 decimal fraction ⁄10 has no exact finite representa- meters and no more than 72, while the width is be- tion in binary notation, the computer had to ap- tween 49 and 51 meters. Then you can state with proximate. Apparently, the conversion constant equal confidence that the area of the field lies stored in the program was the 24-bit binary frac- somewhere between 3,332 square meters and tion 0.00011001100110011001100, which is too 3,672 square meters. This is interval arithmetic in small by a factor of about one ten-millionth. The action: [68, 72] × [49, 51] = [3,332, 3,672]. (The discrepancy sounds tiny, but over four days it bracketed pair [a, b] signifies the interval from a to built up to about a third of a second. In combina- b inclusive, with a ≤ b. Another useful notation is – tion with other peculiarities of the control soft- [–x,x], where the underscore indicates a lower lim- ware, the inaccuracy caused a miscalculation of it and the overscore an upper limit.) almost 700 meters in the predicted position of the Doing basic arithmetic with intervals is not incoming missile. Twenty-eight soldiers died. much more difficult than with ordinary (“point- Of course it is not to be taken for granted that like”) numbers. Indeed, a single formula extends better arithmetic would have saved those 28 the definition of the four standard arithmetic op- lives. (Many other Patriots failed for unrelated erations to intervals. If ° represents any of the reasons; some analysts doubt whether any Scuds operations +, –, × or ÷, then the corresponding were stopped by Patriots.) And surely the under- interval operation is defined as: lying problem was not the slight drift in the clock – – – – – – [–u, u] [–v, v] = [min(–u –v, –u v, u –v, u v), but a design vulnerable to such minor timing ° ° °– –° –°– max(–u°–v, –u°v, u°–v, u°v)]. Brian Hayes is Senior Writer for American Scientist. Address: In other words, compute the four possible com- 211 Dacian Avenue, Durham, NC 27701; [email protected] binations of the upper and lower bounds, then

484 American Scientist, Volume 91 – choose whichever of these results is the smallest real number line. Or the interval [–x, x] could be as the new lower bound, and likewise take the taken as denoting a new kind of number, in much largest as the new upper bound. Every possible the same way that two real numbers x and y com- combination u°v must lie within these limits. bine to specify the complex number x+iy (where i The min-max formula is a convenient definition represents the square root of –1). This last view is of interval operations, but it is not always the best the most ambitious. It suggests the goal of a com- implementation. For example, in the case of addi- puting system where intervals are just another tion it’s obvious that –u+–v will always be the small- data type, interchangeable with other kinds of est sum, and u–+v– the largest, so that interval ad- numbers. Wherever a pointlike number can ap- – – – – dition is simply [u–, u]+[v–, v] = [u–+v–, u+v]. By pear in a program, an interval can be substituted. – – similar reasoning, subtraction is just [–u,u]–[–v,v] = Conversely, exact pointlike numbers can be rep- – – [u––v, u–v–]. Multiplication is not quite as well- resented by “degenerate” intervals of zero width; behaved. Although shortcuts are sometimes pos- the number 2 could be written [2, 2]. sible (depending on the signs of the operands), in the worst case there is no choice but to compute all Perils of Precision four of the combinations and select the extrema. Why should we have to put up with errors and Division is much like multiplication, but with approximations in computing? Why can’t the a further annoyance—the possibility of a zero di- computer just give the right answer? visor. With pointlike numbers, if you try to carry Sometimes it can. Calculations done entirely out an operation such as 2 ÷0, the error is obvi- with integers yield exact results as long as the ous, and the system software will prevent you numbers are not too big for the space allotted. Of- from doing the impossible. An interval division ten the allotted space is quite scanty—as little as such as [2, 4] ÷[–1, 1] has the same problem, but 16 bits—but this is an artificial constraint; in prin- in disguise. You could very well perform the nec- ciple a computer can store the exact value of any essary calculations on the end points of the inter- integer that will fit in its memory. vals without raising any alarms and without Integers have the pleasant property that they noticing that the divisor interval includes the form a closed set under addition, subtraction and value zero. But the answer you would arrive at in multiplication; in other words, when you add, this way, [–4,4], is utterly wrong. It’s not just subtract or multiply any two integers, you always wrong in the formal sense that it might be tainted get another integer. Absent from this list of opera- by an illegal operation. It’s also wrong because tions is division, because the quotient of two inte- the interval [–4, 4] does not enclose all possible gers is not always an integer. If we allow numbers quotients, even if the zero point itself is excluded to be divided, we must go beyond the integers to 2 3 from the divisor. A reliable system for interval the rational numbers, such as ⁄3 or ⁄2. But rational arithmetic needs protection against this hazard. values can also be represented exactly in the com- Usually, division by an interval that includes zero puter; all that’s needed is to keep track of a pair of is simply forbidden, although there are also other integers, which are interpreted as the numerator 1 ways of coping with the problem. and the denominator. Thus the constant ⁄10, which Apart from the rules for manipulating intervals caused such havoc in the Patriot software, could arithmetically, there remains the question of what have been encoded in the two binary integers 1 an interval really is and how we ought to think and 1010. A few programming languages—no- about it. In the context of dealing with errors and tably Lisp and its offspring—provide integers of – uncertainties of computation, we may see [–x, x] unlimited size (“bignums”) and exact rationals as as standing for some definite but unknown value built-in data types. Similar facilities can be added ≤ ≤ – – x such that –x x x. But [–x, x] could also be inter- to other languages. preted as the set of all real numbers between –x If we can have exact numerical computation, and x–—in other words, as a closed segment of the why would anyone choose approximate arith-

√ 2 π integers

0 1 2 3 456 7

rationals

0 1 1 3 1 5 3 7 2 9 5 11 3 13 7 15 4 17 9 19 5 21 11 23 6 25 13 27 7 1 4 2 4 1 4 2 4 1 4 2 4 1 4 2 4 1 4 2 4 1 4 2 4 1 4 2 4 1

floats

0.0 0.50.751.0 1.25 1.5 1.75 2.0 2.5 3.0 3.5 4.0 5.0 6.0 7.0

Figure 1. Computer numbering systems include integers, rationals and floating-point numbers, but none of these schemes can represent all possible quantities. For irrational values such as π and the square root of 2, the best you can do is choose the nearest representable number. Intervals (shown here in yellow) bracket an unrepresentable number and thereby put bounds on the error of approximation. www.americanscientist.org 2003 November–December 485 1.0 decimal number 6.0 can be expressed as 0.75 × 23, 0.9 with significand 0.75 and exponent 3. In this case 0.8 1 – cos(x) the representation is exact, in binary as well as in x 2 decimal (the binary significand is 0.11). Other 0.7 numbers are not so lucky. As noted above, no fi- 0.6 nite significand corresponds exactly to the deci- 1 0.5 mal fraction ⁄10. Furthermore, it’s obvious that 0.4 some numbers must be missing from the system 0.3 simply because it has only a finite capacity. In one 0.2 common floating-point format, the total space available for storing the significand and the expo- 0.1 nent is 32 bits, and so the system cannot possibly 0.0 hold more than 232 distinct numbers, or about 4 0.009 0.008 0.007 0.006 0.005 0.004 0.003 0.002 0.001 billion of them. If you need a number that is not a Figure 2. Numerically unstable formulas can cause a catastrophic loss of member of this finite set, the best you can do is accuracy; although interval arithmetic cannot prevent such failures, it choose the nearest member as an approximation. can issue a warning. Here the blue lines represent the correct value of The difference between the true value and the ap- 1 – cos(x)/x2 for values of x near 0. The red dots are the results of a float- proximation is the roundoff error. ing-point calculation done with 20 bits of precision; for values of x be- Interval arithmetic cannot eliminate roundoff low about 0.004 these results become unreliable. An interval calculation error, but it can fence it in. When a result x falls (yellow) shows the progressive loss of certainty. The example was first between two floating-point values, those nearest discussed by Michael J. Schulte and Earl E. Swartzlander, Jr., of the representable numbers become the lower and up- University of Texas at Austin. – per bounds of the interval [–x,x]. But this is not the metic? One reason is that there are numbers be- end of the story. Subsequent computations could – yond the rationals: No ratio of finite integers yield a new interval for which –x and x are them- √– π gives the exact value of 2 or or log2(3). Per- selves numbers that have no exact floating-point haps more important, exact computations tend representation. In this situation, where even the to become hopelessly unwieldy. Consider the se- interval has to be approximated, rounding must 1 1 1 1 ries 1 + ⁄2 + ⁄4 + ⁄8 + ⁄16 + .... If you sum a thousand be done with care. To preserve the guarantee that terms, the result is vanishingly close to 2, but the the true value always lies within the interval, the exact rational representation fills two thousand end points of the interval must be rounded “out- – binary digits. Doing arithmetic with such obese ward”: –x is rounded down and x is rounded up. numbers is slow and cumbersome. And outside the realm of pure mathematics the cost of main- Historical Intervals taining exactness is seldom justified. Nothing in Interval arithmetic is not a new idea. Invented the physical world can be measured with such and reinvented several times, it has never quite precision anyway. made it into the mainstream of numerical com- The usual alternative to exact rational compu- puting, and yet it has never been abandoned or tations is floating-point arithmetic, a scheme that forgotten either. resembles scientific notation. A number takes the In 1931 Rosalind Cicely Young, a recent Cam- form D×β E, where D is called the significand, E is bridge Ph.D., published an “algebra of many- the exponent, and β is the base (which in modern valued quantities” that gives rules for calculating computer systems is always 2). For example, the with intervals and other sets of real numbers. Of course Young and others writing in that era did < < < < not see intervals as an aid to improving the relia- bility of machine computation. By 1951, however, < < < = in a textbook on linear algebra, Paul S. Dwyer of the University of Michigan was describing arith- < < < > < < = > < = > > < < = = = = > > metic with intervals (he called them “range numbers”) in a way that is clearly directed to the < = = > < < > > < > < > = = = = needs of computation with digital devices. A few years later, the essential ideas of interval < > > > < > = > < = < > = > = > < = < = arithmetic were set forth independently and almost simultaneously by three mathematicians— = > > > Mieczyslaw Warmus in Poland, Teruo Sunaga in Japan and Ramon E. Moore in the United States. > > > > Moore’s version has been the most influential, in part because he emphasized solutions to problems Figure 3. Comparisons between intervals are more complicated than of machine computation but also because he has those of pointlike numbers. There are 18 meaningful relations between continued for more than four decades to publish intervals (including cases where one or both intervals are in fact point- on interval methods and to promote their use. like). It’s unclear even how to name all these comparisons; one encod- Today the interval-methods community in- ing lists the relations of the four pairs of end points in a fixed sequence. cludes active research groups at a few dozen uni-

486 American Scientist, Volume 91 versities. A web site at the University of Texas at El Paso (www.cs.utep.edu/interval-comp) pro- 6 vides links to these groups as well as a useful 5 archive of historical documents. The journal Reli- able Computing (formerly Interval Computations) is 4 the main publication for the field; there are also mailing lists and annual conferences. Implemen- 3 tations of interval arithmetic are available both [–3,2] 2 as specialized programming languages and as li- [1,4] braries that can be linked to a program written in 1 a standard language. There are even interval spreadsheet programs and interval calculators. –6 –56–45–34–23–1201 One thing the interval community has been ar- dently seeking—so far without success—is sup- Figure 4. Diagrammatic scheme introduced by Zenon Kulpa of the port for interval algorithms in standard comput- Polish Academy of Sciences represents an interval as a point on a er hardware. Most modern processor chips come plane, somewhat like the representation of a complex number. Position equipped with circuitry for floating-point arith- of the point along the horizontal axis gives the value of the midpoint of metic, which reduces the process of manipulating the interval; height on the vertical axis encodes the radius (or half the significands and exponents to a single machine- width) of the interval. Diagonals extended to the horizontal axis reveal the interval itself. Any point within the shaded region represents an language instruction. In this way floating-point interval that includes the value zero. calculations become part of the infrastructure, available to everyone as a common resource. bound any true value of the original expression— Analogous built-in facilities for interval compu- but the narrower interval is surely preferable. tations are technologically feasible, but manufac- Another example: squaring an interval. The – 2 – × – turers have not chosen to provide them. A 1996 obvious definition [–x, x] =[–x, x] [–x, x] seems to article by G. William Walster of Sun Microsys- work in some cases, such as [1,2]2 = [1,4]. But tems asks why. Uncertainty of demand is surely what about [–2,2]2 = [–4,4]? Whoops! The square one reason; chipmakers are wary of devoting re- of a real number cannot be negative. The source sources to facilities no one might use. But Walster of the error is treating the two appearances of – cites other factors as well. Hardware support for [–x, x] in the right-hand side of the formula as if floating-point arithmetic came only after the they were independent variables; in fact, what- IEEE published a standard for the format. There ever value x assumes in one instance, it must be have been drafts of standards for interval arith- the same in the other. The same phenomenon can metic (the latest written by Dmitri Chiriaev and arise in expressions such as 2x/x. Suppose x is Walster in 1998), but none of the drafts has been the interval [2,4]; then naïvely following the rules adopted by any standards-setting body. of interval arithmetic yields the answer [1,4]. But of course the correct value is 2 (or [2,2]) for any Gotchas nonzero value of x. Although the principles of interval computing Comparisons are yet another murky area. may seem obvious or even trivial, getting the al- Computer programs rely heavily on conditional gorithms right is not easy. There are subtleties. expressions such as “if (x < y) then....” When x There are gotchas. The pitfalls of division by an and y are intervals, the comparison gets tricky. Is interval that includes zero have already been [1, 3] less than [2, 4], or not? Whereas there are mentioned. Here are a few more trouble spots. just three elementary comparisons for pointlike In doing arithmetic, we often rely on mathe- numbers (<, = and >), there are as many as 18 matical laws or truths such as x +–x = 0 and well-defined relations for intervals. It’s not al- (a + b)x = ax + bx. With intervals, some of these ways obvious which one to choose, or even how rules fail to hold. In general, an interval has no to name them. (Chiriaev and Walster refer to additive inverse; that is, given a nondegenerate “certainly relations” and “possibly relations.”) – – interval [–u,u], there is no interval [–v,v] for which Finally, look at what happens if a naïve im- – – [–u,u]+[–v,v] = [0,0]. There is no multiplicative in- plementation of the sine function is given an inverse either—no pair of nondegenerate intervals terval argument. Sometimes there is no prob- – × – for which [–u,u] [–v,v] = [1,1]. The reason is clear lem: sin([30°,60°]) yields the correct interval and fundamental: No valid operation can ever di- [0.5,0.866]. But sin([30°,150°]) returns [0.5,0.5], minish the width of an interval, and [0,0] and [1,1] which is an error; the right answer is [0.5,1.0]. are intervals of zero width. What leads us astray is the assumption that in- The distributive law also fails for intervals. In terval calculations can be based on end points an expression such as [1,2] × ([–3,–2] + [3,4]), it alone, which is true only for monotonic functions makes a difference whether you do the addition (those that never “change direction”). For other first and then multiply, or do two multiplications functions it is necessary to examine the interior of and then add. One sequence of operations gives an interval for minima and maxima. the result [0,4], the other [–3,6]. Strictly speaking, In fairness, it should be noted that many cher- either of these results is correct—both of them ished mathematical truths fail even in ordinary www.americanscientist.org 2003 November–December 487 (noninterval) floating-point arithmetic. An iden- knowledge, but they’re not much help when you – tity such as x = √x2 is not to be trusted in floating have to decide whether to light the fuse or take point. And there are remedies for all the interval the umbrella. But this is a psychological problem gotchas mentioned above—or at least strategies more than a mathematical one. Perhaps the solu- for coping with them. M. H. van Emden has tion is to compute with intervals, but at the end shown that by building on the existing IEEE let the machine report a definite, pointlike answer, floating-point standard (including its facilities for chosen at random from within the final interval. representing infinity), it would be possible to cre- ate a system of interval arithmetic that would Bibliography never fall into an error state, not even as a result Chiriaev, Dmitri, and G. William Walster. 1998. Interval of division by zero. (Of course the system would arithmetic specification. http://www.mscs.mu.edu/ sometimes return results such as [–∞,+∞], which ~globsol/Papers/spec.ps Dwyer, Paul S. 1951. Linear Computations. New York: John may be of questionable utility.) Wiley and Sons. Hass, Joel, Michael Hutchings and Roger Schlafly. 1995. The Intervals at Work double bubble conjecture. Electronic Research Announce- The interval community can point to a number ments of the AMS 1:95–102. of success stories. In 1995 Joel Hass, Michael Holzmann, Oliver, Bruno Lang and Holger Schütt. 1996. Hutchings and Roger Schlafly proved part of the Newton's constant of gravitation and verified numerical “double-bubble conjecture” by a method that en- quadrature. Journal of Reliable Computing 2(3):229–239. tailed extensive numerical calculations; they used Hyvönen, Eero, and Stefano De Pascale. 1996. Interval computations on the spreadsheet. In Applications of Interval interval methods to establish rigorous bounds on Computations, R. Baker Kearfott and Vladik Kreinovich computational errors. The conjecture concerns (eds.), pp. 169–209; Dordrecht, Boston: Kluwer Academic. soap films enclosing a pair of volumes, and states Kearfott, R. Baker. 1996. Rigorous Global Search: Continuous that the common configuration of two conjoined Problems. Dordrecht, Boston: Kluwer Academic. quasi-spherical bubbles has the smallest surface- Kulpa, Zenon. 2003. Diagrammatic analysis of interval lin- to-volume ratio. Hass, Hutchings and Schlafly ear equations. Part 1: Basic notions and the one-dimen- proved the conjecture for the case of two equal sional case. Reliable Computing 9:1–20. Markov, Svetoslav, and Kohshi Okumura. 1999. The contri- volumes, essentially by calculating the best possi- bution of T. Sunaga to interval analysis and reliable com- ble ratio for all configurations. The calculations puting. In Developments in Reliable Computing, Tibor did not have to be exact, but any errors had to be Csendes (ed.), pp. 167–188. Dordrecht, Boston: Kluwer. smaller than the differences between the various Moore, Ramon E. 1966. Interval Analysis. Englewood Cliffs, ratios. Interval methods provided this guarantee. N.J.: Prentice-Hall. (The general case of the double-bubble conjecture Moore, Ramon E. 1979. Methods and Applications of Interval was proved a few years later by Hutchings, Frank Analysis. Philadelphia: Society for Industrial and Ap- plied Mathematics. Morgan, Manuel Ritoré and Antonio Ros—with- Schulte, Michael J., and Earl E. Swartzlander, Jr. 1996. Soft- out interval arithmetic and indeed without com- ware and hardware techniques for accurate, self-validat- puters, using “only ideas, pencil, and paper.”) ing arithmetic. In Applications of Interval Computations, R. A quite different application of interval meth- Baker Kearfott and Vladik Kreinovich (eds.),pp. 381–404; ods was reported in 1996 by Oliver Holzmann, Dordrecht, Boston: Kluwer Academic. Bruno Lang and Holger Schütt of the University Semenov, Alexander L. 1996. Solving optimization problems with help of the Unicalc solver. In Applications of of Wuppertal. Instead of trying to control the er- Interval Computations, R. Baker Kearfott and Vladik rors of a calculation, they were estimating the Kreinovich (eds.), pp. 211–225; Dordrecht, Boston: Kluw- magnitude of errors in a physical experiment. The er Academic. experiment was a measurement of Newton’s Skeel, Robert. 1992. Roundoff error and the Patriot missile. gravitational constant G, done with two pendu- SIAM News 25(4):11. lums attracted to large brass weights. The interval Sunaga, Teruo. 1958. Theory of interval algebra and its ap- analysis assessed various contributions to the un- plication to numerical analysis. In RAAG Memoirs, Ggu- jutsu Bunken Fukuy-kai. Tokyo, Vol. 2, pp. 29–46. Also at certainty of the final result, and discovered a few http://www.cs.utep.edu/interval-comp/sunaga.pdf surprises. An elaborate scheme had been devised van Emden, M. H. 2002. New developments in interval for measuring the distance between the swinging arithmetic and their implications for floating-point stan- pendulums, and as a result this source of error dardization. http://arXiv.org/abs/cs.NA/0210015 was quite small; but uncertainties in the height of Walster, G. William. 1996. Stimulating hardware and soft- the brass weights were found to be an important ware support for interval arithmetic. In Applications of Interval Computations, R. Baker Kearfott and Vladik factor limiting the overall accuracy. Kreinovich (eds.), pp. 405–416; Dordrecht, Boston: Kluw- Would we be better off if intervals were used er Academic. for all computations? Maybe, but imagine the Warmus, M. 1956. Calculus of approximations. Bulletin de plight of the soldier in the field: A missile is to be l'Academie Polonaise des Sciences 4(5):253–257. Also at fired if and only if a target comes within a range http://www.cs.utep.edu/interval-comp/warmus.pdf of 5 kilometers, and the interval-equipped com- United States General Accounting Office. 1992. Patriot Missile puter reports that the distance is [4,6] kilometers. Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia. Washington: General Accounting Office. This is rather like the weather forecast that Young, Rosalind Cecily. 1931. The algebra of many-valued promises a 50-percent chance of rain. Such state- quantities. Mathematische Annalen 104:260–290. Also at ments may accurately reflect our true state of http://www.cs.utep.edu/interval-comp/young.pdf

488 American Scientist, Volume 91