NUMERICAL REPRODUCIBILITY AND INTERVAL ALGORITHMS 1 Numerical Reproducibility and Parallel Computations: Issues for Interval Algorithms Nathalie Revol, Member, IEEE, and Philippe Theveny´ Abstract—What is called numerical reproducibility is the The inclusion property is satisfied by the defi- problem of getting the same result when the scientific nition of the arithmetic operations, and other op- computation is run several times, either on the same erations and functions acting on intervals. Imple- machine or on different machines, with different types mentations of interval arithmetic on computers often and numbers of processing units, execution environments, computational loads etc. This problem is especially strin- rely on the floating-point unit provided by the gent for HPC numerical simulations. In what follows, the processor. The inclusion property is preserved via focus is on parallel implementations of interval arithmetic the use of directed roundings. If an interval is using floating-point arithmetic. For interval computations, given exactly for instance by its endpoints, the left numerical reproducibility is of course an issue for testing endpoint of its floating-point representation is the and debugging purposes. However, as long as the computed rounding towards −∞ of the left endpoint of the result encloses the exact and unknown result, the inclusion property, which is the main property of interval arithmetic, exact interval. Likewise, the right endpoint of its is satisfied and getting bit for bit identical results may not floating-point representation is the rounding towards be crucial. Still, implementation issues may invalidate the +1 of the right endpoint of the exact interval. inclusion property. Several ways to preserve the inclusion Similarly, directed roundings are used to implement property are presented, on the example of the product of mathematical operations and functions: in this way, matrices with interval coefficients. roundoff errors are accounted for and the inclusion Index Terms—interval arithmetic, numerical repro- property is satisfied. ducibility, parallel implementation, floating-point arith- In this paper, the focus is on the problems en- metic, rounding mode. countered while implementing interval arithmetic, using floating-point arithmetic, on multicore archi- tectures. In a sense these issues relate to issues I. INTRODUCTION known as problems of numerical reproducibility in Interval arithmetic is a mean to perform numeri- scientific computing using floating-point arithmetic cal computations and to get a guarantee on the com- on emerging architectures. The common points and puted result. In interval arithmetic, one computes the differences and concerns which are specific to with intervals, not numbers. These intervals are interval arithmetic will be detailed. guaranteed to contain the (set of) exact values, both The main contributions of this paper are the arXiv:1312.3300v1 [cs.NA] 11 Dec 2013 input values and computed values, and this property identification of problems that cause numerical is preserved throughout the computations. Indeed, irreproducibility of interval computations, and the fundamental theorem of interval arithmetic is the recommendations to circumvent these problems. inclusion property: each computed interval contains The classification proposed here distinguishes be- the exact (set of) result(s). To learn more about tween three categories. The first category, addressed interval arithmetic, see [29], [30], [32], [42], [43], in Section IV, concerns problems of variable com- [50]. puting precision that occur both in sequential and parallel implementations, both for floating-point and Universite´ de Lyon - LIP (UMR 5668 CNRS - ENS de Lyon - interval computations. These behaviours are mo- INRIA - UCBL), ENS de Lyon, France Nathalie Revol is with INRIA - tivated by the quest of speed, at the expense of [email protected] - WWW home page: accuracy on the results, be they floating-point or http://perso.ens-lyon.fr/nathalie.revol/ intervals results. However, even if these behaviours Philippe Theveny´ is with ENS de Lyon - [email protected] - WWW home page: hinder numerical reproducibility, usually they do http://perso.ens-lyon.fr/philippe.theveny/ not threaten the validity of interval computations: NUMERICAL REPRODUCIBILITY AND INTERVAL ALGORITHMS 2 interval results satisfy the inclusion property. modification of shared data structures. The order The second category, developed in Section V, in which communications from different senders concerns the order of operations. Due mainly to arrive at a common receiver, the relative order in the indeterminism in multithreaded computations, which threads or processes are executed, the order the order of operations may vary. It is the most in which shared data structures such as working acknowledged problem. Again, this problem is due lists are consulted or modified, is not deterministic. to the quest of short execution time, again it hin- Indeed, for the example of threads, the quest for ders numerical reproducibility of floating-point or speed leads a greedy scheduler to start a task as interval computations, but at first sight it does soon as it is ready, and not at the time or order at not invalidate the inclusion property. However, an which it should start in a sequential execution. One interval algorithm will be presented, whose validity consequence of this indeterminism is that speed is relies on an assumption on the order of operations. traded against numerical reproducibility of floating- The problems of the third category, detailed in point computations: computed results may differ, Section VI, are specific to interval computations and depending on the order in which events happened have an impact on the validity of interval results. in one run or another. However, numerical repro- The question is whether directed rounding modes, ducibility is a desirable feature, at least for debug- set by the user or the interval library, are respected ging purposes: how is it possible to find and fix a by the environment. These problems occur both for bug that occurs sporadically, in an indeterministic sequential and parallel computations. Ignoring the way? Testing is also impossible without numeri- rounding modes can permit to reduce the execution cal reproducibility: when a program returns results time. Indeed, changing the rounding mode can incur which differ from the results of a reference imple- a severe penalty: the slowdown lies between 10 mentation, nothing can be concluded in the absence and 100 on architectures where the rounding modes of numerical reproducibility. To address the issue of are accessed via global flags and where changing a numerical reproducibility, in [45], not only the use rounding mode implies to flush pipelines. However, of higher computing precision is recommended, but it seems that the most frequent motivation is either also the use of interval arithmetic or of some variant the ignorance of issues related to rounding modes, of it in order to get numerical guarantees, and the or the quest for an easier and faster development use of tools that can diagnose sensitive portions of of the execution environment by overlooking these code and take corrective actions. We follow these issues. tracks and we focus on interval arithmetic, the same For each category, we give recommendations conclusions can be drawn for its “variants” such as for the design of interval algorithms: numerical affine arithmetic or polynomial models, e.g. Taylor reproducibility is still not attained, but the impact models. of the problems identified is attenuated and the First, let us introduce the problem of numer- inclusion property is preserved. ical reproducibility in floating-point arithmetic. Floating-point arithmetic is standardised and the Before detailing our classification, numerical re- IEEE-754 standard [18], [19] specifies completely producibility is defined in Section II and then an the formats of floating-point numbers, their binary extensive bibliography is detailed in Section III. representation and the behaviour of arithmetic op- erations. One of the goals of this standardisation was to enable portability and reproducibility of II. PROBLEM OF NUMERICAL REPRODUCIBILITY numerical computations across architectures. How- In computing in general and in scientific com- ever, several runs of the same code on different puting in particular, the quest for speed has led platforms, or even on the same platform when the to parallel or distributed computations, from mul- code is parallel (typically, multithreaded) may yield tithreaded programming to high-performance com- different results. Getting the same result on the same puting (HPC). Within such computations, contrary platform is sometimes called repeatability. Only the to sequential ones, the order in which events occur word reproducibility will be used throughout this is not deterministic. Events here is a generic term paper. which covers communications, threads or processes, Let us give a first explanation to the lack of NUMERICAL REPRODUCIBILITY AND INTERVAL ALGORITHMS 3 numerical reproducibility. On some architectures, that rounding modes are not respected by numerical and with some programming languages and com- libraries such as BLAS [23], nor by compilers pilers, intermediate computations such as the in- when default options are used, nor by execution termediate sum in s a + b + c can take place environment for multithreaded computations, nor with variable precision. This precision can depend