Iowa State University Patents Iowa State University Research Foundation, Inc. 9-14-1993 Method and system for benchmarking computers John L. Gustafson Iowa State University Follow this and additional works at: http://lib.dr.iastate.edu/patents Part of the Applied Mathematics Commons, and the Numerical Analysis and Scientific Computing Commons Recommended Citation Gustafson, John L., "Method and system for benchmarking computers" (1993). Iowa State University Patents. 128. http://lib.dr.iastate.edu/patents/128 This Patent is brought to you for free and open access by the Iowa State University Research Foundation, Inc. at Iowa State University Digital Repository. It has been accepted for inclusion in Iowa State University Patents by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. Method and system for benchmarking computers Abstract A testing system and method for benchmarking computer systems. The system includes a store containing a scalable set of tasks to be performed to produce a solution in ever-increasing degrees of resolution as a larger number of the tasks are performed. A timing and control module allots to each computer a fixed benchmarking interval in which to perform the stored tasks. Means are provided for determining, after completion of the benchmarking interval, the degree of progress through the scalable set of tasks and for producing a benchmarking rating relating to the degree of progress for each computer. Keywords applied mathematics Disciplines Applied Mathematics | Computer Sciences | Numerical Analysis and Scientific omputC ing | Physical Sciences and Mathematics This patent is available at Iowa State University Digital Repository: http://lib.dr.iastate.edu/patents/128 U S005245638A United States Patent [191 [11] Patent Number: 5,245,638 Gustafson [45] Date of Patent: Sep. 14, 1993 [54] METHOD AND SYSTEM FOR munications of the ACM, vol. 31, No. 5, May 1988, pp. BENCHMARKING COMPUTERS 532-533. [75] Inventor: John L. Gustafson, Ames, Iowa J. L. Gustafson et al., “Development of Parallel Meth ods for a l024-Processor Hypercube,” SIAM Journal on [73] Assignee: Iowa State University Research Scientific and Statistical Computing, vol. 9, No. 4, Jul. Foundation, Inc., Ames, Iowa 1988, pp. 609-638. [21] Appl.No.: 605,237 M. Berry et al., “The Perfect Club Benchmarks: Effec tive Performance Evaluationof Supercomputers,” Cen [22] Filed: Oct. 29, 1990 ter for Supercomputing Research and Development Report [51] 1111.01; ............................................ .. G06F 11/34 No. 827, Nov. 1988. [52] us. (:1. .................................. .. 395/575; 364/550; P. Messina et al., “Benchmarking Advanced Architec 364/514; 364/551.01 ture Computers,” Caltech Report C3P- 712, Jun. 1989. [58] Field of Search ...................................... .. 395/575; “i86OTM Microprocessor Performance Brief,” Intel 364/200-2769 MS, 551.01, 550, 514,200 MS Corporation Release 1.1, Aug. 1989. [56] References Cited C. M. Goral et al., “Modeling the Interaction of Light Between Diffuse Surfaces,” Computer Graphics, vol. 18, U.S. PATENT DOCUMENTS No. 3, Jul. 1984, pp. 213-222. 4,849,879 7/1989 Chinnaswamy et al. ......... .. 364/200 Primary Examiner-Robert W. Beausoliel, Jr. 4,980,8255,031,124 12/19907/1931 TootellBosinoff et et al. a]. .. Assistant Examiner-Phung My Chung Attorney, Agent, or Firm—-Leydig, Voit & Mayer 5,072,3715,086,386 12/19312/1992 BennerIslam ........... et a1. .. [57] ABSTRACT 5,088,058 2/1992 Salsburg ......... .. 5,161,116 11/1992 Schneider et a1. ........... .. 364/55101 A testing system and method for benchmarking com OTHER PUBLICATIONS puter systems. The system includes a store containing a scalable set of tasks to be performed to produce a solu H. J. Curnow and B. A. Wichmann, “A Synthetic tion in ever-increasing degrees of resolution as a larger Benchmark,” Computer Journal, Feb. 1976. number of the tasks are performed. A timing and con D. H. Bailey and J. T. Barton, “The NAS Kernel trol module allots to each computer a ?xed benchmark Benchmark Program,” NASA Technical Memorandum ing interval in which to perform the stored tasks. Means 86711, Aug. 1985. “MIPS, Dhrystones and Other Tales,” Supermicro, are provided for determining, after completion of the SM-54, Apr. 30, 1986, pp. 6-15. benchmarking interval, the degree of progress through J. J. Dongarra, “Performance of Various Computers the scalable set of tasks and for producing a benchmark Using Standard Linear Equations Software in a Fortran ing rating relating to the degree of progress for each Environmen ,” Argonne National Laboratory, Techni computer. cal Memorandum No. 23, Feb. 2, 1988. J. L. Gustafson, “Reevaluating Amdahl’s Law,” Com 19 Claims, 2 Drawing Sheets OPERATOR NFUT l COMFUTE OUTPUT (SOL'NS) DBNNmgL WA l J 12 NSTR. 11 r STORE IETHOD~ DIHBISIONS EMISSIVITY IEFLECTIVITY 1 4 J IENCHMAIKNG SYSTEM US. Patent Sep. 14, 1993 Sheet 1 of 2 5,245,638 OPERATOR FIG- 1 INPUT i COMPUTER A A ’ OUTPUT (SOL'NS) EVAITJ'A TIMING I ‘' TION ICONTROL DATA MEAN J INsTR.& ‘age:. 11 [12 ' S'ON STORE METHOD- SOé-ONS CHECK OLOGY 8* M‘ L“, \_13A LIBRARY PUTER INFO. DIMENSIONS A 80W EMISSIVITY ' VERIF. REFLECTIVITY \ 141 1 3B BIENCHMARKING SYSTEM \13 v k 10 RATING OUTPUT (PROBLEM SIZE) US. Patent Sep. 14, 1993 V Sheet 2 of 2 5,245,638 LEFT FACE (RED): TOP FACE (LIGHT SOURCE): EMISSIVITY = (O, 0, 0) EMISSIVITY = (1.27, 1.27, 1.27) REFLECTIVITY = (0.99. O, 0) REFLECTIVITY = (0.80. 0.80, 0.80) BACK FACE (LIGHT GRAY): EMISSIVITY = (0, 0, 0) K I REFLECTIVITY =(O.84, 0.84, 0.84) FRONT FACE (GRAY): -/ I - - - - EMISSIVITY = (o, 0, 0) / RIGHT FACE (BLUE): REFLECTIVITY = (0.54, EMISSIVITY = (O, O, O) 0.54, 0.54) REFLECTIVITY = (O, O, 0.99) BOTTOM FACE (LIGHT GRAY): EMISSIVITY = (O, O, O) REFLECTIVITY =(O.84, 0.84, 0.84) FIG. 2 FIG. 3 20 SET - A TIME INTERVAL Ql I22 SET VALUE OF n COMPUTER READS DATA AND INSTRUCTIONS DEFINING PROBLEM, OPERATES ON PROB LEM TASKS AND OUTPUTS AND STORES SOL'NS TO TASKS 5,245,638 1 2 control allots to each computer a ?xed benchmarking METHOD AND SYSTEM FOR BENCHMARKING interval in which to perform the stored tasks. Means are COMPUTERS provided for determining, after completion of the benchmarking interval, the degree of progress through FIELD OF THE INVENTION the scalable set of tasks and for producing a benchmark This invention relates generally to computers and, ing rating relating to the degree of progress for each more speci?cally, to a method and system for bench computer. The inventive benchmarking system is pro marking computers having a wide range of perfor vided in various computer architecture forms and Ian mance capabilities. guages and is designed to evaluate complete task perfor 10 mance, including input, set-up of equations, and compu BACKGROUND OF THE INVENTION tation and output of a solution. Computer power has increased over 70% per year BRIEF DESCRIPTION OF THE DRAWINGS for the last 50 years (or over 11 orders of magnitude), thus making it difficult to measure and compare perfor FIG. 1 a schematic block diagram of the inventive mance of computers having vastly different perfor benchmarking system; mance capabilities with a benchmarking system that FIG. 2 illustrates a problem which is de?ned in a does not scale. Furthermore, since a given make of preferred embodiment of the inventive benchmarking parallel processor may offer a performance range of system; and over 8000 to 1, the scaling problem exists even if applied FIG. 3 is a ?ow chart generally illustrating the to computers of current vintage. Any benchmark of method used by the inventive benchmarking system to ?xed size is soon obsoleted by hardware advances that evaluate user computers. render the time and space requirements of the bench mark unrepresentative of realistic use of the equipment. DESCRIPTION OF THE PREFERRED A common workaround consists of performing a ?xed EMBODIMENTS size task repetitively, but this has proven to be less than 25 While the invention will be described in connection satisfactory. with particular preferred embodiments, it will be under A related issue is the dif?culty of scienti?cally com stood that it is not intended to limit the invention to paring computers with vastly different architectures or those particular embodiments. On the contrary, it is programming environments. A benchmark designed for intended to cover all alternatives, modi?cations and one architecture or programming model puts a different equivalents as may be included within the spirit and architecture at a disadvantage, even when nominal per scope of the invention as de?ned by the appended formance is otherwise similar. Assumptions such as claims. arithmetic precision, memory topology, and “legal” Ideally, a benchmark should be scalable, broad in language constructs are typically wedded to the job to be timed, in the interest of controlling as many variables 35 architectural scope, simple to apply and understand, as possible. This “ethnocentrism” in benchmark design representative of the way people actually use comput has hampered comparison of novel parallel computers ers, and scienti?cally honest. A proper benchmark is with traditional serial computers. Examples of popular both a task engineered to meet these goals and a set of benchmarks that have some or all of the foregoing rules governing the experimental procedures. It is more drawbacks are LINPACK, the “PERFECT TM Club” than just an application program or excerpt. , the Livermore Loops, SPEC, Whetstones, and Dhry Many of these goals are at odds with one another. As stones. with any engineering design, a certain amount of com promise is necessary. In particular, a single benchmark SUMMARY OF THE INVENTION with a single ?gure-of-merit cannot fully characterize It is a primary object of the present invention to pro performance for the entire range of computing tasks. vide an improved benchmarking system for evaluating It has historically been an assumption that in measur computers having vastly different performance capabil ing computer performance the problem being solved be ities.