The Green500 List: Escapades to Exascale
Total Page:16
File Type:pdf, Size:1020Kb
Noname manuscript No. (will be inserted by the editor) The Green500 List: Escapades to Exascale Tom Scogland, Balaji Subramaniam, and Wu-chun Feng ftom.scogland,balaji,[email protected] Department of Computer Science Virginia Tech Blacksburg, VA 24060 Abstract Energy efficiency is now a top priority. The first is more open-ended, but it focuses on maximum achievable four years of the Green500 have seen the importance of en- performance. This persistent drive toward more speed at any ergy efficiency in supercomputing grow from an afterthought cost has brought about an age of supercomputers that con- to the forefront of innovation as we near a point where sys- sume enormous quantities of energy, resulting in the need tems will be forced to stop drawing more power. Even so, for extensive and costly cooling facilities to operate these the landscape of efficiency in supercomputing continues to supercomputers (Markoff and Hansell, 2006; Atwood and shift, with new trends emerging, and unexpected shifts in Miner, 2008; Belady, 2007). previous predictions. In 2007, we created the Green500 (Feng and Cameron, This paper offers an in-depth analysis of the new and 2007) to bring awareness to this issue and to provide a venue shifting trends in the Green500. In addition, the analysis of- for supercomputers to compete on efficiency as a comple- fers early indications of the track we are taking toward exas- ment to the TOP500’s (Meuer, 2008) focus on speed. Since cale, and what an exascale machine in 2018 is likely to look that time, many milestones have been achieved, most no- like. Lastly, we discuss the new efforts and collaborations tably, (1) the first petaflop supercomputer and (2) the first toward designing and establishing better metrics, method- supercomputer GFLOP/watt supercomputer. Concurrently, ologies and workloads for the measurement and analysis of efficiency has entered the consciousness of the supercom- energy-efficient supercomputing. puting community at large and is now a primary concern in the design of new supercomputers. In this paper, we 1 Introduction – Explore the emergence of green high-performance com- As with all the great races in modern history, the supercom- puting (HPC). puting race has been myopically focused on a single metric – Track trends across the past four years of the Green500. of success. With the space race, the metric was which coun- – Discuss the results of the shift toward thinking green. try could reach the moon first. In supercomputing, the race – Investigate the implications of the above as we move into the future. This work was supported in part by NSF grant CCF-0848670, a DoD National Defense Science & Engineering Graduate Fellowship (ND- Of particular interest are the implications of past and SEG), and Supermicro. current trends on the feasibility of exascale computing sys- T. R. W. Scogland tems in the timeframe discussed in the DARPA IPTO exas- 2202 Kraft Drive cale study (Bergman et al, 2008). The study itself indicates Blacksburg, VA 24060 that power will be the determining factor in the success or failure of any exascale program. Past trends in the Green500 B. Subramaniam have led us to believe that at least one of the targets listed in 2202 Kraft Drive the study may be feasible, but as we look at the progression Blacksburg, VA 24060 in total power use and the aggregate progression of the list, some questions are raised as to whether the optimistic figure W. Feng 2202 Kraft Drive of 20 megawatts (MW) will be possible by 2018. Blacksburg, VA 24060 The rest of the paper is organized as follows. First, Sec- tion 2 discusses the background of and motivation behind 2 T. Scogland, B. Subramaniam, W. Feng the Green500. Section 3 offers a high-level analysis of ef- It delivered two orders of magnitude better energy efficiency ficiency, along with power, and a more in-depth analysis of than the Earth Simulator and ASCI Q and debuted at 205 the efficiency characteristics of the different types of ma- MFLOPS/W on the first Green500. Clearly, there was an chines that achieve high energy efficiency. Projections from inkling of the need for efficiency to achieve the greatest pos- current and past lists to exascale are discussed along with sible performance, but even so, it continues to be an uphill their implications in Section 4. Section 5 presents the inno- battle. vations and collaborations behind the continuing evolution The original goal of the Green500 was to raise aware- of the Green500 as well as directions for future growth. Fi- ness of the state of energy efficiency in supercomputing and nally, Section 6 presents concluding remarks along with fu- bring the importance of energy efficiency to the community ture work. on par with the importance of performance. In order to mea- sure and rank supercomputers in terms of their energy ef- 2 Background ficiency, the Green500 employs the LINPACK benchmark, Large-scale, high-performance computing (HPC) has reached as provided by the TOP500 list, combined with power mea- a turning point. Prior to 2001, the cost of purchasing a 1U surements. LINPACK solves a dense linear algebra problem server exceeded the annualized infrastructure and energy using LU factorization and backward substitution. It is de- (I&E) cost for that server , as depicted in Figure 1. By 2001, signed and tuned for load balancing and scalability. this annualized I&E cost for a server matched the cost of While the Green500 remains a ranking of the efficiency purchasing the server. By 2004, the annualized infrastruc- of the 500 fastest supercomputers in the world, we launched ture cost by itself matched the cost of purchasing the server, three additional lists in 2009, based on feedback from the and by 2008, the annualized energy cost by itself matched HPC community: the Little Green500, the HPCC Green500 the cost of purchasing the server. Ignoring power and energy and the Open Green500. As of now, the HPCC Green500 efficiency to pursue performance at any cost is no longer and Open Green500 have been discontinued due in part to feasible. Data centers and HPC centers are already feeling lack of participation and interest from the community. The the pinch across the industry from Yahoo! (Filo, 2009) to Little Green500 continues to operate and broadens the def- Google (Markoff and Hansell, 2006) to the National Secu- inition of a supercomputer to help guide purchasing deci- rity Agency (Gorman, 2006; croptome.org, 2008). sions for smaller institutions. To be eligible for the Little Green500, a supercomputer must be as “fast” as the 500th- 4000 Annual I&E ranked supercomputer on the TOP500 list 18-months prior Infrastructure Cost 3500 to the release of the Little Green500. Energy Cost Server Cost 3000 3 Efficiency 2500 Across these last four years, we have observed a steady climb 2000 Dollars in the energy efficiency of the Green500. We have been track- 1500 ing the average efficiency as compared to Moore’s Law and 1000 finding that the average does track closely, while the maxi- 500 mum surges ahead, improving at a rate faster than Moore’s Law. The extreme weight at the high end of this list, exem- 0 1990 1995 2000 2001 2004 2005 2008 2010 2014 plified by IBM’s BlueGene/Q machines occupying the top Year five slots of the current Green500, draws the average up to closely track Moore’s law while the median lags well be- Fig. 1 Annual amortized costs in the data center (Belady, 2007) hind, as shown in Figure 2. With each release, we see the The supercomputing community has been particularly distance between the high efficiency machines and the aver- guilty of seeking speed at the exclusion of all else. When age grow wider, to the point where many of the top machines we released the first Green500 in November of 2007, the can be considered to be outliers as they are more than 1.5 computers that resulted from the event known colloquially times the interquartile range above the median. as Computenik, were still in the TOP500, namely the Earth While on the topic of the high end of the list, we have Simulator supercomputer from Japan and ASCI Q from the a first this release — for the first time, the maximum ef- U.S. The former far outstripped the performance of the latter ficiency of the list actually decreased from the June 2011 by five-fold. Furthermore, the Earth Simulator had an effi- list to the November 2011 list. The BlueGene/Q supercom- ciency of only 5.60 MFLOPS/W while ASCI Q had an even puter at #1 grew in size and evidently lost efficiency as a lower score of 3.65 MFLOPS/W. result. Even so, the four production BlueGene/Q comput- In 2004, the first U.S. machine to regain the number one ers hold the top four slots with an energy efficiency of ap- position on the TOP500 was an IBM BlueGene/L prototype. proximately 2 GFLOPS/W, as shown in Table 1. The orig- The Green500 List: Escapades to Exascale 3 # Gf/W Computer 1 2.026 BlueGene/Q Custom (IBM Rochester) puter currently at #1 on the TOP500 draws a whopping 12 2 2.026 BlueGene/Q Custom (IBM Watson) megawatts of power, more than half the optimistic estimate 3 1.996 BlueGene/Q Custom2 (IBM Rochester) 4 1.988 BlueGene/Q Custom (DOE/NNSA/LLNL) for the power required to run an exaflop supercomputer in 5 1.689 Blue Gene/Q Prototype 1 (NNSA/SC) the latter part of the decade at 1/100th of the performance. 6 1.378 DEGIMA: ATI Radeon GPU (Nagasaki U.) 7 1.266 Bullx B505, NVIDIA 2090 (BSC-CNS) We will discuss trends toward exascale in greater detail in 8 1.010 Curie: Bullx B505, NVIDIA M2090 (GENCI) Section 4.