Power-Aware High-Performance Cluster with Commodity
Total Page:16
File Type:pdf, Size:1020Kb
MegaProto/E: Power-Aware High-Performance Cluster with Commodity Technology £ £ Taisuke Boku £ Mitsuhisa Sato Daisuke Takahashi Hiroshi Nakashima† £ Hiroshi Nakamura‡ Satoshi Matsuoka Ü Yoshihiko Hotta £ University of Tsukuba, taisuke,msato,daisuke,hotta @hpcs.cs.tsukuba.ac.jp Ý Toyohashi University of Technology, [email protected] Þ University of Tokyo, [email protected] Ü Tokyo Institute of Technology, [email protected] Abstract 1. Introduction In our research project named “Mega-Scale Computing The PC cluster solution continues to be a highly at- Based on Low-Power Technology and Workload Modeling”, tractive method for achieving high performance comput- we have been developing a prototype cluster not based on ing (HPC) with its high performance/cost ratio supported by ASIC or FPGA but instead only using commodity technol- commodity technology. However, there is concern about its ogy. Its packaging is extremely compact and dense, and its applicability to highly scalable systems with tens of thou- performance/power ratio is very high. Our previous pro- sands or millions of processors. In order to allay this con- totype system named “MegaProto” demonstrated that one cern, we at least have to solve the problems associated cluster unit, which consists of 16 commodity low-power pro- with power consumption, space and dependability. Our re- cessors, can be successfully implemented on just 1U height search project named “Mega-Scale Computing Based on chassis and it is capable of up to 2.8 times higher perfor- Low-Power Technology and Workload Modeling” aims to mance/power ratio than ordinary high-performance dual- establish fundamental technologies for this purpose. Our Xeon 1U server units. research investigates the feasibility and dependability of We have improved MegaProto by replacing the CPU million-scale parallel systems as well as the programmabil- and enhancing the I/O performance. The new cluster unit ity on such a large scale. named “MegaProto/E” with 16 Transmeta Efficeon proces- For the feasibility study, we especially focused on the sors achieves 32 GFlops of peak performance, which is 2.2- performance/power/space ratio problem. To this end, we fold greater than that of the original one. The cluster unit is have developed the prototype system based on commodity- equipped with an independent dual network of Gigabit Eth- only technology, that is, we only used commodity pro- ernet, including dual 24-port switches. The maximum power cessors and network elements. In order to achieve a high consumption of the cluster unit is 320 W, which is compa- density implementation we developed cluster chassis units rable with that of today’s high-end PC servers for high per- which can house a number of processors in a small space. formance clusters. The recent trend of dual-core CPUs such as Pentium-D or Performance evaluation using NPB kernels and HPL Opteron-D clearly demonstrates that the best way to im- shows that the performance of MegaProto/E exceeds that prove total performance is to introduce multi-processors of a dual-Xeon server in all the benchmarks, and its per- having relatively low-performances, instead of increasing formance ratio ranges from 1.3 to 3.7. These results re- the CPU clock frequency on a single processor. Based on veal that our solution of implementing a number of ultra this concept, an ideal platform is a cluster of ultra low- low-power processors in compact packaging is an excellent power processors implemented in a small space with high way to achieve extremely high performance in applications density. with a certain degree of parallelism. We are now building a Green Destiny[1] is a successful example of the above multi-unit cluster with 128 CPUs (8 units) to prove that this concept. While it consists of a commercial blade-style pro- advantage still holds with higher scalability. cessor card, we have designed and implemented a higher density collection of processors in a 1U height chassis con- taining 17 Transmeta Crusoe processors. This prototype 1-4244-0054-6/06/$20.00 ©2006 IEEE unit was named “MegaProto”[2, 6] which provides 14.9 high performance/cost ratio supported by a non-HPC mar- Gflops of peak performance. When we designed the first ket. prototype of MegaProto, we also intended to build an en- In our Mega-Scale Computing Project[5], we investi- hanced version incorporating more powerful processor and gated (i) hardware/software cooperative low-power technol- I/O bus. This enhanced version has now been completed ogy and (ii) workload modeling and model-based manage- with the name of “MegaProto/E” (‘E’ stands for Efficeon, ment of large scale parallel tasks together with the faults oc- the name of the new processor). In this paper, we describe curring in such a system. Our study covers the processor ar- the design, implementation and performance evaluation of chitecture, compiler, networking, cluster management and the MegaProto/E cluster unit. programming for a system based on the above concept. The rest of the paper consists of the following. Section 2 gives an overview of our Mega-Scale computing project 2.2. Conceptual Design of MegaProto along with the conceptual design of the MegaProto series Hereafter, the word “MegaProto” is used to refer to our cluster unit. In Section 3, we describe in detail the de- overall prototyping system used for the feasibility study of sign and implementation of MegaProto/E. After describing our Mega-Scale Computing concept. However, we called the performance evaluation in Section 4, the cluster design our first prototype version by the same name in previous with higher scalability based on MegaProto/E is described papers[2, 6]. In order to distinguish between the two dif- in Section 5. Finally, conclusions and future works are given ferent versions of the prototype system, we explicitly call in Section 6. the first one “MegaProto/C” and the second one “Mega- 2. Mega-Scale Project and MegaProto Clus- Proto/E”. The two suffix letters ‘C’ and ‘E’ stand for the code names of the CPUs used, Crusoe and Efficeon, respec- ter Unit tively. The overview of our Mega-Scale Computing Project and In considering the Peta-Flops scale systems based on the conceptual design were described in more detail in [2]. commodity CPU and network technologies, it is impossi- In this section, we give just a brief description of them. ble to avoid the issues of power consumption and spacing. We consider the approximate upper limits of such a system 2.1. Overview of the Mega-Scale Computing as being 10 MW for power consumption and 1,000 racks Project for space. These specifications are still hard to realize in practice, but are not impossible. With these limitations, our Today, Peta-Flops computing is not just a dream any- primary goal is to attain 1 TFlops/10 kW/rack as the per- more, and several projects have been launched aiming to formance/power/space ratio. MegaProto is a series of pro- achieve this level of computational performance. In these totype systems designed to demonstrate that such systems projects, the common key issues include (i) how to imple- can be built. Since the system was to be based on commod- ment ultra large-scale systems, (ii) how to reduce the power ity system formation, we built it using 19-inch 42U height consumption per Flop, and (iii) how to control ultra large- standard racks. With the exception of the space for the net- scale parallelism. From the viewpoint of hardware technol- work switch, 32U can be assigned for computation nodes. ogy, the first two issues are critical. As demonstrated by Thus, the goal was to make a 32 GFlops/310 W/1U build- BlueGene/L[3], the state-of-the-art MPP today, one promis- ing block as the basic unit. Neither the performance or the ing way to achieve Peta-Flops computing is to build an MPP power consumption criteria can be achieved with today’s having very low-power processors and a simple switch- high-end CPUs such as Intel Xeon, Intel Itanium2 or AMD less network to reduce both space and power consumption. Opteron, even with multi-way SMP configuration. How- However, such a system requires a dedicated hardware plat- ever, it is possible to satisfy both criteria using state-of-the- form including specially designed processor chips and net- art, low power CPUs with DVS and very low voltage drive work and system racks, which are expensive to implement if we can aggregate 10 to 20 CPUs in a single 1U chas- and require a long time to design and implement. sis. Even using blade-style cards, it is impossible to achieve On the other hand, the rapid progress of computation such high densities, but we have finally solved the prob- and communication technologies that are not limited to lem of developing a mother board composed of 16 daughter HPC applications suggests another approach on low power cards with the size of a one dollar bill. commodity technology, both for the CPU and the network. Another important issue regarding the design of the unit While an ordinary PC cluster for HPC applications consists chassis is selecting an interconnection network suitable for of high-performance CPUs consuming tens of Watts and such a basic design. If we choose a high-end CPU as the a wide-bandwidth network such as Infiniband[4], we can node processor, we will require a high-end network inter- combine a number of very low-power and medium-speed face with 1 GByte/sec bandwidth or greater for efficient par- CPUs designed for laptops and Gigabit Ethernet with a very allel processing.