Japan’s 10 Peta FLOPS Supercomputer Development Project and its energy saving designs
Ryutaro Himeno 1) Group Director of R&D 2) Deputy Program Director 3) Director of Advance group, NGS R&D Center for Computational Science Center for Computer & Communication Contents
1. Self introduction 2. Japan’s next generation supercomputer R&D project: 1. Before the project started 2. Introduction and current status 3. Energy issues 3. Introduction of focused application area
2 Who am I?
1985 1987
1988
1995
1993
1990 Courtesy of Nissan 1992 3 4 Hardware development at RIKEN
Special purpose computers MD-GRAPE2 Theoretical Peak In 1989, GRAPE Performance developed by U. Tokyo 100Peta MD-GRAPE3 MDM:MD-GRAPE2 develop by RIKEN 10Peta Japan US GRAPE-DR MDM was integrated in (U.Tokyo/RIKEN) BlueGene/Q installed 1PetaFLOPS RSCC, 2004 MD-GRAPE3 1Peta Planed HPCS MD-GRAPE3 developed in (RIKEN) NLCF BlueGene/L 2004 367TeraFLOPS 100 ASC nd Purple 2 fastest record on MD- Tera Earth Simulator ASC Tsubame 41TeraFLOPS Red (TITech) GRAPE3: Gordon Bell Storm ASCI Q PACS-CS Prize, 2006. 10Tera RSCC ASCI (Tsukuba) Blue Pacific ASCI White (Riken) MD-GRAPE-3 will be ASCI Red integrated in RSCC soon. ASCI Blue Mountain 1Tera
*ASCI: Accelerated Strategic CP-PACS Computing Initiative
'96 '98 '00 '02 '04 '06 '08 '10 [year] 5 Current RSCC System installed in 2004 at Riken Advance Center for Computing & Communication
HDD User User Tape Storage User 20 TB User 200 TB
POWER4 Server: 7 Itanium2 Server: 2 Front end Tape Drive: 4 (16) Server Disk Cache: 2TB Myrinet InfiniBand
128Nodes Vector InfiniBand Parallel (256CPUs) Cluster×4 512Nodes (1024CPUs) NEC SX-7/32 Cluster Intel Xeon 3.06GHz 282.5GFLOPS 2GB/Node,Myrinet/IB, 256GB 1.56TFLOPS×4 Intel Xeon 3.06GHz MD-GRAPE3 4GB/Node, IB 64TFLOPS 6.26TFLOPS 6 Basic concept of RSCC
Each computer architecture has each advantage and disadvantage There is no single architecture which fits wide variety of applications RSCC is composed of 3 subsystem, Linux cluster with Xeon, Vector Parallel computer and MD-GRAPE3
We made a proposal of the NGSP based on this concept to MEXT
7 Our Proposal
Needs of Multi- Needs of Integration of scale Multi- multiple multiple physic simulation computation architecture components
Tightly-coupled Present Switch heterogeneous computer Slower connection Faster Faster Faster interconnect interconnect interconnect Proposing Faster Vector Scalar MD architecture interconnect Node Node Node Vector Scalar MD Node Node Node
FPGA Node 8 Policy and Outline of A Next Generation Supercomputer Project
Purpose of policy: development, installation and application of an advanced high performance supercomputer system, as one of Japan’s “Key Technologies of National Importance”
Total Budget: about 115 billion Yen ( 1.15 billion US dollars )
Period of Project: FY2006 – FY2012
9 9 Formation of the 10 Peta R&D Project
MEXT: Policy & Funding Office for Supercomputer Development Planning Industries
Industrial Forum Project Committee for Promotion of Supercomputing
R&D Scheme
RIKEN: Project HQ Next-Generation NII: Grid Middleware Advisory Supercomputer R&D Center and Infrastructure (Ryoji Noyori) Evaluation Scheme Board IMS: Nano Science (MEXT and CSTP) Project Leader: Tadashi Watanabe Simulation R&D G.: Ryutaro Himeno Evaluation Riken Wako Institute: Life Science Simulation Committees
Visiting Researchers from Univ. & National Labs.
(Note) NII: National Institute of Informatics, Computer Companies IMS: Institute for Molecular Science 10 Goals of the Next Generation Supercomputer Project
1. Development and installation of the most advanced high performance supercomputer system
2. Development and wide use of application software to utilize the supercomputer to the maximum extent
3. Provision of flexible computing environment by sharing the next generation supercomputer through connection with other supercomputers located at universities and research institutes
4. Establishment of “Advanced Computational Science and Technology Center (tentative name)”
11 11 Expansion of Highest Computer for Global Usage
National Leadership Sustained Performance (FLOPS) Government National Investment Leadership National 100P National Next-next- Infrastructure 100P Leadership next Institute, （The Next Generation Supercomputer） Generation University Project The Next Generation Supercomputer Project Next-next National Infrastructure 1P Generation 1P National Project Institute, Leadership University (Earth Simulator) National Earth Simulator Project Infrastructure Institute, Enterprise National University Company, Leadership 10T Laboratory 10T CP-PACS, Enterprise NWT Company, National Laboratory Infrastructure Enterprise Institute Company, University Laboratory 100G Personal 100G Personal Entertainment Personal National Entertainment PC, Home Server Infrastructure Entertainment Enterprise PC, Home Server Workstation, Institute, PC, Home Server Company, Workstation, Game Machine, Digital TV University Workstation, Game Machine, Digital TV Laboratory Game Machine, Digital TV 1990 2000 2010 2015 2020 2025 12 Schedule of Project
FY2006 FY2007 FY2008 FY2009 FY2010 FY2011 FY2012
Conceptual Prototype and Production,Production, installation,installation, Processing unit DetailedDetailed designdesign design evaluation andand adjustmentadjustment ytmBuildings System Front-end unit Basic Basic Detailed design Tuning and improvement (total system design Detailed design Production and evaluation Tuning and improvement software) design
Shared file BasicBasic DetailedDetailed designdesign Production,Production, installation,installation, andand adjustmentadjustment system designdesign
Applications Next-Generation Integrated Nanoscience Development,Development, production,production, andand evaluationevaluation VerificationVerification Simulation Next-Generation Integrated Development, production, and evaluation Verification Life Simulation Development, production, and evaluation Verification
Computer building DesignDesign ConstructionConstruction
Research building DesignDesign ConstructionConstruction
present
13 Optimization of Computer Design Requirement from Grand Challenges Requirement from Computer Centers
Power, Space 1400 Computer center in 1200 Japan 1000
800
600 設置面積（㎡) Nano Sci. Life Sci. Others Reliability, operability 400 (Engineering etc) 200 0 0 500 1000 1500 2000 2500 3000 3500
電力（kW）
Cost (development, manufacturing, maintenance
Popular numerical scheme
FMO MO FVM Fast transform Searching algorism Optimum system Other Project WatchTechnology Survey Real space Linear RISM density function Monte Carlo FEM Solver method Fastest in the world
Operation & Utilization
21 Selected Target applications Essential Element technologies Power savving SOI Light transmit. Software LSI process Low-k tech. OS, compiler/// High-k Nano Science Life ScienceEnv./ DesasterEngineering.Physics, Astro. Spin off to the consumer electronics （6) （6） prev. （3） （4） （2） Technology Limit
14 Target applications and BM suit The application software committee started selecting target applications for making a benchmark suit in January, 2006. 21 codes were selected in March and NextBMT suit was developed.
15 NextBMT and Peta-scale BMT(1/2) Peta-scale BMT
Fields Program name Description
SimFold Prediction of Protein Structure
GNISC Inference of Genetic Networks from Experimental Data of Gene Expression
MLTest Validation of statistical significance for development of individualized medicine Life (6) MC-Bflow Simulation for blood-flow analysis
sievgene/myPresto Docking simulation of protein and drug
ProteinDF Ab-initio Molecular Dynamics Calculation in Large-Scale Protein Systems
GAMESS/FMO FMO Molecular Orbital Calculation
Modylas Massively-parallel multipurpose software for molecular dynamics calculation
RSDFT Ab-initio Molecular Dynamics Calculation in Real Space Nono (6) RISM/3D-RISM Analysis of Electron States of Proteins in Solution with the 3D-RISM/FMO Method
PHASE First-Principles Molecular Dynamics Simulation within the Plane-Wave Pseudopotential formalism
Octa Coarse-Grained Molecular Dynamics Calculation
16 NextBMT and Peta-scale BMT(2/2)
Peta-scale BMT LatticeQCD Study of elementary particle and nuclear physics based on Lattice QCD Physics simulation Astronomy NINJA/ASURA Super large-scale gravitational many-body simulation for finding the celestial (2) origin
NICAM Nonhydrostatic ICosahedral Atmospheric Model (NICAM) for Global-Cloud Resolving Simulations
Geophysics Seism3D Simulation of Seismic-Wave Propagation and Strong Ground Motions (3)
COCO Super-resolution Ocean General Circulation Model
Cavitation Computation of Unsteady Cavitation Flow by the Finite Difference Method
LANS Computation of compressible fluid in aircraft and spacecraft analysis
Engineering(4) FrontSTR Structural Calculation by Finite Element Method (FEM)
FrontFlow/Blue Unsteady Flow Analysis based on Large Eddy Simulation (LES)
17 System Configuration
The Next-Generation Supercomputer is designed as hybrid general-purpose supercomputer that provides the optimum computing environment for a wide range of simulations. •Calculations will be performed in processing units that are suitable for the particular simulation. •Parallel processing in a hybrid configuration of scalar and vector units will make larger and more complex simulations possible.
18 Characteristics
Hybrid Supercomputer System composed of Scalar and Vector Units
Vector unit is suitable for continuum physics simulation Scalar unit is suitable for Data Base application, particle-based simulation Hybrid system has both strong points for users to choose better units for different applications Control unit Control
Vector Unit Front end computer Scalar Unit (Integrated OS) Vector processor Data base, based on the Processing unit Processing homology search Scalar Unit Vector Unit Earth Simulator e-- - - Similarity with PC I3 e- strong for I- • high speed low continuum Solar cell power CPU physics Homology • New strong search Climate change CFD simulation network for Nono device • New generation enormous unit Disk Plazma Physics low-power vector parallelism processor with Local file Local file Global共有ファイル file optical network
19 For Energy Saving,
Cogeneration system: gas turbine power generator & steam generator using natural gas Generates electric power and steam Steam is used in cooling system which makes child water Child water is used in two ways Water-air heat exchanger Directly provided to computer Clock frequency is decided from energy saving point of view
20 Cogeneration system: 2 points of Saving Energy
Total energy efficiency of cogeneration system is better than ordinary commercial electric power plant
Using natural gas as fuel, CO2 will be reduced at the same time Direct water cooled computer system can save pumping loss of air blowing for cooling Full water cooled system has difficulty in maintenance We will use partial water cooling system
21 Photographs of the facilites
2009/02/06
2009/03/04
2009/04/25 22 Major Applications of Next Generation Supercomputer
Targeted as grand challenges 23 Nano-Science Simulation Software
Next-Generation Next-Generation information Energy Function Materials Solar energy Medicines, New drug, Nonlinear optical Device fixation Next-Generationand DDS Nano quantum devices Fuel alcohol Nano BiomoleculesViruses Spin electronics Nafion Fuel cells Anticancer drugs Ultra high-density 15nm Protein control storage devices Electric energy MesoscaleWater structure of Liposome naflon membrane Nano processes46 nm for DDC Integrated electronic storage Electronic devices Integrated Semi- conduction in macroscopic system 27 nm integrated systems 5nm Nafion membrane Polio virus Self- organized Molecular magnetic nanodots Condensed assembly Orbiton matters
RMSD 4.8 Å(all Cα) (orbital waves) Domain Protein folding “off” “on” Ferromagnetic light Capsulation Doping of fullerene Optical Electrons half-metals light Molecular and carbon switch and Quantum nanotubes Electron theory lightElectrons molecules chemistry dynamics of solids To create next-generation nano-materials (new semiconductor materials, etc.) by integrating
theories (such as quantum chemistry, statistical dynamics and solid electron theory) One-dimensional and - simulationSelf- techniques in the fields of new-generation information functions/materials, assembly Water molecules crystal of silicon nano-biomaterials, and energyinside lisozyme 24 cavity Toward therapeutic Life-Sciencetechnology Simulation Software
Surgical Drug Drug Delivery High Intensity procedures development System Focused Catheters Ultrasound Micromachines Tailor-made Regenerative Sizemedicine Molecular medicine Cellular Organ andHyperthermia Proteins/ scale scale body scale DNA Whole Organs body Cells Tissues Cardiova -8~-6 -5~-4 -3~-2 -1 scular 0 10 10 10 10 system 10 Micro Meso Macro Protein structural Molecular Fluids, heat, Vascular system analysis network structures modeling Drug response analysis Achievement of analysis chemicalMacroscopic reactions approach Skeleton model MD/first principle/quantum Continuous entity simulations chemistry
25 Total Simulation of Living Matter
Goal Understand the life phenomena: reproducing life program on computer Develop new medicine and medical equipment
Approach combination of analytical approach and Data-driven approach on Peta-scale computer Molecule Data-Based •Understand the scale approach to life phenomena: Analytical analysis approach reproducing life program on Cell Peta scale simulation scale computer New •Develop new medicine and Organ/Whole Fusion of Data methodology medical body scale based approach and Anaｌytical approach equipment
Basic principal Experimental (Equations) data 26 Cyber Science Infrastructure Plan
ＶＯs ( Virtual Organizations ） University/inter- university research Next Generation institutes VO Supercomputer Project VO
Virtual research environment for Industrial project VO various fields Infrastructural middleware （GRID、Infrastructure for certification, etc.)
Operated by National Institute of Informatics (NII)
(Note) VO :Virtual Organization 27 27 CACST: Center for Advanced Computational Science and Technology (tentative name)
Computer science and Computational science Both researchers will gather and expect to develop new research fields and methodologies Currently, we are designing the center and operation policy of the supercomputer The users will be chosen by a new committee independent from RIKEN to pick up valuable subjects
Kobe Port- Island
28 Tentative Plan of the computer center
Hub of the Computer Science and Computational Science in Japan and Asia. Cross point of various sciences and technologies using HPC Life Eng. Operation of 10-Peta FLOPS SupercomputerNono User support and spreadingPhysics the usage in various fieldGeo Development of the next-next supercomputer Meteology Research on computational science Astro Education COEセ CACST Nuclear Industrial ンター use
Computer Science 29 Summary From energy saving point of view, gas cogeneration system is adopted For saving pumping loss of air handling, water cooling system is partially adopted We are focusing on application development in Nano science and Life science Currently all development of hardware and software are on schedule
30