The IBM-Astron DOME energy efficient microserver: status, plans and demo Ronald P. Luijten – Data Motion Architect [email protected] IBM Research - Zurich 26 May 2015 DISCLAIMER: This presentation is entirely Ronald’s view and not necessarily that of IBM. COMPUTE is FREE – DATA is NOT Ronald P. Luijten – Data Motion Architect [email protected] IBM Research - Zurich 26 May 2015 DISCLAIMER: This presentation is entirely Ronald’s view and not necessarily that of IBM. IBM Research - Zurich: From Atoms to Big Data Analytics © 2015 International Business Machines Corporation •3 The World is Our Lab World's largest information More than 3,000 IBM invested technology research scientists and $6B on R&D in organization engineers 2014 Africa India T.J Watson Zurich China Austin Ireland Haifa Tokyo Almaden Brazil Australia © 2015 International Business Machines Corporation IBM Research - Zurich - Established in 1956 - 45+ different nationalities - Open Collaboration: - Framework Programme7: 277 projects engaged, 68 funded, 1,900 partners - Horizon2020: 52 applications, 341 partners - Two Nobel Prizes (1986 and 1987) - Binnig and Rohrer Nanotechnology Centre opened in 2011 (Public Private Partnership with ETH Zürich and EMPA) © 2015 International Business Machines Corporation Scientific Departments Big Data Cognitive Computing & Computational Sciences: next generation cognitive systems and technologies, big data and secure information Analytics management, HPC and computational sciences Industry and Cloud Solutions: transforming industries through data and domain specific algorithms, delivering secure enterprise solutions on the Cloud, and enabling systems of engagement for enterprises Cloud & Computing Infrastructure: novel server and accelerator architectures, software-defined environments, security, infrastructure for big data management, technologies for moving big data on logical & physical layers, all-Flash storage systems and next-generation memory technologies Science & Technology: underpin each of the new organizations and will explore science and research projects to discover new technologies including a new focus on quantum technologies or storing and processing information Atoms © 2015 International Business Machines Corporation LONG RANGE SIGNALING & CONTROL A long-range, low-data-rate communications infrastructure that needs fewer base stations to serve more simple devices like online smoke detectors, temperature sensors or smart electrical heating controllers. I&CS © 2015 International Business Machines Corporation •7 FLASHSYSTEM 820 The IBM FlashSystem 820 achieved more than 6 million Input/Output Operations Per Second (IOPS) running an IBM DB2 workload on IBM Power servers. the first time Flash storage has outperformed hard disks in all aspects, including capacity and performance density, and cost per IOPS and energy efficiency. C&CI © 2015 International Business Machines Corporation •8 HYBRID MEMORY New memory being tested by EPFL, ETH and IBM which combines128 terabytes of flash and 64 terabytes of DRAM to handle data of detailed brain models for the Blue Brain and Human Brain Projects. •9 © 2014 IBM Corporation CC&CS SOLAR CONCENTRATOR Airlight Energy, a Swiss-based supplier of solar power technology has partnered with IBM Research to bring affordable solar technology to the market by 2017. The system can concentrate the sun’s radiation 2,000 times and convert 80 percent of it into useful energy to generate 12 kilowatts of electrical power and 20 kilowatts of heat on a sunny day—enough to power several average homes. S&T © 2015 International Business Machines Corporation •1 0 How will you change the world? Join IBM Research and find out. Openings in Our Openings in Our Global Labs Zurich Lab DOME: • ppp Astron, IBM, Dutch gvt • 20MEur funding over 5 years • Started feb 2012 Ronald P. Luijten – ETHz lecture – 26 May 2015 12 SKA (Square Kilometer Array) to measure Big Bang Start of Big Protons nucleosynthesis End of nucleo- Bang Inflation created through fusion synthesis Modern Universe 0 10 -32 s 10 -6s 0.01s 3min 380’000 years 13.8 Billion years Picture source: NZZ march 2014 © 2012 IBM Corporation 13 Ronald P. Luijten – ETHz lecture – 26 May 2015 13 CSP SDP ~ 1 PB/Day. 330 disks/day ~ 10 Pb/s ?? ? 120’000 disks/yr 86’400 sec/day 15 ExaByte/day Top-500 Supercomputing(11/2013)…. 0.3Watt/Gflop/s Too hard Today’s industry focus is 1 Eflop @ 20MW. (2018) ( 0.02 Gflop/s) Most recent data from SKA: CSP….max. power 7.5MW SDP….max. power 1 MW Latest need for SKA – 4 Exaflop (SKA1 - Mid) Too easy (for us) 1.2GW…80MW Factor 80-1200 Moore’s law multiple breakthroughs needed ©© 20142012 IBMIBM CorporationCorporation Ronald P. Luijten – ETHz lecture – 26 May 2015 14 •IBM at IBMCeBIT 2013 –/ Rethink ASTRON your business DOME project DOMETechnology Project: roadmap 5 development Years, 33M Euro •Sustainable •User (Green) Computing •Nanophotonics •Data & Streaming Platform •System Analysis -Student projects -Events •Algorithms & Machines -Research Collaboration •Computing •Transport •Storage -Microservers -Nanophotonics -Access Patterns -Accelerators -Real Time Communications -Compressive Sampling •15 •©•© 20132012 IBMIBM CorporationCorporation •15 Ronald P. Luijten – ETHz lecture – 26 May 2015 15 Ronald’s “More’s law” – the result of Moore’s law Ronald P. Luijten – ETHz lecture – 26 May 2015 16 Dream on Ronald P. Luijten – ETHz lecture – 26 May 2015 17 The reports on my life are greatly exaggerated… (it used to be 2x every 18 months, remember?) Source: Rajeeb Hazra, Intel, 14 th SOS meeting, March 2010 Ronald P. Luijten – ETHz lecture – 26 May 2015 18 You get what you measure (Lord Kelvin) • Corollary: Don’t expect to get what you don’t measure • Key is to decide what to measure (seems trivial, right?) Incumbents dominate Sustaining battles Entrants typically win at disruption DOME µServer Motivation & Objectives •Create the worlds highest density 64 bit µ-server drawer –Useful to evaluate both SKA radio-astronomy and IBM future business –Platform for Business Analytics appliance pre-product research –High energy efficiency / very low cost –Commodity components, HW + SW standards based –Leverage ‘free computing’ paradigm –Enhance with ‘Value Add’: packaging, system integration, … –Density and speed of light •Most efficient cooling using IBM technology (ref: SuperMUC June 2012 TOP500 machine) •Must be true 64 bit to enable business applications •Must run server class OS (SLES11 or RHEL6, or equivalent) –Precluded ARM (64-bit Silicon was not available) –PPC64 is available in SoC from FSL since 2011 –(no $$$ to build a new SoC…) •This is the DOME project capability demonstrator – not a product Ronald P. Luijten – ETHz lecture – 26 May 2015 21 Definition µServer: The integration of an entire server node motherboard * into a single microchip except DRAM, Nor-boot flash and power conversion logic. 139mmx55mm 245mm 305mm * no graphics Ronald P. Luijten – ETHz lecture – 26 May 2015 22 Definition µServer: The integration of an entire server node motherboard * into a single microchip except DRAM, Nor-boot flash and power conversion logic. This does NOT imply low performance! 139mmx55mm 245mm 305mm * no graphics Ronald P. Luijten – ETHz lecture – 26 May 2015 23 T4240 Chip Overview 12 core – fully dual threaded 1.8 GHz ppc64 (e6500) 12 DP-FPU; 12 128b Altivec 3 DDR3 channels at 1.86GT/s 3x 0.5MB L3 cache 4x 10GbE + 2x SATA PCIe 3.0 HW packet acceleration RegEx Pattern Match acc. Crypto acceleration 28nm TSMC Bulk CMOS 239mm2 - ~1.7B transistors 7 Power states (2 power gating) 111Mbit SRAM, 6M FF Ronald P. Luijten – ETHz lecture – 26 May 2015 24 T4240 Chip Overview This is NOT the ideal part However, a very good one Built for Embedded market Impressive power management features Not great for HPC: not enough DP-FP units No DDR prefetching Ronald P. Luijten – ETHz lecture – 26 May 2015 25 DOME Compute node board form factor T4240 SoC 55 mm 55 Standard 240 pin DDR3 (lid removed) memory DIMM board 30 mm FRONT 133 mm P5020 SoC Decoupling 55 mm Capacitors (Lid Removed) area 133 mm BACK 139 mm 139 mm P5020/P5040 T4240 (Generation 1) Generation 2 Ronald P. Luijten – ETHz lecture – 26 May 2015 26 Planned System: 2U rack unit 19” 2U Chassis w/ Combined Cooling & Power 128 compute node boards 1536 cores / 3072 Threads 6 TB DRAM 1.28Tbps Ethernet (@40Gbps) Datacenter-in-a-box • Expected 2U unit total power: ~ 6kW • Integrated mains power converter to 12V distribution: 12V / 500A • Each compute node has own 12V / 40W converter • Common Power Converter boards for all other supplies • High radix 10GbE / 40GbE switch boards (under construction) • Connects to Mains, Rack level Water, 32x 40Gbps Ethernet • Hot-water cooled for efficiency and density Ronald P. Luijten – ETHz lecture – 26 May 2015 27 Node Cooling Design & Validation Compute Nodes Electrical + Power converter boards Thermal Storage boards Interface Water In 3 layer Laminated Copper Plate SoC FR4 Carrier Water In CeBIT Demo, april 15 Inlet Junction Measured Maximum cooling water temp Tj thermal Res. capacity Cooling variant [C] [C] [K/W] [W] OF R240 Cu,no heat pipe 45 85 1.11 36 OF R240 Cu with heat pipe 45 85 0.85 47 OF R240 Cu with heat pipe 45 75 0.85 36 Performance Measurement Results CPU Freescale T4240 Intel Xeon E3-1230L v3 12 cores; 24 thr. 4 cores; 8 threads 28nm Bulk 22nm FinFet CPU2006 Benchmark System: T4240RDB-PB System: Supermicro X10SAE Test Environment 1.666 GHz core clock, 1.8 GHz core clock; Turbo disabled 1.866 GT/s 6GB DRAM, 3 channels 1.666 GT/s 8 GB DRAM, 2 channels Fedora 20, Kernel 3.12.19 Fedora 19, Kernel 3.13.9 GCC 4.7.2 GCC 4.8.2 gcc options: -O3 -mcpu=powerpc64 gcc options: -O3 -march=native -mtune=native 6.86 20.7 CINT-base – 1 thread CINT-base – all threads 109.34 (24 threads) 77.6 (8 threads) Coremark - all threads 188K (24 threads) 65K (8 threads) Ronald P. Luijten – ETHz lecture – 26 May 2015 29 Performance Measurement Results CPU Freescale T4240 Intel Xeon E3-1230L v3 12 cores; 24 thr.