POWER® Processor-Based Systems
Total Page:16
File Type:pdf, Size:1020Kb
IBM® Power® Systems RAS Introduction to IBM® Power® Reliability, Availability, and Serviceability for POWER9® processor-based systems using IBM PowerVM™ With Updates covering the latest 4+ Socket Power10 processor-based systems IBM Systems Group Daniel Henderson, Irving Baysah Trademarks, Copyrights, Notices and Acknowledgements Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: Active AIX® POWER® POWER Power Power Systems Memory™ Hypervisor™ Systems™ Software™ Power® POWER POWER7 POWER8™ POWER® PowerLinux™ 7® +™ POWER® PowerHA® POWER6 ® PowerVM System System PowerVC™ POWER Power Architecture™ ® x® z® Hypervisor™ Additional Trademarks may be identified in the body of this document. Other company, product, or service names may be trademarks or service marks of others. Notices The last page of this document contains copyright information, important notices, and other information. Acknowledgements While this whitepaper has two principal authors/editors it is the culmination of the work of a number of different subject matter experts within IBM who contributed ideas, detailed technical information, and the occasional photograph and section of description. These include the following: Michael Mueller, George Ahrens, Jim O’Connor, K Paul Muller, Steven Gold, Marc Gollub, Ravi A. Shankar, Kevin Reick, Peter Heyrman, Kanwal Bahri, Dan Hurlimann, Kaveh Naderi, Kanisha Patel, Hoa Nguyen, and Julissa Villarreal. POWER9 and Power10 processor-based systems RAS Page 2 Table of Contents Trademarks, Copyrights, Notices and Acknowledgements .................. 2 Trademarks ......................................................................................................................... 2 Notices ................................................................................................................................ 2 Acknowledgements ............................................................................................................. 2 Table of Contents ..................................................................................... 3 Introduction .............................................................................................. 7 Section 1: Overview of POWER9 and Power10 Processor-based systems ..................................................................................................... 8 Comparative Discussion ..................................................................................................... 8 Figure 1: POWER9/Power10 Server RAS Highlights Comparison ............................................ 8 IBM Power E1080 System.................................................................................................. 10 System Structure ............................................................................................................... 10 Figure 2: Power E1080 System Structure ................................................................................ 10 Power E1080 Processor RAS ............................................................................................ 11 Power E1080 System Memory .......................................................................................... 11 Figure 3: DDMIM Memory Features ......................................................................................... 11 Buffer ................................................................................................................................................ 12 OMI ................................................................................................................................................... 12 Memory ECC .................................................................................................................................... 12 Dynamic Row Repair ........................................................................................................................ 12 Spare Temperature Sensors ............................................................................................................ 12 Unique to 4U DDIMM: Spare DRAMs .............................................................................................. 12 Power Management ......................................................................................................................... 12 Power E1080 SMP Interconnections ................................................................................ 13 Power E1080 Processor to Processor SMP Fabric Interconnect With a CEC Drawer ........ 13 Power E1080 CEC Drawer to CEC Drawer SMP Fabric Interconnect Design.................... 13 Figure 4: SMP Fabric Bus Slice ................................................................................................ 13 Figure 5: Time Domain Reflectometry ...................................................................................... 14 Power E1080 SMP Fabric Interconnect Design within a CEC Drawer ............................... 14 Internal I/O .......................................................................................................................... 15 IBM Power® E980 .............................................................................................................. 16 Introduction ....................................................................................................................... 16 Figure 6: Power E980 System Structure .................................................................................. 16 Overview Compared to POWER8 ...................................................................................... 16 Across Node Fabric Bus RAS ............................................................................................ 17 Figure 7: POWER9 Two Drawer SMP Cable Connections ...................................................... 17 Clocking ............................................................................................................................ 18 Internal I/O ........................................................................................................................ 18 Figure 8: I/O Connections ......................................................................................................... 18 NVMe Drives ..................................................................................................................... 18 Processor and Memory RAS ............................................................................................. 19 Infrastructure RAS Enhancements .................................................................................... 19 POWER9 and Power10 processor-based systems RAS Page 3 Power® E950 ...................................................................................................................... 20 Introduction ....................................................................................................................... 20 Figure 9: Power E950 System Illustration (Four Socket System) ............................................ 20 Figure 10: Power E950 Memory Riser ..................................................................................... 21 I/O ..................................................................................................................................... 21 Figure 11: Power E950 I/O Slot Assignments .......................................................................... 21 PCIe CAPI 2.0 ................................................................................................................... 22 DASD Options ................................................................................................................... 22 Infrastructure ..................................................................................................................... 22 IBM Power® S914, IBM Power® S922, IBM Power® S924 ............................................... 23 Introduction ....................................................................................................................... 23 System Features ............................................................................................................... 23 Figure 12: Simplified View of 2 Socket Scale-Out Server ........................................................ 24 I/O Configurations ............................................................................................................. 24 Figure 13: Scale-out Systems I/O Slot Assignments for 1 and 2 Socket Systems .................. 24 Infrastructure and Concurrent Maintenance ....................................................................... 25 POWER9: PCIe Gen3 I/O Expansion Drawer .................................................................... 27 Figure 14: PCIe Gen3 I/O Drawer RAS Features ...................................................................