Advanced POWER Virtualization on IBM Eserver p5 Servers: Architecture and Performance Considerations

Detailed description of the POWER5 architecture

In-depth analysis of Advanced POWER Virtualization features

Performance analysis and application tuning

Ben Gibbs, Balaji Atyam, Frank Berres, Bruno Blanchard, Lancelot Castillo, Pedro Coelho, Nicolas Guerin, Lei Liu, Cesar Diniz Maciel, Carlos Sosa, Ravikiran Thirumalai ibm.com/redbooks

International Technical Support Organization

Advanced POWER Virtualization on IBM Eserver p5 Servers: Architecture and Performance Considerations

November 2005

SG24-5768-01 Note: Before using this information and the product it supports, read the information in “Notices” on page ix.

Second Edition (November 2005)

This edition applies to IBM ^ p5 servers that include the POWER5 microprocessor architecture and the IBM AIX 5L Version 5.3 operating system.

© Copyright International Business Machines Corporation 2005. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents

Notices ...... ix Trademarks ...... x

Preface ...... xi The specialists who wrote this redbook ...... xii Become a published author ...... xv Comments welcome...... xv

Chapter 1. Introduction...... 1 1.1 Performance tuning redefined ...... 3 1.1.1 Understanding performance ...... 3 1.1.2 Performance considerations ...... 6

Chapter 2. IBM POWER5 architecture ...... 9 2.1 Introduction ...... 10 2.2 Chip design ...... 12 2.3 POWER5 enhancements ...... 13 2.4 POWER5 instruction pipelines ...... 14 2.4.1 Instruction fetching ...... 15 2.4.2 Branch prediction ...... 16 2.4.3 Instruction decoding and preprocessing ...... 17 2.4.4 Group dispatch ...... 17 2.4.5 Register renaming...... 18 2.4.6 Instruction execution ...... 19 2.5 Caches ...... 21 2.5.1 Level 2 (L2) cache...... 25 2.5.2 Level 3 (L3) cache...... 27 2.5.3 Summary of caches on POWER5...... 30 2.5.4 Address translation resources...... 30 2.6 Timing facilities ...... 31 2.7 Dynamic power management ...... 33 2.8 Processor Utilization Resource Register (PURR) ...... 34 2.9 Large POWER5 SMPs ...... 36 2.10 Summary ...... 40

Chapter 3. Simultaneous multithreading ...... 41 3.1 What is multithreading?...... 42 3.2 POWER5 simultaneous multithreading features ...... 44 3.2.1 Dynamic switching of thread states...... 46

© Copyright IBM Corp. 2005. All rights reserved. iii 3.2.2 Snooze and snooze delay...... 47 3.3 Controlling priority of threads ...... 49 3.3.1 Dynamic resource balancing (DRB) ...... 49 3.3.2 Adjustable thread priorities ...... 50 3.3.3 Thread priority implementation ...... 52 3.4 Software considerations ...... 55 3.4.1 Simultaneous multithreading aware scheduling ...... 55 3.4.2 Thread priorities on AIX 5L V5.3 ...... 56 3.4.3 Thread priorities on Linux ...... 58 3.4.4 Cache effects ...... 58 3.5 Simultaneous multithreading performance ...... 59 3.5.1 Engineering and scientific applications ...... 59 3.5.2 Simultaneous multithreading benchmarks ...... 61 3.6 Summary ...... 71

Chapter 4. POWER Hypervisor...... 73 4.1 POWER Hypervisor implementation ...... 76 4.1.1 POWER Hypervisor functions ...... 79 4.1.2 Micro-Partitioning extensions ...... 85 4.1.3 POWER Hypervisor design...... 87 4.2 Performance considerations ...... 90

Chapter 5. Micro-Partitioning ...... 93 5.1 Partitioning on the IBM eServer p5 systems ...... 94 5.2 Micro-Partitioning implementation ...... 96 5.2.1 Virtual processor dispatching ...... 104 5.2.2 Phantom interrupts ...... 112 5.3 Performance considerations ...... 115 5.3.1 Micro-Partitioning considerations ...... 116 5.3.2 Locking considerations ...... 121 5.3.3 Memory affinity considerations ...... 126 5.3.4 Idle partition consideration ...... 127 5.3.5 Application considerations in Micro-Partitioning ...... 128 5.3.6 Micro-Partitioning planning guidelines ...... 133 5.4 Summary ...... 142

Chapter 6. Virtual I/O...... 143 6.1 Introduction ...... 144 6.2 POWER Hypervisor support for virtual I/O ...... 145 6.2.1 Virtual I/O infrastructure ...... 147 6.2.2 Types of connections ...... 149 6.3 The IBM Virtual I/O Server ...... 152 6.3.1 Providing high availability support...... 156 6.4 Virtual Serial Adapter (VSA) ...... 163 iv Advanced POWER Virtualization on IBM Eserver p5 Servers 6.5 Virtual Ethernet ...... 164 6.5.1 Virtual LAN ...... 164 6.5.2 Virtual Ethernet connections ...... 169 6.5.3 Benefits of virtual Ethernet ...... 170 6.5.4 Limitations and considerations ...... 171 6.5.5 POWER Hypervisor switch implementation ...... 171 6.5.6 Performance considerations ...... 174 6.5.7 VLAN throughput at different processor entitlements ...... 176 6.5.8 Comparing throughput of VLAN to physical Ethernet ...... 178 6.5.9 Comparing CPU utilization ...... 180 6.5.10 Comparing transaction rate and latency ...... 182 6.5.11 VLAN performance ...... 183 6.5.12 VLAN implementation guidelines ...... 185 6.6 Shared Ethernet Adapter...... 186 6.6.1 Shared Ethernet Adapter performance ...... 190 6.6.2 Request/response time and latency ...... 193 6.7 Implementation guidelines...... 196 6.7.1 Guidelines for Shared Ethernet Adapter sizing ...... 197 6.7.2 Guidelines for physical Ethernet sizing ...... 202 6.7.3 Control of threading in the Shared Ethernet Adapter ...... 204 6.8 Virtual SCSI...... 205 6.8.1 Client and server interaction ...... 209 6.8.2 AIX 5L V5.3 device configuration for virtual SCSI...... 210 6.8.3 Interpartition communication ...... 212 6.8.4 Disk considerations ...... 215 6.8.5 Configuring for redundancy...... 217 6.8.6 Performance considerations ...... 220 6.8.7 Sizing a virtual SCSI server ...... 226 6.9 Summary ...... 230

Chapter 7. AIX 5L Version 5.3 operating system support ...... 233 7.1 Introduction ...... 234 7.1.1 Processors ...... 234 7.1.2 Dynamic re-configuration ...... 239 7.1.3 Existing performance commands enhancement ...... 239 7.1.4 New performance commands ...... 248 7.1.5 Paging space ...... 251 7.1.6 Logical Volume Manager (LVM) ...... 252 7.1.7 Virtual local area network (VLAN) ...... 254 7.1.8 EtherChannel ...... 255 7.1.9 Partition Load Manager...... 255

Chapter 8. POWER5 system performance ...... 257

Contents v 8.1 Performance commands ...... 258 8.1.1 lparstat command ...... 258 8.1.2 mpstat command...... 264 8.1.3 vmstat command...... 268 8.1.4 iostat command...... 270 8.1.5 sar command ...... 272 8.1.6 topas command...... 275 8.1.7 xmperf command ...... 278 8.2 Performance tuning approach ...... 283 8.2.1 Global performance analysis...... 283 8.2.2 CPU analysis...... 289 8.2.3 Memory analysis ...... 294 8.2.4 Disk I/O analysis ...... 296 8.2.5 Network I/O analysis ...... 304

Chapter 9. Application tuning ...... 311 9.1 Performance bottlenecks identification ...... 312 9.1.1 Time commands, time utilities, and time routines ...... 314 9.2 Tuning applications using only the compiler ...... 317 9.2.1 Compiler brief overview...... 317 9.2.2 Most commonly used flags ...... 321 9.2.3 Compiler directives for performance ...... 327 9.2.4 POWER5 compiler features ...... 332 9.3 Profiling applications ...... 336 9.3.1 Hardware performance monitor ...... 336 9.3.2 Profiling utilities ...... 343 9.4 Memory management ...... 350 9.5 Optimization of critical sections in the code ...... 351 9.5.1 General rules for optimization strategies...... 353 9.5.2 Array optimization ...... 353 9.5.3 Loop optimization ...... 355 9.6 Optimized libraries...... 360 9.6.1 MASS Library ...... 361 9.6.2 ESSL library ...... 368 9.7 Parallel programming general concepts ...... 370

Chapter 10. Partition Load Manager ...... 373 10.1 When and how should I use Partition Load Manager? ...... 374 10.1.1 Partition Load Manager and other load-balancing tools ...... 374 10.1.2 When to use Partition Load Manager ...... 376 10.1.3 How to deploy Partition Load Manager ...... 382 10.2 More about Partition Load Manager installation and setup ...... 383 10.2.1 Overview of Partition Load Manager behavior ...... 383

vi Advanced POWER Virtualization on IBM Eserver p5 Servers 10.2.2 Management versus monitoring modes ...... 385 10.2.3 Configuration file and tunables ...... 386 10.3 Managing and monitoring with Partition Load Manager ...... 390 10.3.1 Managing multiple partitions ...... 391 10.3.2 Extra tips about the xlplm command ...... 392 10.3.3 Examples of Partition Load Manager commands output ...... 393 10.4 Partition Load Manager performance impact ...... 396 10.4.1 Partition Load Manager resource requirements ...... 396 10.4.2 Partition Load Manager impact on managed partitions...... 397

Related publications ...... 401 IBM Redbooks ...... 401 Other publications ...... 402 Online resources ...... 402 How to get IBM Redbooks ...... 403 Help from IBM ...... 403

Index ...... 405

Contents vii viii Advanced POWER Virtualization on IBM Eserver p5 Servers Notices

This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested tho