AIX Version 7.2 Performance Tools Guide and Reference IBM AIX Version 7.2 Performance Tools Guide and Reference IBM Note Before using this information and the product it supports, read the information in “Notices” on page 315. This edition applies to AIX Version 7.2 and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 2015, 2018. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents About this document ......... v The procmon tool ............ 212 Highlighting .............. v Overview of the procmon tool....... 212 Case-sensitivity in AIX ........... v Components of the procmon tool...... 212 ISO 9000................ v Filtering processes........... 215 Performing AIX commands on processes ... 215 Performance Tools Guide and Reference 1 Profiling tools ............. 215 The timing commands ......... 215 What's new in Performance Tools Guide and The prof command .......... 216 Reference ............... 2 The gprof command .......... 217 CPU Utilization Reporting Tool (curt) ...... 2 The tprof command .......... 219 Syntax for the curt Command ....... 2 The svmon command .......... 226 Measurement and Sampling ........ 3 Security .............. 227 Examples of the curt command ....... 4 The svmon configuration file ....... 227 Simple performance lock analysis tool (splat) ... 32 Summary report metrics......... 227 splat command syntax.......... 32 Report formatting options ........ 228 Measurement and sampling ........ 33 Segment details and -O options ...... 230 Examples of generated reports ....... 35 Additional -O options ......... 234 Hardware performance monitor APIs and tools .. 50 Reports details ............ 238 Performance monitor accuracy ....... 51 Remote Statistics Interface API Overview .... 259 Performance monitor context and state .... 51 Remote Statistics Interface list of subroutines 260 Performance monitoring agent ....... 52 RSI Interface Concepts and Terms ..... 261 POWERCOMPAT events ......... 53 A Simple Data-Consumer Program ..... 266 Thread accumulation and thread group Expanding the data-consumer program.... 269 accumulation ............. 57 Inviting data suppliers ......... 270 Security considerations ......... 57 A Full-Screen, character-based monitor .... 272 The pmapi library ........... 58 List of RSI Error Codes ......... 272 The hpm library and associated tools ..... 72 The nmon recording tool ......... 274 Perfstat API programming ......... 81 Asynchronous I/O statistics ....... 274 API characteristics ........... 81 nmon recording tool commands ...... 275 Global interfaces............ 82 I/O statistics ............ 278 Component-Specific interfaces ....... 101 Kernel statistics............ 283 WPAR Interfaces ........... 166 Memory statistics ........... 284 RSET Interfaces............ 176 Network statistics ........... 288 Cached metrics interfaces ........ 182 Process statistics ........... 299 Node interfaces............ 185 Processor statistics........... 300 Change history of the perfstat API ..... 193 Recording configuration details ...... 312 Kernel tuning ............. 198 WLM statistics ............ 314 Migration and compatibility ....... 199 Workload partition statistics ....... 314 Tunables file directory ......... 199 Tunable parameters type ........ 200 Common syntax for tuning commands .... 200 Notices .............. 315 Tunable file-manipulation commands .... 202 Privacy policy considerations ........ 317 Initial setup ............. 206 Trademarks .............. 317 Reboot tuning procedure ........ 206 Recovery Procedure .......... 206 Index ............... 319 Kernel tuning using the SMIT interface.... 206 © Copyright IBM Corp. 2015, 2018 iii iv AIX Version 7.2: Performance Tools Guide and Reference About this document The Performance Tools Guide and Reference provides experienced system administrators, application programmers, service representatives, system engineers, end users, and system programmers with complete, detailed information about the various performance tools that are available for monitoring and tuning AIX® systems and applications running on those systems. The information contained in this document pertains to systems running AIX 7.1, or later. Any content that is applicable to earlier releases will be noted as such. Highlighting The following highlighting conventions are used in this document: Bold Identifies commands, subroutines, keywords, files, structures, directories, and other items whose names are predefined by the system. Also identifies graphical objects such as buttons, labels, and icons that the user selects. Italics Identifies parameters whose actual names or values are to be supplied by the user. Monospace Identifies examples of specific data values, examples of text similar to what you might see displayed, examples of portions of program code similar to what you might write as a programmer, messages from the system, or information you should actually type. Case-sensitivity in AIX Everything in the AIX operating system is case-sensitive, which means that it distinguishes between uppercase and lowercase letters. For example, you can use the ls command to list files. If you type LS, the system responds that the command is not found. Likewise, FILEA, FiLea, and filea are three distinct file names, even if they reside in the same directory. To avoid causing undesirable actions to be performed, always ensure that you use the correct case. ISO 9000 ISO 9000 registered quality systems were used in the development and manufacturing of this product. © Copyright IBM Corp. 2015, 2018 v vi AIX Version 7.2: Performance Tools Guide and Reference Performance Tools Guide and Reference The performance of a computer system is based on human expectations and the ability of the computer system to fulfill these expectations. The objective for performance tuning is to make those expectations and their fulfillment match. The path to achieving this objective is a balance between appropriate expectations and optimizing the available system resources. The performance-tuning process demands great skill, knowledge, and experience, and cannot be performed by only analyzing statistics, graphs, and figures. If results are to be achieved, the human aspect of perceived performance must not be neglected. Performance tuning also takes into consideration problem-determination aspects as well as pure performance issues. Expectations can often be classified as either of the following: Item Descriptor Throughput expectations A measure of the amount of work performed over a period of time Response time expectations The elapsed time between when a request is submitted and when the response from that request is returned The performance-tuning process can be initiated for a number of reasons: v To achieve optimal performance in a newly installed system v To resolve performance problems resulting from the design (sizing) phase v To resolve performance problems occurring in the run-time (production) phase Performance tuning on a newly installed system usually involves setting some base parameters for the operating system and applications. Throughout this book, there are sections that describe the characteristics of different system resources and provide guidelines regarding their base tuning parameters, if applicable. Limitations originating from the sizing phase will either limit the possibility of tuning, or incur greater cost to overcome them. The system might not meet the original performance expectations because of unrealistic expectations, physical problems in the computer environment, or human error in the design or implementation of the system. In the worst case, adding or replacing hardware might be necessary. Be particularly careful when sizing a system to permit enough capacity for unexpected system loads. In other words, do not design the system to be 100 percent busy from the start of the project. When a system in a productive environment still meets the performance expectations for which it was initially designed, but the demands and needs of the utilizing organization have outgrown the system's basic capacity, performance tuning is performed to delay or even to avoid the cost of adding or replacing hardware. Many performance-related issues can be traced back to operations performed by a person with limited experience and knowledge who unintentionally restricted some vital logical or physical resource of the system. Note: The metrics reported by any statistics tool such as lparstat, vmstat, iostat, mpstat and so on including the applications that are based on Perfstat API or SPMI API varies to a certain extent at any point of time. If the command is run multiple times for an instance, the values may not be similar for that instance. © Copyright IBM Corp. 2015, 2018 1 What's new in Performance Tools Guide and Reference Read about new or significantly changed information for the Performance Tools Guide and Reference topic collection. How to see what's new or changed In this PDF file, you might see revision bars (|) in the left margin that identify new and changed information. October 2017 The following information is a summary of the updates made to this topic collection: v Added information about the “Multiplexing of PMU context” on page 71 topic. CPU Utilization Reporting Tool (curt) The CPU Utilization Reporting Tool (curt) command converts an AIX trace file into a number of statistics related to CPU utilization and either