z/OS Best Practices: Large Stand-Alone Dump Handling - Version 4 Update: 07/28/2014 Nobody ever wants to take a Stand-alone Dump (SADMP) of a z/OS system. Nevertheless, when your z/OS system is not responding due to an error in a critical system component, or it enters a wait state, a Stand-alone Dump is your only means of capturing sufficient diagnostic data to allow IBM Service to diagnose why your system entered the condition and recommend a fix to prevent the issue from happening again. Therefore, one needs to plan for the dump to be taken and processed as quickly as possible. Several z/OS releases have made improvements in taking and processing large stand- alone dumps. The system allows you to define a multiple-volume dump data set that simulates “striping,” and writing blocks of data in I/O priority order to each of the volumes defined in the data set. This greatly improved the elapsed time to capture a stand-alone dump. Stand-alone dump captures the page-frame table space and uses it as a pre-built dump index that relates to each absolute storage page, allowing IPCS to map the data ranges in the dump and improve the time to handle typical IPCS dump analysis requests for virtual storage access. Other enhancements allow stand-alone dump to be initiated from the operator’s console with a VARY command. z/OS functions also improve the handling of large stand-alone dumps, including the ability to subset a dump using the IPCS COPYDUMP command, allowing you to send a dump of the core system components to IBM while the rest of the dump is being processed and transferred. Most recently, in z/OS V1R13, the Problem Documentation Upload Utility allows transmission of multi-gigabyte files much more quickly, and encrypts the data all in the same process. This paper describes a set of “best practices” for ensuring the stand-alone dump is successful at capturing the necessary information for use by IBM Service, optimizing stand-alone dump data capture, and optimizing problem analysis time, . In particular, the following areas are described: Stand-Alone dump data set definition and placement IPCS performance considerations Preparing documentation to be analyzed Sending documentation to IBM support Testing your stand-alone dump setup This paper replaces an existing stand-alone dump best practices information, previously published as “z/OS Best Practices: Large Stand-Alone Dump Handling Version 2”. The URL remains the same: (http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD103286). © 2013 IBM Advanced Technical Support Techdocs - Washington Systems Center Version 7/28/2014 Page 1 http://www.ibm.com/support/Techdocs z/OS Large Stand-Alone Dump Best Practices This paper is also being updated with information which was described in “Every picture tells a story: Best practices for stand-alone dump”, published in the February 2007 issue of the z/OS Hot Topics Newsletter. You can access all issues of the Newsletter at: www.ibm.com/servers/eserver/zseries/zos/bkserv/hot_topics.html This paper assumes the installation is operating at least at the z/OS V1R13 level. Related enhancements available on lower z/OS levels are noted where appropriate. The planning steps are highlighted, followed by background information. 1. Plan a multi-volume stand-alone dump data set, being sure to place each volume on a separate DASD volume, and on a separate control unit. The best dump performance is realized when the dump is taken to a multiple-volume, DASD, stand-alone dump data set. Stand-alone dump exploits multiple, independent DASD volume paths to accelerate data recording. The dump data set is actually spread across all of the specified volumes, and not on each volume in succession. Stand-alone dump processing does not treat a multi-volume DASD dump data set as multiple, single data sets. (The creation of the multi-volume data set is covered in step 2.) A key to the performance of stand-alone dump is the rate at which the data is written to DASD. Modern DASD uses cache in the control unit to improve the performance of write operations. The placement of the multi-volume, stand-alone dump data set across Logical Subsystems (LSS) is strongly recommended to ensure contention in the I/O subsystem, (channels, director ports, cache, or Control unit back store) is avoided, thereby providing well performing DASD throughput throughout the duration of the stand-alone dump process. WSC Flash 10143 demonstrated that there are significant performance improvements observed with writing the data to a multi-volume DASD stand-alone dump data set, or to specific types of DASD. For more information, review the IBM performance analysis reports at the following URL: http://www- 03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10143 Therefore, when defining the placement of a multi-volume DASD, stand-alone dump data set, the following best practices apply: 1. Configure each volume on a separate Logical Subsystem (LSS) to ensure maximum parallel operation. The best performance of stand-alone dump can be attained when the multi-volume DASD data sets have the most separation, that is, separate physical control units and separate channel paths. 2. Configure, if possible, the control units to minimize the occurrence of other activity at the time of the stand-alone dump. For example, DB2 database recovery writing to a local database volume, from an LPAR that is still running against the same control unit as the stand-alone dump volume, will result in slower dump speed and may affect the elapsed time needed to restart an alternative DB2 on an LPAR that is still running. © 2013 IBM Advanced Technical Support Techdocs - Washington Systems Center Version 7/28/2014 Page 2 http://www.ibm.com/support/Techdocs z/OS Large Stand-Alone Dump Best Practices 3. Use FICON-attached DASD volumes where possible, which typically yield the best data transfer rates. FICON channels will deliver much better performance than ESCON channels. 4. For the best overall performance, dedicate a minimum of 4-5 DASD volumes to stand-alone dump, plus any additional capacity needed to contain the dump size, up to a maximum of 32 volumes. See the performance analysis report cited above for more information. IBM test results on z/OS V1R11 demonstrated that there is benefit to using up to 16 volumes spread over multiple channels. The test environment wrote a stand-alone dump at a rate of 1.2 to 1.5 Gb/second. However, you can configure up to 32 volumes in the multiple-volume data set to handle larger capacity if needed. 5. Do not define your stand-alone dump volumes as targets of a hyperswap. You could lose a completed stand-alone dump or a dump could get interrupted in the middle of the dump process. 6. Starting with the RSM Enablement offering for z/OS V1R13, you can use storage class memory (SCM), available on IBM zEnterprise EC12 servers, for paging. Doing so provides greatly-improved paging performance than can be achieved using DASD, and provides substantial improvements in SVC Dump and stand-alone dump capture time. The dump improvement is from capturing paged-out data on SCM faster than from DASD page data sets. While not recommended by IBM, stand-alone dump can also write to a fast tape subsystem. When directing the stand-alone dump to a tape drive, the dump will only use a single device and will not prompt for another device, so you cannot switch back to using a DASD device for this stand-alone dump. 2. Create the multi-volume stand-alone dump data set Define a stand-alone dump data set using the AMDSADDD utility1. Specify a volume list (VOLLIST) in AMDSADDD to designate a list of VOLSERs corresponding to each DASD volume making up the data set. A multi-volume data set will be allocated using the specified list of volumes. The utility uses the device number of the first volume to specify the data set to stand-alone dump. To achieve parallelism each volume should be on a different LSS when writing the dump. See “IBM System Test Example,” at the end of this paper, for a sample job using AMDSADDD to generate the stand-alone dump data set. 1 See MVS Diagnosis: Tools and Service Aids (GA22-7589), Chapter 4, for information on the AMDSADDD utility and any other Stand-alone Dump functions discussed in this paper. © 2013 IBM Advanced Technical Support Techdocs - Washington Systems Center Version 7/28/2014 Page 3 http://www.ibm.com/support/Techdocs z/OS Large Stand-Alone Dump Best Practices Be sure to catalog your stand-alone dump data set to prevent the possibility of accessing the “wrong version” of the data set when using IPCS COPYDUMP. (Step 4 covers information about IPCS COPYDUMP). Alternatively, IPCS offers a “SADMP Dump Data Set Utility,” available from the IPCS Utility menu. From the data set utility panel, specify whether to define, clear, or reallocate the stand-alone dump data set, specify its name and the volume serial numbers for the stand-alone dump data striping. This panel invokes the stand-alone dump allocation program to define the data set requested. The volume names, device type, and allocated space are confirmed as well. 3. Define a dump directory with the right attributes to facilitate post- processing of large stand-alone dumps and large SVC dumps Use IPCS to consolidate and/or extract ASIDs from the dump, as well as format and analyze the dump. IPCS uses a dump directory to maintain information about the layout and content of the dump.