Session 8926 z/OS Debugging: a review of System Trace MVS Core Technologies Project – March 3rd 2011 Evan Haruta Jerry Ng IBM Poughkeepsie [email protected] [email protected] SHARE Session August 2011 © 2011 IBM Corporation Trademarks The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. •MVS •OS/390® •z/Architecture® •z/OS® * Registered trademarks of IBM Corporation The following are trademarks or registered trademarks of other companies. Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation. UNIX is a registered trademark of The Open Group in the United States and other countries. SET and Secure Electronic Transaction are trademarks owned by SET Secure Electronic Transaction LLC. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non- IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. SHARE Session March 2011 2 © 2011 IBM Corporation Agenda What is the System Trace Table? How do you format it in a dump? How do you read the system trace entries? A recent enhancement SHARE Session March 2011 3 © 2011 IBM Corporation What is the MVS System Trace Table? Consists of trace buffers (one per CPU) residing in the Trace Address Space (ASID 4) System trace entries are inserted continuously into the trace buffers Contains detailed system activity Default size of 1 Meg per CPU Entries from trace buffers are merged in a dump SHARE Session March 2011 4 © 2011 IBM Corporation Why look at the System Trace? It contains the sequence of system events leading up to the problem It contains historical information that can be very useful for debugging It helps you to identify the source of the problem or confirm a failing scenario SHARE Session March 2011 5 © 2011 IBM Corporation How are system trace entries generated? By hardware, when certain instructions are issued: z Cross Memory instructions such as PC, PT, PR and SSAR z A branch is taken via a Branch and Link instruction such as BALR, BASR, BASSM, BAKR.... (if Branch Tracing is active) z Instructions causing switch of AMODE between 24/31 and 64 such as SAM (if Mode Tracing is active) By the operating system, when the TRACE/TRACG instruction is issued by system component modules SHARE Session March 2011 6 © 2011 IBM Corporation How can the System Trace options be changed? Via the TRACE operator command: z Stop or start system trace z Turn on Branch Tracing or Mode Tracing (or both) z Change the size of the trace table Note: System Trace is started during IPL with Branch and Mode Tracing off SHARE Session March 2011 7 © 2011 IBM Corporation How do I find the System Trace Table when debugging? The System Trace Table is dumped in SVC and Standalone dumps Use the IPCS subcommand SYSTRACE to format the System Trace Table in a dump SYSTRACE is documented in: z z/OS MVS IPCS Commands SHARE Session March 2011 8 © 2011 IBM Corporation Frequently used SYSTRACE parameters Current z Select entries of current address space (this is the default) All z Select all entries ASID(x,y....) z Select entries of specific address space(s) by ASID Jobname(name1,name2.....) z Select entries of specific address space(s) by jobname TCB(tcb_address) z Select entries of a TCB in an address space SHARE Session March 2011 9 © 2011 IBM Corporation More SYSTRACE parameters... CPU z Select entries of a specific CPU Time(z) z z = hex, local or GMT (default is hex) Exclude(x) z x = BR or MODE or BR,MODE z Exclude branch or mode trace entries, or both SHARE Session March 2011 10 © 2011 IBM Corporation Examples of using SYSTRACE SYSTRACE SYSTRACE ALL TIME(LOCAL) SYSTRACE ASID(X'5C') SYSTRACE ALL EX(BR,MODE) TIME(LOCAL) SYSTRACE ASID (X’1F’) TCB(X’9D2380’) TIME(LOCAL) SYSTRACE CPU(6) ALL SHARE Session March 2011 11 © 2011 IBM Corporation Reading the System Trace IPCS OUTPUT STREAM ------------------------------------------------------------------------------------------ Command ===> ************************************************************ TOP OF DATA *********************************** --------------------------------------------------- SYSTEM TRACE TABLE ------------------------------------- -- ------------------ -- ------------------ PR ASID WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3 PSACLHS- PSALOCAL PASD SASD TIME UNIQUE-4 UNIQUE-5 UNIQUE-6 PSACLHSE 00 0021 009FE030 DSP 070C1000 814C9DDE 00000000 814C9DDE 00FB3788 00000000 00000000 0021 0021 23:0 00 0021 009FE030 SVC 4F 070C1000 814CA396 009FDCB0 00000011 FFFFFFEF Status Start SRBs only 23:0 00 0021 009FE030 SVCR 4F 070C1000 814CA396 00000000 00000000 05616F00 23:0 02 0165 009FFB00 DSP 070C4000 8137DA1C 00800000 00000001 05882000 00000000 00000000 0049 0165 23:0 02 0165 009FFB00 PR ... 0 09570126 29A0530E 0165 02 0165 009FFB00 PC ... 0 095701CE 01F01 02 0165 009FFB00 PR ... 0 095701CE 29A06B5A 0165 00 0010 009CEAD0 SSRV 132 00000000 3750E5C0 00004000 7E0EC000 Storage Obtain 23:0 00100000 Entries are presented chronologically and intermixed Oldest entry at the top Newest entry at the bottom SHARE Session March 2011 12 © 2011 IBM Corporation Reading the System Trace: who did it? IPCS OUTPUT STREAM ------------------------------------------------------------------------------------------ Command ===> ************************************************************ TOP OF DATA *********************************** --------------------------------------------------- SYSTEM TRACE TABLE ------------------------------------- -- ------------------ -- ------------------ PR ASID WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3 PSACLHS- PSALOCAL PASD SASD TIME UNIQUE-4 UNIQUE-5 UNIQUE-6 PSACLHSE 00 0021 009FE030 DSP 070C1000 814C9DDE 00000000 814C9DDE 00FB3788 00000000 00000000 0021 0021 23:0 00 0021 009FE030 SVC 4F 070C1000 814CA396 009FDCB0 00000011 FFFFFFEF Status Start SRBs only 23:0 00 0021 009FE030 SVCR 4F 070C1000 814CA396 00000000 00000000 05616F00 23:0 02 0165 009FFB00 DSP 070C4000 8137DA1C 00800000 00000001 05882000 00000000 00000000 0049 0165 23:0 02 0165 009FFB00 PR ... 0 09570126 29A0530E 0165 02 0165 009FFB00 PC ... 0 095701CE 01F01 02 0165 009FFB00 PR ... 0 095701CE 29A06B5A 0165 00 0010 009CEAD0 SSRV 132 00000000 3750E5C0 00004000 7E0EC000 Storage Obtain 23:0 00100000 PR (processor id) ASID WU-ADDR ( TCB mode: TCB address / SRB mode: WEB address or zero) PSW SHARE Session March 2011 13 © 2011 IBM Corporation Reading the System Trace: what was done? IPCS OUTPUT STREAM ------------------------------------------------------------------------------------------ Command ===> ************************************************************ TOP OF DATA *********************************** --------------------------------------------------- SYSTEM TRACE TABLE ------------------------------------- -- ------------------ -- ------------------ PR ASID WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3 PSACLHS- PSALOCAL PASD SASD TIME UNIQUE-4 UNIQUE-5 UNIQUE-6 PSACLHSE 00 0021 009FE030 DSP 070C1000 814C9DDE 00000000 814C9DDE 00FB3788 00000000 00000000 0021 0021 23:0 00 0021 009FE030 SVC 4F 070C1000 814CA396 009FDCB0 00000011 FFFFFFEF Status Start SRBs only 23:0 00 0021 009FE030 SVCR 4F 070C1000 814CA396 00000000 00000000 05616F00 23:0 02 0165 009FFB00 DSP 070C4000 8137DA1C 00800000 00000001 05882000 00000000 00000000 0049 0165 23:0 02 0165 009FFB00