Linux on IBM Z and LinuxONE How to troubleshoot July 16, 2020 — Sa Liu Linux on IBM Z and LinuxONE Service & Support Trademarks The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. CICS* Global Business Services* MQ* SPSS* XIV* z/VSE* Cognos* IBM* Parallel Sysplex* System Storage* zEnterprise* DataStage* IBM (logo)* QualityStage System x* z/OS* DB2* InfoSphere Rational* Tivoli* z Systems* GDPS Maximo* Smarter Cities WebSphere* z/VM* * Registered trademarks of IBM Corporation Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a Registered Trade Mark of AXELOS Limited. ITIL is a Registered Trade Mark of AXELOS Limited. Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. UNIX is a registered trademark of The Open Group in the United States and other countries. VMware, the VMware logo, VMware Cloud Foundation, VMware Cloud Foundation Service, VMware vCenter Server, and VMware vSphere are registered trademarks or trademarks of VMware, Inc. or its subsidiaries in the United States and/or other jurisdictions. Other product and service names might be trademarks of IBM or other companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non- IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. This information provides only general descriptions of the types and portions of workloads that are eligible for execution on Specialty Engines (e.g, zIIPs, zAAPs, and IFLs) ("SEs"). IBM authorizes customers to use IBM SE only to execute the processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for IBM Machines” provided at www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”). No other workload processing1 is authorized for execution on an SE. IBM offers SE at a lower price than General Processors/Central Processors because customers are authorized to use SEs only to process certain types and/or amounts of workloads as specified by IBM in the AUT. INTRODUCTION DATA STORAGE NETWORK PERFORMANCE CUSTOMER COLLECTION TROUBLESHOOTING TROUBLESHOOTING TROUBLESHOOTING CASES Agenda 2 Linux on Z troubleshooting § Understand the problem § What are the symptoms of the problem? § Where / When / Under which condition does the problem occur? § Can the problem be reproduced? § What do you need to do? § Collect data before recovery § Collect data right after the problem occurs § Collect data from a healthy system § Keep track of the system setup and the latest changes 3 Linux on Z troubleshooting § How to do it? § Get the system prepared for data collection § Install packages for Linux tools (s390-tools, sysstat, perf …) § Enable Linux kdump / have disks ready for standalone dump § Enable regular system activity monitoring § Learn to use the tools for data analysis 4 § System data § Performance data Data Collection § System dump 5 Collect System Data § dbginfo.sh § sosreport - RHEL § supportconfig - SLES § sosreport - Ubuntu 6 Collect System Data § dbginfo.sh – a script that collects data for debugging Linux on Z (requires root authority ) § It collects: § System information and generic configuration data Run before reboot! § List of devices and their configurations § System logs / trace data Distribution Package name § s390 debug buffer RHEL s390util § z/VM or KVM basic data SLES s390-tools (if Linux runs under z/VM or KVM) Ubuntu s390-tools 7 dbginfo.sh output root@system: # dbginfo.sh dbginfo.sh: Debug information script version 2.11.0-7.27 Copyright IBM Corp. 2002, 2018 Hardware platform = s390x Kernel version = 5.3.18 (5.3.18-22-default) Runtime environment = LPAR root@system # cd /tmp/DBGINFO-2020-07-15-08-44-18-test-1CA1E7/ s8315028:/tmp/DBGINFO-2020-07-15-08-44-18-test-1CA1E7 1 of 13: Collecting command output root@system # ls -t 2 of 13: Collecting z/VM command dbginfo.log etc sysfs.tgz 3 of 13: Collecting procfs osa_oat_eth3.raw usr proc 4 of 13: Collecting sysfs osa_oat.out lib zvm_runtime.out 5 of 13: Collecting log files osa_oat_eth1.raw boot runtime.out 6 of 13: Collecting config files osa_oat_eth2.raw run journalctl.out 7 of 13: Collecting osa oat output osa_oat_eth0.raw var 8 of 13: Collecting ethtool output 9 of 13: Collecting OpenVSwitch output skipped 10 of 13: Collecting domain xml files skipped 11a of 13: Collecting docker container output skipped Linux commands output z/VM commands output 11b of 13: Collecting docker network output skipped 12 of 13: Collecting nvme output 13 of 13: Postprocessing Finalizing: Creating archive with collected data Make sure you have enough disk space under /tmp Collected data was saved to: >> /tmp/DBGINFO-2020-07-15-08-44-18-test-1CA1E7.tgz << Use “dbginfo.sh -d <directory>” to specify another location for the tarball Review the collected data before sending to your service 8 organization. Collect Performance Data § sadc – system activity data collector § perf – performance analysis tool § iostat – monitors I/O device load and the CPU utilization § dasdstat – display DASD performance data § ziomon / ziorep – collect FCP performance data and generate reports § z/VM MONWRITE – collects CP *MONITOR data § hyptop – dynamic real-time view of hypervisor For more details refer to the book Troubleshooting Guide 9 Collect Performance Data § sadc – system activity data collector (sysstat package) root@system: #/usr/lib64/sa/sadc -S XALL 10 sadc_output § Collect all counters at 10 second interval to a binary output file § sar – system activity report root@system: # sar -A -f sadc_output > sar_output § Convert the binary output file to a plain text report § Start sysstat service as a permanent service (recommended) § sadc default configuration at 10-minute intervals (/var/log/sa/) § Under z/VM, collect MONWRITE data in the same time period and same interval as sadc data! 10 Collect Performance Data § kSar – graphical sar analysis tool 11 Collect Performance Data § Recommended data collection process § 1) run dbginfo.sh # /sbin/dbginfo.sh § 2) start sadc at 5 second interval # /usr/lib64/sa/sadc -S XALL 5 /tmp/server_sadc.out & § 3) run the test with workload § 4) stop the sadc with # killall sadc § 5) run dbginfo again # /sbin/dbginfo.sh § 6) convert sadc output file to a report # sar -A -f /tmp/server_sadc.out > server_sar 12 Collect System Dump For more details refer to the book Using the Dump Tools 13 kdump – Kernel Crash Dump § To boot a new instance of the kernel in a pre- reserved memory section § To copy the existing memory untouched to storage or via network § kexec tool § crashkernel=auto ( minimum amount of memory required: 4GB) 14 Configure the kdump on RHEL § Install kdump service # yum install