Customer Customer Coffee Corner for SAP IQ – OS Health Checklist for IQ SAP Product Support October, 2016 Agenda Objectives What is OS health checklist for IQ ? Choosing profiler of interest OS diagnostics collection tool demo ( Various ) Tips Closing remarks Open discussion © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 2 Objectives • Proactive outreach based on feedbacks • Target audience – IQ DBAs Novice/Beginner • IQ OS health checklist awareness which may help in proactive diagnostic collection and better root cause analysis of IQ incidents. © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 3 What is OS health checklist for IQ ? OS health checklist is using various well known profilers which may provide insight in performance issues of any product including IQ. General issues with OS health checklist collection – Each profiler has its own variation. o Different output style for different command line flag o Ability/Inability to collect diagnostic snapshot at predefined intervals o Different OS may have different profiler for same functionality o Profiler output may or may not contain timestamp of snapshot – Logistically it is difficult to remember o Various profilers o Various permutations and combinations of profiler command line flags o To collect certain number of diagnostics samples at predefined intervals o To timestamp diagnostic files “OS collection diagnostic” tool standardizes all of the above which may help in collecting diagnostic snapshot o when the problem is occurring before even opening an incident o when the problem in NOT occurring as well, which helps in comparing work/fail scenarios © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 4 Choosing profiler of interest Profiler objective o Profiler links for every platform for easy reference o Performance impact of profiler on target ( Stop / Slowdown / No impact ) o Preferred execution syntax on Linux platform for easy reference, along with flag cheat sheet End user considerations between test window T1-T2 o Profiler output naming convention - <profiler_name>.MMDDYYHHMMSS o If profiler cannot take predefined number of snapshots at predefined time interval, end user is advised to take it. o End user is responsible to set test window so that number of profiler files and sizes are under control Tips on reading profiler output, KPIs and red flags. Which tool or strategy is best suited for reading profiler output. © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 5 Process thread stack Attach to active process and print out execution threads stacks o Linux gstack , Sparc pstack , HP pstack , AIX procstack o Target process halts while thread stack is getting printed o gstack 12345 > gstack.12345.102716102030 Between time window T1-T2 o End-user advised to take at least 3 snapshots at 3-5 minutes apart. eg, gstack.102820161030 , gstack.102820161034 , gstack.102820161038 Notepad++ , diff o Compare 2 files for no change ( Hang ) , only few threads moving ( Bottleneck ) No difference => Hang. Minor difference => bottleneck © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 6 Cpu and IO statistics for devices Monitoring system IO device loads o Linux iostat , Sparc iostat , HP iostat , AIX iostat o No impact on target process o iostat -d -x -z -m 10 > iostat.102716102030 -d Display the device utilization report -x Display extended statistics -z omit output for any devices for which there was no activity during the sample period -m Display statistics in megabytes per second 10 10 seconds apart Between time window T1-T2 o End-user advised to take multiple snapshots at 10 seconds apart. © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 7 Cpu and IO statistics for devices High service time can indicate IO bottleneck. Sample output $ iostat -d -x -z -m 2 3 > iostat.102716102030 ; cat iostat.102716102030 Linux 3.10.0-327.36.1.el7.x86_64 (oakl00604022a) 10/26/2016 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.11 0.04 0.01 0.00 134.48 0.00 31.69 5.40 110.96 6.19 0.09 sda 0.00 1.89 0.09 3.63 0.00 0.05 25.77 0.04 11.26 4.71 11.43 4.81 1.79 dm-0 0.00 0.00 0.11 0.04 0.01 0.00 130.93 0.01 34.98 5.42 115.19 6.03 0.09 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 12.01 0.00 14.31 31.66 2.16 13.31 0.00 dm-2 0.00 0.00 0.09 5.25 0.00 0.05 17.97 0.06 11.40 6.21 11.50 3.35 1.79 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.00 1.00 0.00 0.06 132.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 5.50 0.00 0.01 3.64 0.08 16.36 0.00 16.36 14.09 7.75 dm-0 0.00 0.00 0.00 1.00 0.00 0.06 132.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 5.00 0.00 0.01 4.00 0.08 18.00 0.00 18.00 15.50 7.75 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 2.50 0.00 3.00 0.00 0.02 14.67 0.02 7.83 0.00 7.83 7.83 2.35 dm-2 0.00 0.00 0.00 5.00 0.00 0.02 8.80 0.02 4.70 0.00 4.70 4.70 2.35 Custom awk/perl script to read desired KPI for desired device above red flag of 10 $ awk '{ if ( ( $1 ~ /dm-2/ ) && ( $13 > 10 ) ) print $13 } ' iostat.102716102030 © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 8 Trace system calls and signals Trace system calls and signals o Linux strace , Sparc truss , HP tusc , AIX truss o Target process slows down o strace –o strace.102716102030 -f -r -T -tt -s 128 –p <PID> -f Trace child processes -r Print a relative timestamp upon entry to each system call -T Show the time spent in system calls. -tt If given twice, the time printed will include the microseconds. -s strsize Specify the maximum string size to print Between time window T1-T2 o End-user advised to limit time window & use such profiler only when problem occurs. o Potentially can generate huge file causing big transfer and incident attachment issues. Custom awk/perl script to examine thread of interest OR error code or specific system call Look for o system call return error codes. Eg. ENOACCESS , ETIMEDOUT o number of times particular function call gets executed o Extraordinary time taken by some system call especially related to IO © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 9 OS diagnostics collection tool demo ( Various ) OS diagnostic collection tool o Many more profilers incorporated o Can help collecting OS diagnostics for RCA in a standardized manner o Takes care of various end user considerations while running multiple profilers o Location – collect_OS_Diagnostics.sh o Syntax – collect_OS_Diagnostics.sh <SERVER_PID> <Output_Directory> <Sample_Duration_In_Seconds> <strace_Yes_or_No> o Example - collect_OS_Diagnostics.sh 69121 /tmp/sapIQsupport 300 Y o Disclaimer – AS IS BASIS , UNIX ONLY Demo o Collecting set for 15 minutes o Collecting only strace o Collecting only pstack Roadmap o Recognize & filter IQ devices while collecting iostat o Integrate IQ key stored procedure output snapshots in collection time window © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 10 Tips Before tool use, Check for profiler availability and command line flags used in the tool. iostat o Iostat will list all devices. IQ devices are of interest. Relating file with a device on Linux can be a challenge. Here is one way. Find directory ( pwd ) where file resides Run ( df –h ) Run ( ls –l /dev/mapper ) This example shows iqdemo.db resides on device dm-0 $ pwd /work2/IQ16SP11_spx/N0 $ df -h /work2/IQ16SP11_spx/N0 Filesystem Size Used Avail Use% Mounted on /dev/mapper/workdg-work2 1.4T 1.1T 186G 86% /work2 $ ls -l /dev/mapper | grep /dev/mapper/workdg-work2 lrwxrwxrwx 1 root root 7 Oct 14 15:29 workdg-work2 -> ../dm-0 © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 11 KBA’s - specific to today’s topic 1843189 – SAP Sybase IQ Troubleshooting and diagnostics collection checklist IQ Diagnostic files wiki © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 12 KBA’s - product specific 2309381 – Customer Virtual Coffee corner for ASE, IQ, Replication Server, Software Developers Kit … 2137179 – Customer Coffee Corner for SAP IQ – Americas 2373124 How to display IQ database collations and charset – SAP IQ 2379181 [SAP IQ] SAP IQ16 installation failed due to "Unsupported major.minor version 52.0" Error. 2379881 The meaning of Flexible and Inflexible memory in sp_iqstatus and sp_iqsysmon. © 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 13 Closing Remarks What’s next Encourage to use “Collect OS Diagnostics” tool & experience it yourself. Please provide your feedback to IQ VCC coordinators on – Did you learn something new/useful ? – Did this outreach help understanding OS health checklist better ? © 2016 SAP SE or an SAP affiliate company.