Process Thread Stack

Customer Customer Coffee Corner for SAP IQ – OS Health Checklist for IQ SAP Product Support October, 2016 Agenda

 Objectives  What is OS health checklist for IQ ?  Choosing profiler of interest  OS diagnostics collection tool demo ( Various )  Tips  Closing remarks  Open discussion

• Proactive outreach based on feedbacks

• Target audience – IQ DBAs Novice/Beginner

• IQ OS health checklist awareness which may help in proactive diagnostic collection and better root cause analysis of IQ incidents.

 OS health checklist is using various well known profilers which may provide insight in performance issues of any product including IQ.

 General issues with OS health checklist collection – Each profiler has its own variation. o Different output style for different command line flag o Ability/Inability to collect diagnostic snapshot at predefined intervals o Different OS may have different profiler for same functionality o Profiler output may or may not contain timestamp of snapshot – Logistically it is difficult to remember o Various profilers o Various permutations and combinations of profiler command line flags o To collect certain number of diagnostics samples at predefined intervals o To timestamp diagnostic files

 “OS collection diagnostic” tool standardizes all of the above which may help in collecting diagnostic snapshot o when the problem is occurring before even opening an incident o when the problem in NOT occurring as well, which helps in comparing work/fail scenarios

 Profiler objective o Profiler links for every platform for easy reference o Performance impact of profiler on target ( Stop / Slowdown / No impact ) o Preferred execution syntax on Linux platform for easy reference, along with flag cheat sheet

 End user considerations between test window T1-T2 o Profiler output naming convention - .MMDDYYHHMMSS o If profiler cannot take predefined number of snapshots at predefined time interval, end user is advised to take it. o End user is responsible to set test window so that number of profiler files and sizes are under control

 Tips on reading profiler output, KPIs and red flags.

 Which tool or strategy is best suited for reading profiler output.

 Attach to active process and print out execution threads stacks o Linux gstack , Sparc pstack , HP pstack , AIX procstack o Target process halts while thread stack is getting printed o gstack 12345 > gstack.12345.102716102030

 Between time window T1-T2 o End-user advised to take at least 3 snapshots at 3-5 minutes apart. eg, gstack.102820161030 , gstack.102820161034 , gstack.102820161038

 Notepad++ , diff o Compare 2 files for no change ( Hang ) , only few threads moving ( Bottleneck )

 No difference => Hang. Minor difference => bottleneck

 Monitoring system IO device loads o Linux iostat , Sparc iostat , HP iostat , AIX iostat o No impact on target process o iostat -d -x -z -m 10 > iostat.102716102030 -d Display the device utilization report -x Display extended statistics -z omit output for any devices for which there was no activity during the sample period -m Display statistics in megabytes per second 10 10 seconds apart

 Between time window T1-T2 o End-user advised to take multiple snapshots at 10 seconds apart.

 High service time can indicate IO bottleneck.  Sample output

$ iostat -d -x -z -m 2 3 > iostat.102716102030 ; cat iostat.102716102030 Linux 3.10.0-327.36.1.el7.x86_64 (oakl00604022a) 10/26/2016 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.11 0.04 0.01 0.00 134.48 0.00 31.69 5.40 110.96 6.19 0.09 sda 0.00 1.89 0.09 3.63 0.00 0.05 25.77 0.04 11.26 4.71 11.43 4.81 1.79 dm-0 0.00 0.00 0.11 0.04 0.01 0.00 130.93 0.01 34.98 5.42 115.19 6.03 0.09 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 12.01 0.00 14.31 31.66 2.16 13.31 0.00 dm-2 0.00 0.00 0.09 5.25 0.00 0.05 17.97 0.06 11.40 6.21 11.50 3.35 1.79

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.00 0.00 1.00 0.00 0.06 132.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 5.50 0.00 0.01 3.64 0.08 16.36 0.00 16.36 14.09 7.75 dm-0 0.00 0.00 0.00 1.00 0.00 0.06 132.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 5.00 0.00 0.01 4.00 0.08 18.00 0.00 18.00 15.50 7.75

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 2.50 0.00 3.00 0.00 0.02 14.67 0.02 7.83 0.00 7.83 7.83 2.35 dm-2 0.00 0.00 0.00 5.00 0.00 0.02 8.80 0.02 4.70 0.00 4.70 4.70 2.35  Custom awk/perl script to read desired KPI for desired device above red flag of 10

$ awk '{ if ( ( $1 ~ /dm-2/ ) && ( $13 > 10 ) ) print $13 } ' iostat.102716102030

 Trace system calls and signals o Linux strace , Sparc truss , HP tusc , AIX truss o Target process slows down o strace –o strace.102716102030 -f -r -T -tt -s 128 –p -f Trace child processes -r Print a relative timestamp upon entry to each system call -T Show the time spent in system calls. -tt If given twice, the time printed will include the microseconds. -s strsize Specify the maximum string size to print  Between time window T1-T2 o End-user advised to limit time window & use such profiler only when problem occurs. o Potentially can generate huge file causing big transfer and incident attachment issues.  Custom awk/perl script to examine thread of interest OR error code or specific system call

 Look for o system call return error codes. Eg. ENOACCESS , ETIMEDOUT o number of times particular function call gets executed o Extraordinary time taken by some system call especially related to IO

 OS diagnostic collection tool o Many more profilers incorporated o Can help collecting OS diagnostics for RCA in a standardized manner o Takes care of various end user considerations while running multiple profilers o Location – collect_OS_Diagnostics.sh o Syntax – collect_OS_Diagnostics.sh o Example - collect_OS_Diagnostics.sh 69121 /tmp/sapIQsupport 300 Y o Disclaimer – AS IS BASIS , UNIX ONLY

 Demo o Collecting set for 15 minutes o Collecting only strace o Collecting only pstack

 Roadmap o Recognize & filter IQ devices while collecting iostat o Integrate IQ key stored procedure output snapshots in collection time window

 Before tool use, Check for profiler availability and command line flags used in the tool.  iostat o Iostat will list all devices. IQ devices are of interest. Relating file with a device on Linux can be a challenge.

Here is one way. Find directory ( pwd ) where file resides Run ( df –h ) Run ( ls –l /dev/mapper )

This example shows iqdemo.db resides on device dm-0

$ pwd /work2/IQ16SP11_spx/N0 $ df -h /work2/IQ16SP11_spx/N0 Filesystem Size Used Avail Use% Mounted on /dev/mapper/workdg-work2 1.4T 1.1T 186G 86% /work2 $ ls -l /dev/mapper | grep /dev/mapper/workdg-work2 lrwxrwxrwx 1 root root 7 Oct 14 15:29 workdg-work2 -> ../dm-0

1843189 – SAP Sybase IQ Troubleshooting and diagnostics collection checklist

IQ Diagnostic files wiki

2309381 – Customer Virtual Coffee corner for ASE, IQ, Replication Server, Software Developers Kit … 2137179 – Customer Coffee Corner for SAP IQ – Americas

2373124 How to display IQ database collations and charset – SAP IQ 2379181 [SAP IQ] SAP IQ16 installation failed due to "Unsupported major.minor version 52.0" Error. 2379881 The meaning of Flexible and Inflexible memory in sp_iqstatus and sp_iqsysmon.

 What’s next

 Encourage to use “Collect OS Diagnostics” tool & experience it yourself.

 Please provide your feedback to IQ VCC coordinators on – Did you learn something new/useful ? – Did this outreach help understanding OS health checklist better ?