I/O Calibration Performance and Tuning with Linux on System Z

I/O Calibration Performance and Tuning with Linux on System z

David Simpson - Oracle Technical Specialist, IBM ( [email protected] )

For IBM – can be found at http://www.ibm.com/legal/us/en/copytrade.shtml For Oracle – can be found at http://www.oracle.com/us/legal/index.html

Any performance results/observations in this presentation are purely for education and planning purposes. No Test results should be construed as indicative of any particular customer workload or benchmark result.

Normalized Transactional Throughput 160% 140% 120% 100% 80% 60% 40% 20% 0% 4 CPUs 6 CPUs

FICON (20 aliases) FCP (rr_min_io=100)

Normalized CPU cost per transaction 120%

100%

80%

60%

40%

20%

0% CPU per CPU throughput costtransactional 4 CPUs 6 CPUs

HyperPAV allows multiple IO operations on the same sub channel Very important for random access workload with relative small data transfers 10-20 HyperPAV aliases per LCU showed best performance gains in our internal tests.

Recommendation: – Enable HyperPAV whenever using ECKD devices

lsdasd –u 0.0.75bc dasda IBM.750000000NA461.1555.bc.000000000000eac50000000000000000 0.0.75ca dasdk IBM.750000000NA461.1555.ca.000000000000eac50000000000000000 0.0.75d0 alias IBM.750000000NA461.1555.xx.00000000000000000000000000000000 0.0.75d1 alias IBM.750000000NA461.1555.xx.00000000000000000000000000000000

ECKD Devices: Scaling HyperPAV aliases Normalized Transactional throughput and total Disk I/O (read + write)

160% 120 140% 100 120% 100% 80 80% 60 MiB/sec 60% 40 40% 20% 20 normalized transactional throughput transactional normalized 0% 0 0 10 20 40 # alias devices

normalized transactional throughput Total MB/sec

FCP/SCSI SAN Storage offers better throughput ECKD/FICON SAN Storage uses less CPU per transaction You have to tune both environments

Recommendation: it depends

Red Hat 6 / SUSE 11:

Red Hat 7.1+:

SUSE 12 SP1+

/etc/multipath.conf parameters can impact your FCP/SCSI performance particularly with I/O bound workloads: Check with your OS distributor & Storage vendors for recommended settings rr_min_io_rq=1 #Number of I/Os per Path (Newer Linux Kernels) SLES 11 SP3+, RH 6.x+ #rr_min_io #Use for Older Linux Distros (Red Hat 5.x,SLES 11sp1 & older) path_selector "service-time 0" #default for SLES 11 SP3+, RedHat 6.7+ #path_selector "round-robin 0" #round-robin for SLES 11 SP2+ and RedHat 6.x path_grouping_policy "multibus" # SLES 11 SP1+ and Red Hat 6.x+ # path_grouping_policy "failover" # SLES 10, Red Hat 5.x

SUSE 12: /etc/multipath.conf:

To make changes in effect dynamically: systemctl restart multipathd.service

The Linux 2.6 kernel offers a choice of four different I/O schedulers: – Noop Scheduler (noop) – Deadline Scheduler (deadline) – Anticipatory Scheduler (as) – Complete Fair Queuing Scheduler (cfq) – Default for a lot of kernels

Some Linux kernels default is the “cfq” scheduler: – Designed to optimize access to physical disks – Check in /sys/block//queue/scheduler noop anticipatory [deadline] cfq – Default configurable by setting the “elevator=[…]” boot parameter in /etc/zipl.conf

Recommendation: deadline or noop for Oracle database LUNs

© 2017 IBM Corporation 16 Changing I/O scheduler with UDEV Rule I/O settings you may want to consider adjusting (based on storage vendor recommendations): scheduler I/O scheduler (noop or deadline) nr_requests number of I/O requests default 128 (may want to increase to 1024 for Flash Storage devices) rotational 1 default for spinning disk, 0 for Flash nomerges default 0, 1 for Flash max_sectors_kb default 1024, 4096 for Flash rq_infinity default 1, 2 for Flash add_random default 1, 0 for Flash

Sample UDEV rules (/etc/udev/rules.d) /etc/udev/rules.d/12-dm-permissions.rules (Multipath Devices) ENV{DM_NAME}=="ASM*",OWNER:="oracle",GROUP:="dba",MODE:="660",RUN+="/bin/sh -c 'echo deadline > /sys/$devpath/queue/scheduler '"

/etc/udev/rules.d/99-udev-oracle.rules (Single Path Devices) KERNEL=="sda",OWNER="oracle",GROUP="dba",MODE="660",RUN+="/bin/sh -c ‘echo deadline > /sys/$devpath/queue/scheduler'“

Restarting UDEV rules (Sles 11 sp3/Red Hat 6): udevadm control --reload-rules udevadm trigger

/etc/multipath.conf: multipath { wwid 20020c2400d1221a8 alias ASMFLASH0D }

# multipath -ll ASMFLASH0D ASMFLASH0D (20020c2400d1221a8) dm-21 IBM, FlashSystem size=256G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 5:0:1:13 sdbv 68:144 active ready running |- 7:0:1:13 sdep 129:16 active ready running `- 7:0:0:13 sdfd 129:240 active ready running

[root@zlnxl2 mapper]# ls -lato /dev/mapper/ ASMFLASH0D lrwxrwxrwx 1 root 8 May 13 16:46 ASMFLASH0D -> ../dm-21

[root@zlnxl2 rules.d]# ls -la /dev/dm-21 brw-rw---- 1 root disk 253, 21 May 13 14:54 /dev/dm-21

[root@zlnxl2 rules.d]# udevadm trigger [root@zlnxl2 rules.d]# ls -la /dev/dm-21

LVM – Logical Volume Manager in Linux ASM – Automated Storage Management provided by Oracle – Oracle RAC One and Oracle RAC will require ASM

LVM ASM pro • Direct control on setting and • Automated, out of the box layout environment • Can choose file system • Very good integration with Oracle con • Complex setup • RMAN required for backup

Overall recommendation: ASM Don’t combine both! 19 © 2017 IBM Corporation File System Recommendations SLES 11 SP4+ Supported and Recommended File ext3 xfs Systems on Linux (MOS: 236826.1 ) SUSE xfs for DB files (24.7% increase and 12.9% increase for Reads (NON ASM)

Red Hat 6.x ext4 for database files

Red Hat 7.x - XFS is now recommended.

XFS – difficult to reduce the size of a LVM.

Oracle parameter file systems: filesystemio_options=setall – Provides asynchronous & direct I/O (avoids linux file system cache)

Reduce Linux Read-Ahead for LVM file systems. – lvchange -r none

Consolidated several systems to one system Transaction times > 6 seconds for 100 Transactions per seconds (Normally sub 2 seconds) after migration.

SQL ID: 4ztz048yfq32s: SELECT TO_CHAR(current_timestamp AT TIME ZONE 'GMT', 'YYYY-MM-DD HH24:MI:SS TZD') AS curr_timestamp, COUNT(username) AS failed_count, TO_CHAR(MIN(timestamp), 'yyyy-mm-dd hh24:mi:ss') AS first_occur_time, TO_CHAR(MAX(timestamp), 'yyyy-mm-dd hh24:mi:ss') AS last_occur_time FROM sys.dba_audit_session WHERE returncode != 0 AND timestamp >= current_timestamp - TO_DSINTERVAL('0 0:30:00')

Query 4ztz048yfq32s is an audit check –many failed logon attempts an alert is generated. Slow Performance Of DBA_AUDIT_SESSION Query From "Failed logins" Metric in Enterprise Manager (Doc ID 1178736.1) - Bug 7633167 : SQL PERFORMANCE ISSUE IN DBA_AUDIT_SESSION.

Oracle Note:73408.1 "How to Truncate, Delete, or Purge Rows from the Audit Trail Table AUD$" Be aware that it may be a company requirement to keep a backup of the rows in this table. Ensure that a backup is taken before truncating/deleting as discussed in Note:73408.1

Note:731908.1 "New Feature DBMS_AUDIT_MGMT To Manage And Purge Audit Information"

Bug is fixed in the 12.1.0.5 database plugin. See note:1388143.1 "How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud Control" for instructions on deploying the latest database plug-in.

Apply the latest DB Plugin PSU Patches: Patch:17998764 OMS-Side Patch:18039625 and Patch:18012965 Agent-Side

Finding 2: Undersized SGA Impact is 34.3 active sessions, 56.94% of total activity. ------The SGA was inadequately sized, causing additional I/O or hard parses. The value of parameter "sga_target" was " 33792 M " during the analysis period.

Recommendation 1: Database Configuration Estimated benefit is 34.3 active sessions, 56.94% of total activity. ------Action Increase the size of the SGA by setting the parameter "sga_target" to 59136 M. 26 © 2017 IBM Corporation Linux Large Pages:

Looks like about – 6GB of Large Page memory not in use Based on +26GB recommendation on previous page, suggest +20GB for Linux Guest Increase /etc/sysctl.conf - vm.nr_hugepages to 30000

ziorep_config =>

Slow I/O outside of the mainframe, i.e. in the SAN switched fabric or in the storage. ( 2.7sec fabric latency for one single request).

Linux guest are not scheduled often enough (steal time). – servicing I/O responses from hardware, with response gets "late" timestamps and I/O looks erroneously slow. – lots of pending I/O, scsi_errors can take very long to finally complete.

Have a tool running all the time for historical / analytical purposes. For performance issues log an SR with Oracle & IBM PMR and other vendors as needed let each part know for collaboration. ioscript.sh: nohup bash -c "(ziomon -i 10 -d 15 -o ziomon-ibm$(date --rfc-3339=date)$(hostname -a) /dev/mapper/ASMSAN00)" &> nohup.ziomon & nohup bash -c "(/usr/lib64/sa/sadc -S ALL 10 90 sadc-$(hostname -a) ; sar -f sadc-$(hostname -a) -A > sar-$(date --rfc-3339=date)-$(hostname -a))" &> nohup.sa &

Then for each run... mkdir run2 ../ioscript.sh cd .. tar -czvf run2.tar.gz run2

SQL> SET SERVEROUTPUT ON DECLARE lat INTEGER; iops INTEGER; mbps INTEGER; BEGIN -- DBMS_RESOURCE_MANAGER.CALIBRATE_IO (, , iops, mbps, lat); DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);

DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops); DBMS_OUTPUT.PUT_LINE ('latency = ' || lat); dbms_output.put_line('max_mbps = ' || mbps); end; / max_iops = xxxxxx latency = 0 max_mbps = yyyy

1200 I/O Calibrate %Improvement vs # of I/O Paths

1000

800

600 IOPs MB/s % Improvement % 400

200

0 1 2 4 12 12 12 # of I/O Paths & rr_min_io_rq=1, 100, 4

Baseline %Improve Recommendation IOPs 0 12.98 38.39 575.59 713.32 684.47 MB/s 0 95.01 197.92 1029.96 529.82 994.45 I/O Paths 1 2 4 12 12 12 Multipath rr_min_io_rq=1 rr_min_io_rq=1 rr_min_io_rq=1 rr_min_io_rq=1 rr_min_io_rq=100 rr_min_io_rq=4

After switching to FlashSystem Disk IO wait disappears and waiting is now on host CPU. This graph shows the effect of the low latency of FlashSystem and how it increases the host CPU utilization. © 2017 IBM Corporation Questions?