I/O Calibration Performance and Tuning with Linux on System z David Simpson - Oracle Technical Specialist, IBM ( [email protected] ) © 2017 IBM Corporation Copyright and Trademark Information For IBM – can be found at http://www.ibm.com/legal/us/en/copytrade.shtml For Oracle – can be found at http://www.oracle.com/us/legal/index.html Any performance results/observations in this presentation are purely for education and planning purposes. No Test results should be construed as indicative of any particular customer workload or benchmark result. © 2017 IBM Corporation © 2017 IBM Corporation IBM z13 – 5 I/O Drawers with up to 32 I/O Cards per drawer 4 © 2017 IBM Corporation © 2017 IBM Corporation Oracle Cache, Memory and I/O Access © 2017 IBM Corporation FICON (ECKD) / FCP comparison (1) Normalized Transactional Throughput 160% 140% 120% 100% 80% 60% 40% 20% 0% 4 CPUs 6 CPUs FICON (20 aliases) FCP (rr_min_io=100) 7 Source: Juergen Doelle http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102234 © 2017 IBM Corporation FICON (ECKD) / FCP comparison (2) Normalized CPU cost per transaction 120% 100% 80% 60% 40% 20% 0% CPU per CPU costthroughput transactional 4 CPUs 6 CPUs FICON (20 aliases) FCP (rr_min_io=100) 8 Source: Juergen Doelle http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102234 © 2017 IBM Corporation HyperPAV (1) HyperPAV allows multiple IO operations on the same sub channel Very important for random access workload with relative small data transfers 10-20 HyperPAV aliases per LCU showed best performance gains in our internal tests. Recommendation: – Enable HyperPAV whenever using ECKD devices lsdasd –u 0.0.75bc dasda IBM.750000000NA461.1555.bc.000000000000eac50000000000000000 0.0.75ca dasdk IBM.750000000NA461.1555.ca.000000000000eac50000000000000000 0.0.75d0 alias IBM.750000000NA461.1555.xx.00000000000000000000000000000000 0.0.75d1 alias IBM.750000000NA461.1555.xx.00000000000000000000000000000000 9 © 2017 IBM Corporation HyperPAV (2) ECKD Devices: Scaling HyperPAV aliases Normalized Transactional throughput and total Disk I/O (read + write) 160% 120 140% 100 120% 100% 80 80% 60 MiB/sec 60% 40 40% 20% 20 normalized transactional throughput transactional normalized 0% 0 0 10 20 40 # alias devices normalized transactional throughput Total MB/sec 10 © 2017 IBM Corporation FICON (ECKD) / FCP comparison (3) FCP/SCSI SAN Storage offers better throughput ECKD/FICON SAN Storage uses less CPU per transaction You have to tune both environments Recommendation: it depends 11 © 2017 IBM Corporation ECKD/DASD UDEV Rule – Device Permission Change Red Hat 6 / SUSE 11: Red Hat 7.1+: SUSE 12 SP1+ 12 © 2017 IBM Corporation Linux multipathing Recommendations /etc/multipath.conf parameters can impact your FCP/SCSI performance particularly with I/O bound workloads: Check with your OS distributor & Storage vendors for recommended settings rr_min_io_rq=1 #Number of I/Os per Path (Newer Linux Kernels) SLES 11 SP3+, RH 6.x+ #rr_min_io #Use for Older Linux Distros (Red Hat 5.x,SLES 11sp1 & older) path_selector "service-time 0" #default for SLES 11 SP3+, RedHat 6.7+ #path_selector "round-robin 0" #round-robin for SLES 11 SP2+ and RedHat 6.x path_grouping_policy "multibus" # SLES 11 SP1+ and Red Hat 6.x+ # path_grouping_policy "failover" # SLES 10, Red Hat 5.x 13 © 2017 IBM Corporation SUSE 12 SP1+ multipath.conf Recommendations SUSE 12: /etc/multipath.conf: To make changes in effect dynamically: systemctl restart multipathd.service 14 © 2017 IBM Corporation Source: https://www.suse.com/documentation/sles-12/pdfdoc/stor_admin/stor_admin.pdf Red Hat 7: Multipath Recommendations Red Hat 7 /etc/multipath.conf: To make changes in effect dynamically: systemctl restart multipathd.service 15 © 2017 IBM Corporation Source: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/pdf/DM_Multipath/Red_Hat_Enterprise_Linux-7-DM_Multipath-en-US.pdf I/O Kernel Scheduler: The Linux 2.6 kernel offers a choice of four different I/O schedulers: – Noop Scheduler (noop) – Deadline Scheduler (deadline) – Anticipatory Scheduler (as) – Complete Fair Queuing Scheduler (cfq) – Default for a lot of kernels Some Linux kernels default is the “cfq” scheduler: – Designed to optimize access to physical disks – Check in /sys/block/<device>/queue/scheduler noop anticipatory [deadline] cfq – Default configurable by setting the “elevator=[…]” boot parameter in /etc/zipl.conf Recommendation: deadline or noop for Oracle database LUNs © 2017 IBM Corporation 16 Changing I/O scheduler with UDEV Rule I/O settings you may want to consider adjusting (based on storage vendor recommendations): scheduler I/O scheduler (noop or deadline) nr_requests number of I/O requests default 128 (may want to increase to 1024 for Flash Storage devices) rotational 1 default for spinning disk, 0 for Flash nomerges default 0, 1 for Flash max_sectors_kb default 1024, 4096 for Flash rq_infinity default 1, 2 for Flash add_random default 1, 0 for Flash Sample UDEV rules (/etc/udev/rules.d) /etc/udev/rules.d/12-dm-permissions.rules (Multipath Devices) ENV{DM_NAME}=="ASM*",OWNER:="oracle",GROUP:="dba",MODE:="660",RUN+="/bin/sh -c 'echo deadline > /sys/$devpath/queue/scheduler '" /etc/udev/rules.d/99-udev-oracle.rules (Single Path Devices) KERNEL=="sda",OWNER="oracle",GROUP="dba",MODE="660",RUN+="/bin/sh -c ‘echo deadline > /sys/$devpath/queue/scheduler'“ Restarting UDEV rules (Sles 11 sp3/Red Hat 6): udevadm control --reload-rules udevadm trigger © 2017 IBM Corporation 17 Setting Up Disk FCP/SCSI Devices for Oracle /etc/multipath.conf: multipath { wwid 20020c2400d1221a8 alias ASMFLASH0D } # multipath -ll ASMFLASH0D ASMFLASH0D (20020c2400d1221a8) dm-21 IBM, FlashSystem size=256G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 5:0:1:13 sdbv 68:144 active ready running |- 7:0:1:13 sdep 129:16 active ready running `- 7:0:0:13 sdfd 129:240 active ready running [root@zlnxl2 mapper]# ls -lato /dev/mapper/ ASMFLASH0D lrwxrwxrwx 1 root 8 May 13 16:46 ASMFLASH0D -> ../dm-21 [root@zlnxl2 rules.d]# ls -la /dev/dm-21 brw-rw---- 1 root disk 253, 21 May 13 14:54 /dev/dm-21 [root@zlnxl2 rules.d]# udevadm trigger [root@zlnxl2 rules.d]# ls -la /dev/dm-21 brw-rw---- 1 oracle dba 253, 21 May 13 16:46 /dev/dm-21 © 2017 IBM Corporation 18 ASM or LVM file Systems? LVM – Logical Volume Manager in Linux ASM – Automated Storage Management provided by Oracle – Oracle RAC One and Oracle RAC will require ASM LVM ASM pro • Direct control on setting and • Automated, out of the box layout environment • Can choose file system • Very good integration with Oracle con • Complex setup • RMAN required for backup Overall recommendation: ASM Don’t combine both! 19 © 2017 IBM Corporation File System Recommendations SLES 11 SP4+ Supported and Recommended File ext3 xfs Systems on Linux (MOS: 236826.1 ) SUSE xfs for DB files (24.7% increase and 12.9% increase for Reads (NON ASM) Red Hat 6.x ext4 for database files Red Hat 7.x - XFS is now recommended. XFS – difficult to reduce the size of a LVM. 20 © 2017 IBM Corporation Database files on Filesystem: Disable read ahead Oracle parameter file systems: filesystemio_options=setall – Provides asynchronous & direct I/O (avoids linux file system cache) Reduce Linux Read-Ahead for LVM file systems. – lvchange -r none <lv device name> © 2017 IBM Corporation 21 Customer Scenario - Application Performance Problem Consolidated several systems to one system Transaction times > 6 seconds for 100 Transactions per seconds (Normally sub 2 seconds) after migration. 22 © 2017 IBM Corporation Oracle AWR Report Analysis (1): 23 © 2017 IBM Corporation Oracle AWR Report Analysis (2): SQL ID: 4ztz048yfq32s: SELECT TO_CHAR(current_timestamp AT TIME ZONE 'GMT', 'YYYY-MM-DD HH24:MI:SS TZD') AS curr_timestamp, COUNT(username) AS failed_count, TO_CHAR(MIN(timestamp), 'yyyy-mm-dd hh24:mi:ss') AS first_occur_time, TO_CHAR(MAX(timestamp), 'yyyy-mm-dd hh24:mi:ss') AS last_occur_time FROM sys.dba_audit_session WHERE returncode != 0 AND timestamp >= current_timestamp - TO_DSINTERVAL('0 0:30:00') 24 © 2017 IBM Corporation Oracle DBA_AUDIT_ANALYSIS: Query 4ztz048yfq32s is an audit check –many failed logon attempts an alert is generated. Slow Performance Of DBA_AUDIT_SESSION Query From "Failed logins" Metric in Enterprise Manager (Doc ID 1178736.1) - Bug 7633167 : SQL PERFORMANCE ISSUE IN DBA_AUDIT_SESSION. Oracle Note:73408.1 "How to Truncate, Delete, or Purge Rows from the Audit Trail Table AUD$" Be aware that it may be a company requirement to keep a backup of the rows in this table. Ensure that a backup is taken before truncating/deleting as discussed in Note:73408.1 Note:731908.1 "New Feature DBMS_AUDIT_MGMT To Manage And Purge Audit Information" Bug is fixed in the 12.1.0.5 database plugin. See note:1388143.1 "How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud Control" for instructions on deploying the latest database plug-in. Apply the latest DB Plugin PSU Patches: Patch:17998764 OMS-Side Patch:18039625 and Patch:18012965 Agent-Side 25 © 2017 IBM Corporation Oracle SGA_TARGET Analysis: Finding 2: Undersized SGA Impact is 34.3 active sessions, 56.94% of total activity. --------------------------------------------------------- The SGA was inadequately sized, causing additional I/O or hard parses. The value of parameter "sga_target" was " 33792 M " during the analysis period. Recommendation 1: Database Configuration Estimated benefit is 34.3 active sessions, 56.94% of total activity. --------------------------------------------------------------------