Red Hat Enterprise Linux Update for IBM Z Systems

International Oracle on z Systems Special Interest Group (SIG) Red Hat Update for Oracle Databases on z Systems

Filipe Miranda Global Technical Lead on IBM z Systems and Power Systems Global Partner & Technical Enablement Red Hat Inc.

Red Hat in a Nutshell Open Source Development Model Improvements I/O RHEL 6.5-6.7

High Availability (Software/Hardware) options for Oracle Databases on z Systems

It is time to move to RHEL7 ! RHEL 7.2 Summary of New Features

RHEL 7.1 z Systems Features Summary IBM Redbooks Publications Automated LUN Scanning CPACF Exploitation

RHEL 7.x other features in detail RHEL6 vs RHEL7

INKTANK (CEPH), RHEL OPENSTACK PLATFORM RELEASED FOUNDEDIPO FIRST RELEASEJBOSS OF ENTERPRISEACQUIRED LINUXRED HAT ADDED TO $1S&P BILLION 500 INDEX INRED REVENUE HAT STORAGEOPENSHIFT SERVER ENTERPRISE RELEASED RELEASEDCENTOS JOINS RED HAT ENOVANCE (OPENSTACK) RED HAT ENTERPRISE CLOUDFORMS VIRTUALIZATION & OPENSHIFT RELEASED RELEASED & FEEDHENRY ACQUIRED 1993 1999 2002 2006 2009 2011 2012 2013 2014

• Moving up a minor release, like 6.5 to 6.7 -> many of improvements • New minor release, new kernel, device drivers updates that improve performance for network, I/O, HBAs, cryptography etc…

[2.6.32-431.29.1.el6]: - [s390] af_iucv: wrong mapping of sent and confirmed skbs (Hendrik Brueckner) [1112390 1102248] - [s390] af_iucv: recvmsg problem for SOCK_STREAM sockets (Hendrik Brueckner) [1112390 1102248] - [s390] af_iucv: fix recvmsg by replacing skb_pull() function (Hendrik Brueckner) [1112390 1102248] - [s390] kernel: avoid page table walk on user space access (Hendrik Brueckner) [1111194 1099146] - [s390] qeth: postpone freeing of qdio memory (Hendrik Brueckner) [1112134 1094379] - [s390] qeth: Fix retry logic in hardsetup (Hendrik Brueckner) [1112134 1094379] - [s390] qeth: Recognize return codes of ccw_device_set_online (Hendrik Brueckner) [1112134 1094379] - [s390] qdio: remove API wrappers (Hendrik Brueckner) [1112134 1094379] [2.6.32-431.23.1.el6]: - [s390] crypto: fix aes, des ctr mode concurrency finding (Hendrik Brueckner) [1110168 1096328] - [s390] crypto: fix des and des3_ede ctr concurrency issue (Hendrik Brueckner) [1109885 1065404] - [s390] crypto: fix des and des3_ede cbc concurrency issue (Hendrik Brueckner) [1109883 1065398] - [s390] crypto: fix concurrency issue in aes-ctr mode (Hendrik Brueckner) [1110169 1063478] - [s390] crypto: Fix aes-xts parameter corruption (Hendrik Brueckner) [111017 - [kernel] futex: Forbid uaddr == uaddr2 in futex_wait_requeue_pi()(Mateusz Guzik) [1097759 1097760] {CVE-2012-6647} - [kernel] tracing: Reset ring buffer when changing trace_clocks (Marcelo Tosatti) [1093984 1018138] [2.6.32-431.20.1.el6]: - [s390] fix kernel crash due to linkage stack instructions (Hendrik Brueckner) [1067678 1067679] {CVE-2014-2039}

z SYSTEMS LINUXONE

RAC Application Data Guard Oracle Flashback Provided CRS HA Grid Control

Linux Clustering Operating Linux System Mature Hipervison HA Hardware assist for performance and z/VM recovery

Spare CPUs N+1 power supplies Hardware Chip sparring in memory Z Systems Provided Concurrent Maintenance HA 50 years MTBF (system failures)

Single book systems may not support concurrent memory upgrades

RAC Application Data Guard Oracle Flashback Provided CRS HA Grid Control

Linux Clustering Operating Linux System Mature Hypervisor HA Hardware assist for performance and z/VM recovery

Spare CPUs N+1 power supplies Hardware Chip sparring in memory Z Systems Provided Concurrent Maintenance HA 50 years MTBF (system failures)

Product Components:

• Cluster (fail-over) implementation software • Cluster (fail-over) management and monitoring software • GFS2 file system software and support • Support for LPAR and z/VM implementations • Load Balancing (LVS) implementation software • Load Balancing management GUI, piranha.

Supported RHEL Release Levels:

• RHEL 6.2+ for z Systems • RHEL 7.0+ for z Systems

Other Prerequisite Software Supported Release Levels:

• z/VM 5.4 • z/VM 6.1 • z/VM 6.2 • z/VM 6.2 with Single System Image (SSI) Option • z/VM 6.3

HA (Failover) and Resilient Storage (GFS2)

Features for Red Hat High Availability and Resilient Storage cluster

• A maximum of four nodes is supported, which must be within same physical mainframe on the same LPAR. • Mixed clusters consisting of x86 and z Systems VMs are not supported, nor are mixed clusters consisting of z Systems VMs and VMs from other virtual platforms (such as RHEV, VMWare vCenter, KVM, etc). • The only supported storage is DASD as a shared device with 4k block size • No cmirror support • No ctdb (clustered samba) support • No SSI support at the initial release of Red Hat Enterprise Linux release 7 Update 2

Requirements for IBM z Systems LPAR

• The IBM z Systems LPAR must be running z/VM V6.3 • The IBM z Systems LPAR should be configured with enough storage (memory) to back the storage allocated to the cluster. • The IBM z Systems LPAR should have IBM Directory Maintenance (DirMaint) for z/VM installed. • The IBM z system LPAR should have SMAPI (Systems Management Application Programming Interface) service installed, which must be accessible via TCP/IP. • The IBM z system LPAR should be configured with a Linux VM for each node in the cluster. For information on installing Red Hat Enterprise Linux 7 on IBM z Systems, see "Part III. IBM System z Architecture - Installation and Booting" in the Red Hat Enterprise Linux 7 Installation Guide. • The s390utils package must be installed on each node. This package is installed by default with RHEL on s390x systems, and contains the utilities required to format and partition dasd devices, which will be required when setting up a GFS2 file system.

RAC Server Data Guard Oracle Flashback Provided CRS HA Grid Control

Linux Clustering Operating Linux System Mature Hipervison HA Hardware assist for performance and z/VM recovery

Spare CPUs N+1 power supplies Hardware Chip sparring in memory Z Systems Provided Concurrent Maintenance HA 50 years MTBF (system failures)

IBM z Systems recorded the least amount of downtime of any hardware platform

ITIC survey based on real customer experience:

• z Systems is only server that had no – 0% – unplanned system downtime due to any inherent flaws in the hardware

• Conversely, 6% of Dell PowerEdge x86 machines and 10% of HP ProLiant systems racked up downtime exceeding four hours per year per server

Source: ITIC 2015 Global Server Hardware and Server OS Reliability Survey

Transparent sparing for all CPU types • CP, ICF, IFL, zAAP, zIIP

z13 has 2 spare CPUs per server, zBC12 has 1 • Spares do not have to be local to the same book

Processor Availability Facility (PAF) saves state and switches to spare CPU • Error detection circuits detect a failing processor • Failing processor is stopped • Data contents of failing processor are transferred to spare processor • Processing resumes on spare processor • NO apparent interruption to the workload

Capability z13 x86 ECC on Memory Control Can recognize/repair soft errors while running; Transparent While Running Circuitry limited ability with hard errors Oscillator Failure Transparent While Running Must bring server down to replace

Core Sparing Transparent While Running Must bring server down to replace

Microcode Driver Some OS-level drivers can update while running, While Running Updates not firmware drivers; reboot often required

Book Additions, Replacement While Running Must bring server down

Memory Replacement While Running Must bring server down Memory Bus Adaptor While Running Must bring server down Replacement Must bring server down to replace I/O Upgrades While Running (limited ability to replace I/O in some servers ) Limited – Concurrent Driver Maintenance While Running some drivers replaceable while running “Support processors” can act as Redundant Service Element 2 per System poor man’s SE, but no redundancy

Single book systems may not support concurrent memory upgrades

• Enable RDMA in Mellanox driver for RDMA on System This feature enables the user space programs to initiate PCI I/O operations on System z using the standard Linux device driver program interface. The patches for the Mellanox kernel device driver, for user space Inﬁniband verbs library libibverbs, and for the user space libibverbs library plug-in libmlx4 library modify the following components: • "ib_uverbs" kernel module; • "mlx4_ib" kernel module; • "libibverbs" user space library; • "libmlx4" plug-in library for the libibverbs library.

• Support the Crypto Express5S (CEX5S) crypto adapter in kernel and s390utils/s390-tools Implement full support for new crypto hardware for System z

• SE/HMC ﬁle access support in kernel and s390utils/s390-tools Provide access to ﬁles on the USB/DVD drive of the SE/HMC of System z hardware

• SMT base support in in kernel and s390utils/s390-tools Improved performance by utilizing new HW instructions

• SIMD base support in in kernel

• Use CCA coprocessor random numbers to improve kernel entropy pool The Linux kernel needs input to its entropy pool to generate strong pseudo random numbers, which are used in many security related methods (key generation, pgp, AS randomization, ...).

The numbers generated by the true random number generator of the CCA coprocessor should be fed into the kernel entropy pool if a CCA coprocessor is available.

Check whether /dev/hwrandom is available and if so run the rngd to periodically feed random numbers from /dev/hwrandom in to the kernel entropy pool. This should be triggered with every Linux boot.

• Kernel support to improve Java performance for Linux on System z - Technology Preview

• Enable PCI per default for Linux on IBM z Systems With the PCI and RDMA enabling features in RHEL7.2 PCI should be enabled per default

• Add GPFS partition type to fdasd - s390utils/s390-tools The fdasd tool now supports handling and identification of an additional partition type called "GPFS". This partition type identifies partitions containing GPFS Network Shared Disks (NSD) used to store GPFS file system information. This RAS feature improves reliability and consumability of a GPFS on ECKD setup via the new partition type "GPFS".

• LLVM optimization for System z Deterministic Random Bits Generator (DRBG) support This feature improves the s390 pseudo-random device driver /dev/prandom to use the "perform pseudo random number operation" available with message-security assist (MSA) 5.

2013 2015

http://www.redbooks.ibm.com

- Production Ready -

z Systems z Systems z Systems

z Systems

Example on how simple it is to work with FCP devices:

1 .Unblock the devices

# cio_ignore -r fc00 # cio_ignore -r fd00

2 .Enable the devices

# chccwdev -e fc00 # chccwdev -e fd00

3. LUN already detected on all paths

# lsluns Scanning for LUNs on adapter 0.0.fc00 at port 0x500507630500c74c: 0x4010401700000000 at port 0x50050763050bc74c: 0x4010401700000000 Scanning for LUNs on adapter 0.0.fd00 at port 0x500507630510c74c: 0x4010401700000000 at port 0x50050763051bc74c: 0x4010401700000000

4. Enable multipath:

# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf

# systemctl start multipathd # systemctl enable multipathd

# multipath -ll mpatha (36005076305ffc74c0000000000001017) dm-2 IBM size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 0:0:0:1075265552 sda 8:0 active ready running |- 0:0:1:1075265552 sdb 8:16 active ready running |- 1:0:0:1075265552 sdc 8:32 active ready running `- 1:0:1:1075265552 sdd 8:48 active ready running 5. Make the FCP devices persistent:

# lszfcp -D | awk '{ print $1 }' | sed -e 's/\// /g' >> /etc/zfcp.conf

6. Partition the multipath device:

# parted -s /dev/mapper/mpatha mklabel msdos mkpart primary 0% 100%

- Production Ready -

• Support for Crypto Express4S Driver tolerance for Crypto Express 5S

• Tolerance for Crypto Express5S - Production Ready - • CP Assist for Cryptographic Function (CPACF)

…

z Systems z Systems - Production Ready - z Systems

- Production Ready -

z Systems - Production Ready - - Production Ready - - Production Ready -

systemd is a system and services manager that replaced upstart in RHEL. systemd is the ﬁrst user space process the kernel starts when booting. This process is responsible for starting all the services and their dependencies that allow the system to act as a server. systemd uses units that can be dependent on other units.

There are different unit types such as:

• Service units, used to start services

• Socket units, which allow socket based activations

• Device units, which trigger reactions for devices as they appear or disappear

• Mount point units, that control mount points

• Target units, which allow grouping of units to act as synchronization points

List active service units (running services):

List failed service units:

# systemctl -t service --state=failed

Query the status of a service unit:

Stop, start, and restart a service unit:

# systemctl stop vsftpd.service # systemctl start vsftpd.service # systemctl restart vsftpd.service

You can also omit the sufﬁx and specify multiple units:

# systemctl restart vsftpd sshd

Reload a service unit

# systemctl reload sshd

Not all services support that

List installed service units:

# systemctl list-unit-ﬁles -t service

List enabled service units:

Disable a service:

# systemctl disable vsftpd

Query the default target the system boots into:

# systemctl get-default

Switch to a target: # systemctl isolate multi-user.target

IPL your Linux server from the 3270 console using the following commands:

In rescue.target mode, all of the ﬁle systems in /etc/fstab are mounted, but networking has not been started.

To enter a different systemd target, from the IPL command just type the target you want in the systemd.unit= parameter. For example, to enter the emergency.target, use the following commands:

To load systemd targets from the IPL command when using DASD ECKD/FBA use the following commands from the 3270 terminal:

===> ipl 100 PARM systemd.unit=rescue 00: zIPL v1.23.0-17.el7 interactive boot menu 00: 00: 0. default (linux) 00: 00: 1. linux 00: 00: Note: VM users please use '#cp vi vmsg ' 00: 00: Please choose (default will boot in 5 seconds): 00: Booting default (linux)... 00: Uncompressing Linux...... Welcome to rescue mode! Type "systemctl default" or ^D to enter default mode. Type "journalctl -xb" to view system logs. Type "systemctl reboot" to reboot. Give root password for maintenance (or type Control-D to continue):

The journal is part of systemd and provides a modern logging mechanism. It allows to capture Kernel log messages, regular syslog messages, the stdout/stderr written by services, and messages from the early boot stages.

Viewing the journal

Filtering the journal

Show the log messages of the current boot (ﬁlters out the messages from previous boots):

# journalctl -b

Show today’s log messages:

# journalctl --since today

Show kernel messages of the current boot only:

# journalctl -b -k

Only show errors:

# journalctl -p err

Show the log messages of a speciﬁc unit (like the sshd.service unit):

# journalctl -u sshd.service

Red Hat Enterprise Linux 6

Red Hat Enterprise Linux 7

Red Hat Enterprise Linux 6

Red Hat Enterprise Linux 7

Step 1: Determine the UUID of the multipath disk meant for Oracle ASM use.

#udevadm info --query=all --name=/dev/mapper/mpathXX | grep -i DM_UUID

Step 2: Add below udev rule for a multipath device using it's DM_UUID value under the file /etc/udev/ rules.d/99-oracleasm.rules

ACTION=="add|change", ENV{DM_UUID}=="mpath-", SYMLINK+="asmdisk1", GROUP="oinstall", OWNER=“oracle", MODE="0660" Note: Replace with the one you got it from the step 1 command output.

#vi /etc/udev/rules.d/99-oracleasm.rules Add above udev rule

Step 3: To add or load Udev rules using the below commands.

# /sbin/udevadm control --reload-rules # /sbin/udevadm trigger --type=devices --action=change

Step 4: Check file permissions on the disk paths.

# ls -l /dev/udev-asmdisk1 # ls -l /dev/mapper/mpathn # ls -l /dev/dm-x

Red Hat Enterprise Linux 6

Red Hat Enterprise Linux 7

IBM EC12s Red Hat Enterprise Linux 6 ~48 IFLs Thank you Danke Grazie Merci 拽拽 ͘Π͢;͜ Obrigado!

Filipe Miranda Global Technical Lead on IBM z Systems and Power Systems Global Partner & Technical Enablement