January 2009
High Performance FICON for System z Technical summary for customer planning
Iain Neville System z Consulting IT specialist IBM Systems and Technology Group IBM United Kingdom
System z High Performance FICON customer readiness documentation Page 1
Table of Contents
Who should read this document? ...... 2 What is the purpose of this document? ...... 2 Overview of System z High Performance FICON ...... 2 Mainframe I/O technology milestones: ESCON to zHPF ...... 3 ESCON architecture...... 3 FICON architecture ...... 4 MIDAW facility...... 5 High Performance FICON for System z (zHPF) ...... 6 RAS benefits with zHPF ...... 8 zHPF Frequently Asked Questions...... 8 References...... 14 Acknowledgements ...... 14
System z High Performance FICON customer readiness documentation Page 2
Who should read this document? This document should be referenced by IBM System z® hardware planners, system programmers and those with responsibility for the technical positioning of the System z platform.
What is the purpose of this document? This document provides the following: A high level overview of High Performance FICON for System z (zHPF) to give the reader an understanding of the significance of the enhancements to the z/Architecture® and FICON® interface architecture to increase performance for online transaction processing workloads. At a high level, demonstrate the value of the zHPF architecture. List the high level pre-requisites for the adoption of zHPF. A list of answers for a set of frequently asked questions.
Overview of System z High Performance FICON High Performance FICON for System z (zHPF) is one of IBM’s latest enhancements to the z/Architecture and FICON interface architecture to dramatically improve the performance of OLTP workloads. These architectural enhancements are designed to substantially improve I/O performance by reducing the number of Channel Command Words (CCWs) and Information Units (sequences) which remove overhead on the storage subsystem and the FICON channel microprocessor. The rationalization of layers of handshakes is established without compromising the reliability, availability and serviceability (RAS) implemented with FICON technology today.
These enhancements have been for optimization of online transaction processing (OLTP) workloads using numerous access methods such as DB2®, VSAM, PDSE, HFS, zFS. When exploited by the FICON channel, the z/OS® operating system, and the storage subsystem, zHPF is designed to help reduce overhead and improve performance. Additionally, the changes to the architectures offer end-to-end system enhancements to improve reliability, availability, and serviceability (RAS). System z High Performance FICON customer readiness documentation Page 3
Mainframe I/O technology milestones: ESCON to zHPF The evolution of mainframe I/O architecture includes many significant milestones. When these are reviewed from the outset, the significance of zHPF can be put into context. In particular when we compare a start I/O using ESCON® to that of FICON and now zHPF, the significance of these iterative changes becomes apparent.
A brief summary of the differences between ESCON architecture and FICON and zHPF architecture is captured in Table 1.
ESCON Native FICON, CTC, zHPF Circuit switching Packet switching Read or write Simultaneous read and write Half-duplex data transfers Full-duplex data transfers Connection-oriented Connectionless Pre-established Packets dedicated path individually routed Connection is locked Connection released when data sent when data sent Synchronous transfers Asynchronous transfers CCW architecture CCW architecture for FICON TCW architecture for zHPF
Table 1: Channel Architecture comparisons
ESCON architecture Enterprise Systems Connection (ESCON) is a 17MBps architecture first introduced in September 1990. This is a circuit switching architecture that mandates a channel end and device end (CE/DE) is received for every CCW (command) before the channel is released. ESCON is industry standard – Single Byte Command Code Sets Connection Architecture (SBCON). System z High Performance FICON customer readiness documentation Page 4
ESCON command and data sequences: 4 x 4K reads.
ESCON Channel Storage subsystem Device Control Unit
CCW1 Read 4K CCW1 CCW1 CE/DE END
CCW2 Read 4K CCW2 CCW2 CE/DE END
CCW3 Read 4K CCW3 CCW3 CE/DE END CCW4 Read 4K CCW4 CCW4 CE/DE END
• Each CCW is individual and traverses the link in a serial manner. • Channel End (CE) and Device End (DE) for each CCW
Figure 1: ESCON command and data sequences
Figure 1 shows 4 CCWs being executed. A channel end / device end must be received before another can be executed.
FICON architecture Fibre Connection (FICON) is a packet switching architecture that scales from 1Gbps to 10 Gbps and was first made available in August 1999. FICON creates a logical end to a chain of CCWs. There is no longer a requirement to wait for an individual channel and device end (CE/DE). A logical end is established to the control unit after each CCW is executed. FICON is also industry standard – Fibre Channel Single Byte Command Code Sets-3 (FC-SB-3).
Information Units (IUs or sequences) of up to 8K in size are built for each CCW. Each IU is then split into frames of 2K which are formatted individually before being sent over the optic link as reflected in Figure 2. System z High Performance FICON customer readiness documentation Page 5
FICON command and data sequences: 4 x 4K reads
FICON Channel Storage subsystem Device Control Unit
CMD Prefix +64 CMR
CCW1 Read 4K CCW1 CCW1 Data 4K CCW2 Read 4K CCW2 We no CCW3 Read 4K CCW3 CCW2 longer CCW4 Read 4K CCW4 Data 4K have to CCW3 Data 4K wait….. CCW4 Data 4K
CE / DE Status Status Accept
Note: standard CCWs, not Read track Data
Figure 2: FICON command and data sequences
The picture above is simplified. With Extended Format (EF) datasets each CCW (command) from the channel adds a 32 byte suffix to each command and data block.
This requires an additional 4 x IUs to build into the command sequence and a corresponding 4 x IUs for the data stream coming back from the storage subsystem.
Over time there have been a number of enhancements to improve performance in a FICON environment.
MIDAW facility The Modified Indirect Data Address Word (MIDAW) facility was introduced in July 2005. This enhancement implements an extension to indirect addressing to improve performance by rationalizing the number of CCWs. The restriction of aligning the data to a page boundary is removed. This facility benefits applications using numerous dataset types including Extended Format datasets, VSAM, PDSEs and zFS. The suffix now becomes a different address (MIDAW) to the same CCW, reducing the number of command sequences by up to 46%. System z High Performance FICON customer readiness documentation Page 6
MIDAW: 2 CCWs reading 14 Extended Format (EF) disk records
CCW #1: Write Prefix CC Count=64 4K CCW #2: Read Track Data M Count=57792 Addr=1A000000 buffer
CC = Command Chaining 4K M = MIDAL indicator buffer MIDAW #1: Flag=00 Count=4096 Addr=20000000 ... MIDAW #2: Flag=00 Count=32 Addr=1E000000 MIDAW #3: Flag=00 Count=4096 Addr=25000000 4K MIDAW #4: Flag=00 Count=32 Addr=1E000020 buffer . 1st record suffix . 2nd record suffix . . MIDAW #27: Flag=00 Count=4096 Addr=31000000 . MIDAW #28: Flag=L Count=32 Addr=1E0001A0 . L = Last MIDAW indicator 14th record suffix
Figure 3: MIDAW facility and chained CCWs
With MIDAWs a sequence of chained CCWs can effectively be reduced to two within a limit of 64K. Figure 3 reflects this with an entire track of 4K reads being reduced from fourteen CCWs down to two. However, the data blocks from the storage subsystem are still transferred individually back to memory.
High Performance FICON for System z (zHPF) With the introduction of zHPF, the FICON architecture has been streamlined by removing significant overhead to the storage subsystem and the microprocessor within the FICON channel. A command block is created to chain commands into significantly fewer IUs. The overhead required to convert individual commands into FICON format is removed as multiple System z I/O commands are packaged together and passed directly over the fibre optic link. For zHPF operations, we lose the concept of the Command Control Word (CCW). This is replaced with the Transport Control Word (TCW) which allows multiple channel commands to be sent to the control unit as a single entity instead of a stream of individual commands. Compatibility is established such that existing FICON operations (CCWs) coexist with zHPF operations (TCWs). System z High Performance FICON customer readiness documentation Page 7
One single zHPF command block replaces a series of FICON CCWs. For the example in Figure3 above, the channel processor and the DASD controller had to process five FICON IUs, for each of the commands in the 4x4k read channel program. With zHPF, there is only one IU to process for the single command block as portrayed in Figure 4.
zHPF command and data sequences: 4 x 4K reads.
FICON Channel Storage subsystem Device Control Unit What a CMD Prefix +64, Read 4K, Read 4K, reduction in Read 4K, Read 4K CCWs and IUs…
Data 16K
CMD Block Response
Figure 4: zHPF command and data sequences
Substantial performance improvements can be realized in terms of throughput (MB/sec) and I/Os per second (number of starts per channel). This will be dependant upon the application workload. The initial implementation of zHPF by the IBM DS8000™ is exclusively for I/Os that transfer less than a single track of data. Realistic production workloads with a mix of data transfer sizes can see between 30 and 70% of FICON I/Os utilizing zHPF. This can result in between a 10 to 30% saving in channel utilization. System z High Performance FICON customer readiness documentation Page 8
RAS benefits with zHPF The zHPF architecture has reliability, availability, and serviceability enhancements “built in”. Significantly more information is now provided by the control unit on errors to aid in fault isolation. There is now a dedicated buffer for concurrent sense data making it unnecessary to separate sense commands. Missing interrupt problem determination is provided through an interrogate mechanism providing the operating system the capability to query the state of operation at the control unit when detecting missing interrupts. Previously no information was available on missing interrupts and debugging involved attempting to correlate software, channel, and control unit log and trace data. Furthermore, the reason for aborts is provided to control units using an enhanced “purge path” mechanism.
zHPF Frequently Asked Questions Question 1: What is the business value of zHPF ? Answer: When properly configured, zHPF is designed to help reduce overhead and improve performance. The changes to z/Architecture and the FICON interface architecture offer end- to-end system enhancements to improve reliability, availability, and serviceability (RAS). Furthermore, existing FICON Express4 and FICON Express2 features can be used with no modifications to the CHPID definition.
Question 2: How is my application impacted? Answer: zHPF is transparent to applications. Data accessed by DB2, PDSE, VSAM, zFS and Extended format SAM can benefit from the improved transfer technique.
Question 3: Is zHPF a chargeable feature on my System z server? Answer: No, zHPF is implemented in the Licensed Internal Code of IBM System z10™. There is no chargeable feature.
Question 4: Which servers support zHPF? Answer: zHPF is exclusive to System z10 servers IBM System z10 Enterprise Class and IBM System z10 Business Class™ (z10 EC™ and z10 BC™) at driver level 76.
Question 5: Which storage subsystems support zHPF? Answer: The IBM System Storage™ DS8000 series support zHPF. The minimum Licensed Internal Code (LIC) required is bundle version 64.1.16.0 (release 4.1) or later.
zHPF is supported by IBM System Storage DS8000 series machine types 2107, 239x, 2244, 2421, 2422, 2423 and 2424. Refer to U.S. announcement letters 108-869, 108-870, 108-871 and 108-872 dates 21 October 2008. System z High Performance FICON customer readiness documentation Page 9
Question 6: Is zHPF a chargeable feature on the IBM DS8000 controller? Answer: Yes, the ‘High Performance FICON’ feature is required. This is a priced license feature (one time charge) and has a monthly maintenance charge.
Question 7: Which FICON features support zHPF? Answer: FICON Express2 & FICON Express4 features on System z10 servers at driver level 76. This support applies to CHPID type FC.
Question 8: Which operating systems support zHPF and what are the software prerequisites? Answer: The zHPF function is currently exclusive to the z/OS operating system. It is supported by z/OS V1.8, V1.9 and V1.10 with PTFs as well as z/OS V1.7 with the IBM Lifecycle Extension for z/OS V1.7 (5637-A01) with PTFs. Maintenance specific to zHPF can be found in the zHPF FUCNTION PSP bucket at the following URL: http://www14.software.ibm.com/webapp/set2/psp/srchBroker
Question 9: What are the zHPF prerequisites for my Independent Software Vendors? Answer: All relevant ISVs have been briefed on the zHPF architecture. Contact them directly for any prerequisite fixes.
Question 10: Which Channel Path Identifier (CHPID) types support zHPF? Answer: zHPF is exclusive to CHPID type FC (native FICON).
Question 11: How does zHPF affect my Input/Output Control Program (IOCP)? Answer: There is no change required to the IOCP. FICON channels are still defined the same way, as CHPID type FC.
Question 12: How does zHPF affect my workload performance? Answer: A comprehensive performance reference will be made available at the following address. http://www.ibm.com/systems/z/hardware/connectivity/ficon_performance.html
A general overview of performance potential in terms of I/Os per second and MegaBytes per second is reflected in the chart below.
System z High Performance FICON customer readiness documentation Page 10
High Performance FICON for System z (zHPF) Optimization of storage area network (SAN) traffic using zHPF to improve performance ► Maximum number of I/Os per second can be increased by up to 100%* ► For OLTP workloads (DB2, VSAM, PDSE, and zFS ); small blocks of fixed size data (4K blocks) Exclusive to System z10 - FICON Express4 and FICON Express2 Requires ► Control unit exploitation – IBM DS8000 Release 4.1 ► z/OS V1.7 with the IBM Lifecycle Extension for z/OS V1.7 (5637-A01), V1.8, V1.9, or V1.10 with PTFs ► FICON Express cannot be in the path z H z P H 30000 F 600 P 550 F 27500 I/Os per second 28000 MegaBytes per second (full-duplex) 4k block size, channel 100% utilized 500 Large sequential read/write mix 25000 FICON 520 Express4 22500 450 FICON FICON FICON Express2 Express4 20000 400 Express4 4 Gbps FICON 4 Gbps FICON Express4 17500 Express4 and 350 FICON and FICON Express2 350 15000 FICON Express2 300 2 Gbps Express2 12500 14000 250 FICON 270 FICON 13000 Express 10000 Express 200 FICON 2 Gbps FICON Express Express 9200 7500 FICON 150 1 Gbps 170 7200 5000 FICON 6000 100 FICON 120 z990 z9 z890 z9 2500 3600 z10 50 74 z900 z990 z10 z900 z990 z990 z10 zHPF z10 zHPF G5/G6 z900 z800 z890 z890 G5/G6 z900 z800 z890 z9 0 0 * Some complex channel programs cannot be converted to zHPF protocol Page 1 IBM System z © 2008 IBM Corporation
Question 13: How does zHPF affect my FICON fabric (SAN)? Answer: No change is required to the configuration or design of the fabric. I/O will flow over the fabric as either FICON or zHPF IUs.
Question 14: What maintenance is required on my FICON directors in support of zHPF? Answer: zHPF is transparent to FICON directors and switches.
Question 15: How does zHPF affect my performance monitoring? Answer: SMF TYPE 73 and 79 have been changed in support of zHPF. There are 6 new fields added to the RMF™ Channel activity report to distinguish between FICON and zHPF traffic. System z High Performance FICON customer readiness documentation Page 11
For example, the data below reflects detail from the RMF Channel Path Activity report. A single channel (CHPID 13) was doing 18092 zHPF I/Os per second and on average, there were 8.8 zHPF I/Os simultaneously active.
CHANNEL PATH UTILIZATION(%) READ(MB/SEC) WRITE(MB/SEC) FICON OPERATIONS zHPF OPERATIONS ID TYPE G SHR PART TOTAL BUS PART TOTAL PART TOTAL RATE ACTIVE DEFER RATE ACTIVE DEFER
10 FC_S 5 Y 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.0 0.0 0.0 0.0 0.0 11 FC_S 5 Y 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.0 0.0 0.0 0.0 0.0 12 FC_S 5 Y 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.0 0.0 0.0 0.0 0.0 13 FC_S 5 Y 63.97 63.97 12.98 74.11 74.11 0.00 0.00 0.4 1.0 0.0 18092 8.8 0.0
The RATE is the number of FICON or zHPF I/Os per second at the total physical channel level (not by LPAR) The "ACTIVE" column corresponds to what we refer to as "open exchanges", i.e. the number of I/Os that are simultaneously active within a channel. The “DEFER” column represents the number of deferred FICON or zHPF operations per second that could not be initiated by the channel due to the lack of available resources.
Question 16: How does zHPF impact my configuration planning? Answer: All online CHPIDs to the disk Logical Control Unit must support zHPF. The inclusion of any non-compliant features (for instance FICON Express feature codes 2319, 2320) will result in the entire path group implementing FICON operation, not zHPF. The feature supported by zHPF are FICON Express2 and FICON Express4.
There is potential to achieve some channel aggregation with the adoption of zHPF but careful consideration should be given to meet all your workload requirements. The benefits of zHPF particularly apply to OLTP I/O workloads - DB2, VSAM, PDSE, and zFS which typically transfer small blocks of fixed (4K) data.
Question 17: How would you compare the MIDAW facility with zHPF? Answer: The MIDAW facility and zHPF are completely independent of each other and there are no interdependencies. However, IBM has incorporated many lessons learned from the MIDAW experience into the design of zHPF.
Question 18: Will my configuration still support FICON traffic? Answer: zHPF capable FICON Express4 and FICON Express2 channels and devices support both FICON and zHPF traffic simultaneously. FICON continues to use CCWs and zHPF uses TCWs, so we have compatibility and coexistence. System z High Performance FICON customer readiness documentation Page 12
Question 19: Can I turn zHPF ON / OFF? Answer: Yes, for z/OS exploitation there is a new parameter in the IECIOSxx member of SYS1.PARMLIB (ZHPF=YES/NO) and on the SETIOS command to control whether zHPF is enabled or disabled. The default is ZHPF=NO.
Support is also added for the DIOS, ZHPF command. If DIOS, ZHPF is issued on a server which does not support zHPF, then IOS630I is issued to indicate that zHPF is not supported by the server. Otherwise, message IOS630I is used indicating whether zHPF is enable or disabled. If the zHPF parameter is specified in IECIOSxx or specified on a SETIOS command, then the parameter value is checked to see whether it is YES or NO. If not, message IOS085I xx, rrrr card, ZHPF FACILITY NOT SUPPORTED BY PROCESSOR is issued.
Question 20: Can I determine if zHPF is enabled for my devices? Answer: Yes, the status of zHPF is reflected when the device status is displayed (see the figure directly below).
D M=DEV(410)
IEE174I 11.00.11 DISPLAY M 258 DEVICE 0410 STATUS=ONLINE
CHP A0 A1 A2 A3 DEST LINK ADDRESS A0 A1 A2 A3 PATH ONLINE Y Y Y Y CHP PHYSICALLY ONLINE Y Y Y Y PATH OPERATIONAL Y Y Y Y MANAGED N N N N CU NUMBER 0400 0400 0400 0400 MAXIMUM MANAGED CHPID(S) ALLOWED: 0 DESTINATION CU LOGICAL ADDRESS = 04 SCP CU ND = 002107.000.IBM.TC.02069A00FF04.00FF SCP TOKEN NED = 002107.000.IBM.TC.02069A00FF04.0400 SCP DEVICE NED = 002107.000.IBM.TC.02069A00FF04.0410 FUNCTIONS ENABLED = MIDAW, ZHPF
System z High Performance FICON customer readiness documentation Page 13
Question 21: Can I determine if zHPF is enabled for my channels? Answer: Yes, the status of zHPF is reflected when the channel status is displayed (see the figure directly below).
D M=CHP(A0)
IEE174I 19.38.27 DISPLAY M 343 CHPID A0: TYPE=1A, DESC=FICON POINT TO POINT, ONLINE DEVICE STATUS FOR CHANNEL PATH A0 0 1 2 3 4 5 6 7 8 9 A B C D E F 0041 + + + + + + + + AL AL AL AL AL AL AL AL 0071 + + + + + + + + UL UL UL UL . . . . 0072 + + + + . UL . . UL . . UL . . UL . 0073 + + UL . UL . UL . UL . UL . UL . UL . 1071 ...... UL UL UL UL UL UL UL UL SWITCH DEVICE NUMBER = B000 DEFINED ENTRY SWITCH - LOGICAL SWITCH ID = 20 ATTACHED ND = 006064.001.MCD.01.000000010C17 PHYSICAL CHANNEL ID = 01D3 FACILITIES SUPPORTED = ZHPF
Question 22: Is zHPF being adopted as a Fibre Channel Standard? Answer: IBM has submitted these modifications to the FICON I/O Architecture for inclusion into the Fibre Channel Standard, by the INCITS Fibre Channel (T11) Technical Committee’s for the Fibre Channel – Single-Byte-4 (FC-SB-4) project. System z High Performance FICON customer readiness documentation Page 14
References More detail around the performance aspects of zHPF will be made available at the following address: http://www.ibm.com/systems/z/hardware/connectivity/ficon_performance.html
Acknowledgements Thanks to the following people for their contribution to this document:
Lou Ricci FICON Channel Firmware Team Leader IBM Systems and Technology Group, Poughkeepsie
Connie Beuselinck System z Hardware product planning IBM Systems and Technology Group, Poughkeepsie
Dale Riedy z/OS IOS Design and Development IBM Systems and Technology Group, Poughkeepsie
Cathy Cronin System z I/O performance IBM Systems and Technology Group, Poughkeepsie
Mooheng Zee FICON Channel Microcode development IBM Systems and Technology Group, Poughkeepsie
Mark Bendyk Channel Microcode Development IBM Systems and Technology Group, Poughkeepsie
Parwez Hamid System z Executive IT Consultant IBM Systems and Technology Group IBM United Kingdom
Paul Hagen System z Advanced Technical Support IBM Systems and Technology Group, Poughkeepsie System z High Performance FICON customer readiness documentation Page 15
Sonny Williams Storage Products Performance Evaluation IBM Storage Systems Group, Tucson
Harry Yudenfriend IBM Fellow, Mainframe I/O technology IBM Systems and Technology Group, Poughkeepsie System z High Performance FICON customer readiness documentation Page 16
Copyright IBM Corporation 2009 IBM Systems and Technology Group Route 100 Somers, New York 10589 U.S.A. Produced in the United States of America, 01/2009 All Rights Reserved
IBM, IBM logo, DB2, DS8000, ESCON, FICON, RMF, System Storage, System z, System z10, System z10 Business Class, z/Architecture, z10 BC, z10 EC, and z/OS are trademarks or registered trademarks of the International Business Machines Corporation.
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.
InfiniBand and InfiniBand Trade Association are registered trademarks of the InfiniBand Trade Association. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
All statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.
ZSW03058-USEN-00