Gen 6 Fibre Channel Technology

WHITE PAPER

Better Performance, Better Insight for Your Mainframe Storage Network with Brocade Gen 6

TABLE OF CONTENTS Brocade and the IBM z Systems IO product team share a unique Overview...... 1 history of technical development, which has produced the world’s most Technology Highlights ...... 2 advanced mainframe computing and storage systems. Brocade’s Gen 6 Fibre Channel Technology technical heritage can be traced to the late 1980s, with the creation of Benefits for z Systems and Flash channel extension technologies to extend data centers beyond the “glass- Storage ...... 7 house.” IBM revolutionized the classic “computer room” with the invention Summary...... 9 of the original ESCON Storage Area Network (SAN) of the 1990s, and, in the 2000s, it facilitated geographically dispersed FICON® storage systems. Today, the most compelling innovations in mainframe storage networking technology are the product of this nearly 30-year partnership between Brocade and IBM z Systems.

As the flash era of the 2010s disrupts between the business teams allows both the traditional mainframe storage organizations to guide the introduction networking mindset, Brocade and of key technologies to the market place, IBM have released a series of features while the integration between the system that address the demands of the data test and qualification teams ensures the center. These technologies leverage the integrity of those products. Driving these fundamental capabilities of Gen 5 and efforts are the deep technical relationships Gen 6 Fibre Channel, and extend them to between the Brocade and IBM z Systems the applications driving the world’s most IO architecture and development teams, critical systems. This white paper provides whose close collaboration provides the a summary of these unique mainframe transformative technology to propel features, which reflect Brocade and IBM’s today’s ever-changing data center. dedication to technical excellence. Technology Themes Overview During the past five to seven years, the Partnership technology landscape of mainframe The Brocade and IBM partnership storage networking systems has evolved consists of many teams working to embrace new levels of scale. Not together to produce the best-in-class only are the computational demands products that mainframe users have increasing, but the physical reach of come to expect. The close relationship the systems is also growing. Business continuity and recovery requirements •• IBM Health Checker for z/OS leverages Fabric I/O Priority and FICON multi- have stretched the limits of yesterday’s the internal Control Unit Port (CUP) of hop topologies. These technologies are technology, and the millennial cloud the FICON director to evaluate single- detailed below to provide insight into movement is forcing organizations to point-of-failure conditions as well as Brocade Gen 6/Gen 5 mainframe storage reexamine transactional processing. path integrity and performance. networking functionality. These pressures led IBM to produce •• Problem determination and isolation are Availability Note one of the most innovative and highly enhanced through the Read Diagnostic Included in these descriptions are functioning processors in history—the Parameters (RDP) operations. released, planned, and under- z13. At the same time, Brocade provided development efforts. Please see your the networking dexterity necessary to •• Forward Error Correct (FEC) improves local Brocade representative for details complement the z13’s impact. connection reliability and performance. regarding the Brocade Fabric OS® (FOS) Feature Summary Technology Highlights releases containing a specific feature or The technological leap induced by the z13 Brocade mainframe technology has functionality. is underscored by the inventiveness and produced several “first to market” Speed, Scale, and Performance integration of the Brocade® Gen 5 and innovations, including Gen 6 32 Gbps Inter-Chassis Links Gen 6 feature set: Fibre Channel, Gen 5 16 Gbps Fibre Inter-Chassis Links (ICLs) provide short- Channel with Forward Error Correction, •• Inter-Chassis Links (ICLs) provide distance, chassis-to-chassis connectivity FICON Dynamic Routing, z/OS world-class scalability and flexible for Gen 5 and Gen 6 FICON directors for Health Checker integration, and Read chassis interconnectivity. use with FICON and/or FCP mainframe Diagnostic Parameters functionality. solutions (see Figure 1). This technology •• Multi-hop topologies provide flexibility Leveraging the IBM z Systems IO team is used to build a powerful storage to geographically dispersed data centers partnership, Brocade has also produced networking core without sacrificing to support unique disaster recovery and significant mainframe storage networking device ports for Inter-Switch Link (ISL) business continuity configurations. market exclusives such as ICLs and connectivity. •• 16 GFC FICON Express16S device Port Decommissioning. This shared connectivity and 32 GFC fabric dedication to the success and growth of ICLs minimize the latency between connectivity provide the fastest data the mainframe storage network can also chassis to provide the lowest-latency rates in the industry. be seen in newer endeavors, including switching via a backplane relative to using •• FICON Dynamic Routing utilizes the full capacity of Inter-Switch Link (ISL) and ICL connections in cascaded network topologies, leveraging Brocade Hybrid Exchange-Based Routing (EBR) Local Hybrid technology. AFA •• Fabric I/O Priority capabilities provide HDD Tape a foundation for applications, such as zOS Workload Manager, to prioritize individual IOs for optimal storage AFA network performance. ISLs 16 Gbps •• Port Decommission is used for easier ISLs, 16 Gbps ICLs ICLs 16 Gbps infrastructure management through automated interaction between the fabric Remote and the mainframe IO subsystem. HDD Tape

Figure 1: Inter-Chassis Links versus Inter-Switch Links.

2 comparable ISLs. Frame-based trunking is automatically enabled between four ICLs, which maximizes the load balancing and availability characteristics of the chassis. When used with FICON Dynamic Routing, the IO exchanges are evenly distributed across the ICL infrastructure for optimal performance. Furthermore, since the ICL connection is not considered to be a “hop of concern” in cascaded topologies, it can provide configuration flexibility for large deployments. Multi-hop Topologies The movement to disperse mainframe data centers geographically has forced the evolution of classical FICON SAN topologies. Simple cascaded systems have given way to multi-hop variations Figure 2: Example of multi-hop topology. that exploit Brocade ICL and FCIP channel extension technologies, while maintaining the integrity of the “hop of concern” provision. Advanced diagnostic capabilities, such as the IBM Health Checker for z/OS and the Fibre Channel Read Diagnostic Parameters function, have provided the apparatus necessary to maintain the deterministic reliability required for mainframe storage networks in extended topologies. Multi-hop topologies include two-, three-, and four-site disaster recovery and business continuity configurations (see Figure 2). Each topology is designed to address specific business requirements as well as regulatory provisions. In addition, the supported topologies are verified to operate with the same level of Figure 3: FICON Express16S channel card. predictability and deterministic reliability found in today’s classical cascaded solutions. Gen 5/Gen 6 Performance 32 Gbps Fibre Channel fabric connectivity Availability Note Brocade Gen 5 and Gen 6 FICON provide minimal storage network latency, Brocade and IBM are currently validating directors, shown in Figure 3, provide which significantly enhances data center the most effective topology solutions for optimal interface attachment rates for the application operations and performance. mainframe environments to ensure that FICON Express16S (FX16S) channel as performance and reliability requirements well as storage arrays. The 16 Gbps of z Systems solutions are maintained with Fibre Channel device connectivity and each supported configuration. See your Brocade representative for full details.

3 The faster link speeds provide reduced IO Fabric I/O Priority Availability Note latency for critical middleware IO, such as SAN Fabric I/O Priority integration Brocade and IBM z Systems are exploring database log writing, which significantly translates application importance into the full integration of Fabric I/O Priority improves DB2 transactional latency. Fabric Priority (Quality of Service), which to allow system management functions, The faster link speed also shrinks batch allows the application to maintain the such as zOS Workload Manager, to windows and reduces the elapsed time desired quality of service through the exploit end-to-end, integrated quality of for IO-bound batch jobs. Since faster link fabric. This allows the priority level to be service characteristics. See your Brocade speeds are more sensitive to the quality of communicated between the host and the representative for full details. the cabling infrastructure, new industry- end devices for end-to-end management leading standards are included to provide of application goals (see Figure 5). enhanced error correction on optical connections, ensuring a smooth transition to 16 Gbps and 32 Gbps Fibre Channel technologies. 8 Gbps/ 16 Gbps Disk FICON Dynamic Routing The introduction of the z13 server HDD Disk included several performance 8 Gbps or ISL is assigned at I/0 request time optimizations designed to improve overall 16 Gbps Channel ISLs throughput in the mainframe storage Hybrid Disk network. FICON Dynamic Routing allows FICON devices to utilize both static and dynamic SAN routing policies. It is AFA Disk designed to support the dynamic routing FICON Dynamic Routing (FIDR) policies provided by the FICON director, such as the Brocade Exchange-Based Routing (EBR) protocol. Figure 4: FICON Dynamic Routing. FICON Dynamic Routing enables organizations to use SAN dynamic routing policies across cascaded FICON directors to simplify configuration and capacity Fabric I/O Priority planning, and to provide persistent and repeatable performance and higher resiliency (see Figure 4). In Peer-to-Peer 10110110100110 Remote Copy configurations, the sharing 01100110101011 001100110011011 of switches is simplified, and hardware 00110111101110 costs can be reduced by allowing FICON 110011001101101 and FCP to share the same storage 010101111001100 network infrastructure. 011001110110101 0110011011011 The benefits of FICON Dynamic Routing 11000000111100 are highlighted in the IBM technical reference FICON Dynamic Routing (FIDR) - Technology and Performance WLM sets the Priority Provides I/0 QoS Implications, co-authored by Brocade (see http://www-03.ibm.com/support/ techdocs/atsmastr.nsf/WebIndex/ WP102651). Figure 5: Fabric I/O Priority.

4 Autonomics and Integrity Communication Flows Application to System Automation Port Decommission Application to Device Management Port Decommission is an autonomic, Application to Fabric Control integrated function designed to simplify Target Ports daily port/path maintenance. Routine Control port maintenance that once required Application significant administrative overhead as well as complex operations coordination can now be executed seamlessly without operator intervention. Upon receiving an indication to remove a port or path from active service, Brocade Network Advisor Fabric coordinates the activities with both the host IO subsystem and the fabric. After z/OS sends a confirmation that it has

Hosts Devices migrated the active workload from the designated port/path, Brocade Network Advisor removes the port from operation and indicates it is ready for maintenance Figure 6: Port Decommission. (see Figure 6). Once the maintenance operations have been completed, the port/path is returned to full operational status through the Port Recommission function. As with the decommission operation, Brocade HDD Disk Network Advisor coordinates and communicates the recommission action with z/OS to ensure the operation occurs CUP CUP CUP error-free. Diagnostics IBM Health Checker for z/OS Integration The IBM Health Checker for z/OS extracts information from the FICON director Detecting inconsistent CMR and fabric performance/health issues through the Brocade CUP interface to the switch (see Figure 7). This information IBM Health Checker provides the Health Checker with the ability to identify and report the shared AFA Disk components of the mainframe storage network. The resulting path analysis Figure 7: IBM Health Checker for z/OS integration. provides system administrators with the insight necessary to ensure members of a path group are not susceptible to single- point-of-failure conditions.

In addition to the single-point-of- failure analysis, IBM Health Checker for z/OS leverages the FICON director CUP interface to assess performance

5 characteristics of the members of a path group. The “flow“ information provided includes utilization characteristics, z13 operational conditions, and diagnostic error statistics. This unique solution also incorporates the Brocade Fabric OS® SFP HDD z13 (FOS) bottleneck detection mechanism, Disk SFP which provides alerts to z/OS when fabric anomalies are detected. SFP HDD Enhanced Diagnostics Disk SFP SFP Keeping pace with the ever-growing SFP ISL complexity of mainframe storage networks, Brocade and IBM co-developed in-band diagnostic mechanisms that SFP SFP enrich the view of fabric components. SFP The z Systems channel and control unit SFP solutions include functionality that allows AFA the FICON director to read information Disk SFP about the optical transceiver attached to the z Systems device (see Figure 8). This same functionality is utilized by zOS Hybrid Disk to provide similar information about the transceivers installed in the director. The new Fibre Channel-standard Figure 8: Read Diagnostic Parameters. Read Diagnostic Parameters (RDP) command was created to enhance path evaluation and automatically differentiate between cable hygiene errors and source data failing components. The RDP function Flow provides the optical characteristics of each end of the link, including optical Encoder signal strength and environmental encoder data operating metrics, without requiring manual insertion of optical measurement devices. Furthermore, the RDP operation Bit is performed periodically throughout the errors mainframe storage network to proactively detect marginal conditions that can be received data addressed before they affect production Decoder operations. Forward Error Correction reconstructed data Gen 6 Fibre Channel includes an integrated Forward Error Correction (FEC) technology to provide reliable Figure 9: Forward Error Correction. signal integrity at high data rates (that is, 32 GFC/128 GFC). Gen 5 Fibre Channel offers this same functionality as an optional feature to be exploited

6 at the solution provider’s discretion. Improved Performance with its consistently low latency is able to Brocade includes Gen 5 Fibre Channel Enterprise-class storage performance move data continuously, while the HDD FEC technology with all Brocade FICON has historically focused on four metrics: must continuously return for chunks of director and switch solutions. The IBM latency, response time, throughput data, repeating the mechanical process FICON Express16S channel leverages (bandwidth), and Input/Output Operations and slowing performance. this technology to ensure zOS FICON Per Second (IOPS). There are some Response Time solutions achieve the highest level of end- common misperceptions, including that Latency and response time are often to-end reliability and performance in the the terms “response time” and “latency” confused with each other and are industry. can be used interchangeably, so it is good incorrectly used interchangeably. Gen 6 Fibre Channel to review what each of these metrics Response time is the total amount of time actually measures. Technology Benefits for it takes to respond to a request for service. z Systems and Flash Storage Latency It is the sum of the service time and wait time. The service time is the time it takes While at first glance, given that the Latency is a measure of delay in a system. to do the actual work requested. The wait FICON Express16S channels were GA In the case of storage, it is the time it time is how long the request had to wait in March 2015 on the IBM z13 family of takes to respond to an I/O request. In the in a queue before being serviced, and it processors, the need for Gen 6 and example of a read from disk, this is the can vary greatly depending on the amount 32 Gbps-capable FICON/FCP may not amount of time taken until the device is of IO. Transmission time gets added to be readily obvious. However, considering ready to start reading the block—that is, response time when the request and the reasons Gen 6 Fibre Channel not including the time taken to complete response have to travel over a network, technology was developed in the first the read. In the spinning disk (HDD) and varies with distance. Mainframe place, and the recent z13, Linux ONE, and world, this includes things such as the response time metrics for DASD include flash storage array enhancements, it is seek time (moving the actuator arm to the IOSQ, PEND, CONN, and DISC. Since clear why the Gen 6 technology is a great correct track) and the rotational latency SSD technology is capable of handling fit for many organizations today. When (spinning the platter to the correct sector), significantly more IOPS than HDD, the Gen 5 Fibre Channel made its debut in both of which are mechanical processes, time spent waiting in a queue is typically 2011, many asked similar questions and therefore slow. Latency in a storage shorter for SSD than HDD, leading to about the need for it, given that channel system also includes the finite speed for better response times for SSD. When and storage connectivity was still all light in the optical fiber, and electronic SSD is coupled with Gen 6 Fibre Channel, 8 Gbps, and remained so until 2015. delay due to capacitive components studies have demonstrated a 71 percent Gen 6 Fibre Channel brings three primary, in buses, Host Bus Adapters (HBAs), improvement in response time. immediate technology benefits to a z13 switches, and other hardware in the path. and flash storage architecture: The primary advantage flash storage/ Throughput SSD has over spinning disk technology •• Improved performance Throughput (MB/sec) is the data transfer is the lower latency and the consistency speed. It is the measure of data volume •• Improved Virtual Machine (VM) support in the latency. SSDs can exhibit latency transferred over time—the amount of data that is lower by a factor of 100 compared •• Support for the upcoming NVMe over that can be pushed or pulled though a to HDD technology, and this is primarily Fabrics technology system per second. It is simply a product due to the absence of the mechanical of the number of IOPS and the IO size. processes described above. Equally As a performance metric, throughput important, for random access IO, the SDD matters most for large block-size data

7 traffic. In a mainframe environment, these The IBM z13 family of mainframes uses arrays, but once mainframe storage array would be batch-type processing and the latest industry-standard Peripheral vendors such as EMC, HDS, and IBM data warehousing applications. Batch Component Interconnect Express brought all flash arrays to the market, processing handles requests that are Generation 3 (PCIe Gen3) technology for end users began migrating to the all- cached (prestored) and then executed all its IO subsystem (IOS). It also supports flash array technology. The reduction in at the same time. An example of batch 50 percent more IO devices per FICON latency and improved response times processing is the reconciliation of a bank’s and FCP channel (32,000 IO devices that can be realized by implementing ATM transactions for a given day being per channel), with a maximum capacity of flash storage array technology allow compared against all the customers’ bank 320 FICON channels per z13. A FICON for faster processing of transactions, or records, and then reporting the ATM Express16S channel is capable of 93,000 more transactions processed for a given transaction records at a time outside z High-Performance FICON (zHPF) IOPS time interval. For some industry verticals, normal business hours. Higher throughput running 4,000-byte block-size traffic this can translate directly into more capabilities, such as Gen 6 Fibre typical of most OLTP workloads. In sum, revenue. A TABB Group study from July Channel’s 32 Gbps, will lead to shortened the z13 processors are IO “monsters.” 2010 determined that: “A 1-millisecond batch windows, which can be invaluable They were designed to handle massive advantage in a trading application can to many large z Systems customers (the amounts of IO due to the increases in be worth US$100 million a year to a infamous race to the sunrise). When IBM IO demands brought about by what IBM brokerage firm.” introduced FICON Express16S channels refers to as CAMS (Cloud, Analytics, Improved VM Support in 2015, one of the primary cited benefits Mobile, and Social) workloads. For many The IBM z13 has up to 141 cores compared to FICON Express8S channels end users, mobile workloads and their IO using the world’s fastest commercial was an average 32 percent reduction in requirements alone are driving the need processor, running at 5.0 GHz, to enable the elapsed time (wall clock) of the batch for flash storage attached to z13. high performance coupled with massive window. Brocade Gen 6 technology provides a scaling. The z13 and the IBM LinuxONE IOPS significant leap in IOPS capabilities. The Emperor systems take scalability for VMs For a modern mainframe environment, Condor 4 ASIC is capable of handling to a completely new level and are capable throughput is secondary in importance to 33 percent more IOPS than the prior of supporting up to 8,000 virtual servers. a storage device’s IOPS capability. Flash generation Condor 3 ASIC used in When connected to flash storage via FCP storage/SSD technology is capable of Brocade Gen 5 SAN technology. Brocade channels and Brocade SAN hardware, significantly more IOPS than spinning Gen 5 FICON SAN technology is capable such as Brocade X6 Directors, the z13 disk technology. Mainframes are primarily of outstanding performance in a z13 and the LinuxONE Emperor support Node used for Online Transaction Processing flash storage architecture. Brocade Port ID Virtualization (NPIV). This support (OLTP) workloads. Transaction processing Gen 6 FICON SAN technology takes of NPIV has increased significantly with consists of large numbers of small this performance to a whole new level z13 and LinuxONE Emperor, doubling block-size data IO transactions, often and will allow flash/SSD to reach its full to support 64 virtual images per FCP seen in banking, ATMs, order processing, performance potential. channel from the prior support of reservation systems, and financial The enterprise computing space 32 virtual images per FCP channel. The transactions. The most common block- has been at the forefront of the flash IO subsystem of these IBM servers is built size traffic seen in today’s mainframe storage revolution. Applications that are on Peripheral Component Interconnect environment is 4,000 bytes, which performance-critical have been/are being Express (PCIe) Gen 3 technology, which coincidentally is used for most storage moved from the traditional spinning disk can have up to 40 PCIe Gen 3 fanouts performance benchmarking tests, as DASD arrays to flash storage arrays. and integrated coupling adapters per well as z Systems FICON channel This movement started with hybrid system at 16 GBps each (yes, 16 GB, performance testing. not 16 Gbps). The z13 also supports 10 TB of available RAM (3x its predecessor). This massive amount

8 of memory, as well as server and IO technology is expected to improve along Summary virtualization, allows VMs and applications two dimensions over the next couple The enterprise computing community to run at their full, unconstrained potential. of years: improvements in latency and represents the unique sector of the Gen 6 Fibre Channel is ideal for these the scaling up of the number of NVMe storage networking market that propels environments and is capable of supporting devices in large solutions. the most critical systems in the world. the greater virtualization levels at sustained In 2014, the Fibre Channel Industry These systems must operate cohesively high levels of performance, enabling Association (FCIA) announced a new and under extremely stringent conditions full utilization of the z13 and LinuxONE working group within the INCITS T11 in order to provide the everyday services Emperor virtualization capabilities. committee (responsible for Fibre Channel and operations enjoyed by billions. In the future, zLinux environments standards) to align NVMe with Fibre Brocade and the IBM z Systems IO team connected to storage via Brocade Fibre Channel as part of the NVM Express over share an extraordinary relationship that has Channel technology will be able to gain Fabrics initiative. This was an important yielded countless enhancements to the insight into application performance evolution because it kept Fibre Channel world of enterprise computing. This nearly ® using Brocade Fabric Vision VM Insight at the forefront of storage innovation. FC- 30-year bond has stood the test of time, capabilities. NVMe defines a common architecture that as well as the turmoil of the high-tech supports a range of storage networking Support for Non-Volatile Memory marketplace, to produce reliable solutions fabrics for NVMe block storage protocol Express over Fabrics using Fibre for these critical systems. At a time when Channel over a storage networking fabric. This most companies are boasting of alliances includes enabling a front-side interface Non-Volatile Memory Express (NVMe) is around the technology “flavor of the into storage systems, scaling out to a new and innovative method of accessing day,” Brocade and IBM z Systems have large numbers of NVMe devices, and storage media, and has emerged as the delivered the technology for the ages. extending the distance within a data center new storage connectivity platform that Brocade builds on that 30 years of over which NVMe devices and NVMe will drive massive performance gains. mainframe leadership to deliver the subsystems can be accessed. It is ideal for flash/SSD. Applications industry’s highest performance and most will see better random and sequential FC-NVMe offers compelling performance reliable and scalable FICON infrastructure. performance by reducing latency and that is synergistic with the SSD and With seamless FICON connectivity enabling much more parallelism through mainframe technologies that demand it. and support for innovative features an optimized PCIe interface purpose-built Gen 6 Fibre Channel is NVMe-ready and that only Brocade can offer—including for solid-state storage. The momentum offers the ideal connectivity for high- ClearLink® D_Port, CUP Diagnostics, Port behind NVMe has been increasing since performance mainframe and flash storage. Decommissioning, and ISL encryption— it was introduced in 2011. In fact, NVMe organizations can achieve the full potential from new flash storage and IBM z13 and future mainframe technology. Learn more at www.brocade.com.

Corporate Headquarters European Headquarters Asia Pacific Headquarters San Jose, CA USA Geneva, Switzerland Singapore T: +1-408-333-8000 T: +41-22-799-56-40 T: +65-6538-4700 [email protected] [email protected] [email protected]

Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.