WHITE PAPER
Brocade CUP Diagnostics and the IBM Health Checker for z/OS
TABLE OF CONTENTS The IBM FICON Control Unit Port (CUP) provides an enhanced
Introduction...... 2 communications connection between IBM z Systems hosts and FICON switching devices, over which in-band management and communications Traditional Uses of the FICON Cup...... 2 can occur. Within z Systems, there are management programs that IBM and Brocade Team Up to can supervise switching devices by communicating through CUP to Create CUP Diagnostics...... 3 those switches. First Wave of CUP Diagnostics Enhanced Capabilities with A relatively recent enhancement to this CUP communications capability is Brocade FOS 7.2...... 3 generally known as the CUP Diagnostics function. This enhancement is a
Brocade FOS 7.3 Enhancements collaborative effort between IBM and Brocade to provide z Systems with to the CUP Diagnostic Capability...... 6 visibility into fabric events that the z/OS operating system has historically been unaware of. Brocade FOS 7.4 Enhancements to the CUP Diagnostics Capability...... 11 As z Systems environments have grown, and configurations have become more MAPS Trigger Values and HSC complex, FICON fabric issues can result in unacceptable Input/Output (I/O) service Codes and Text That Are Sent times. Resource Measurement Facility (RMF) device activity reports might show average to the Z/OS Console...... 13 service times that are higher than normal, while matching I/O queuing reports show abnormally high “initial command response” times on a subset of the paths to a device. About Brocade...... 16 Before these CUP Diagnostic enhancements were introduced, it was problematic to identify a single root cause for these issues or to identify where in the fabric a problem might have originated.
CUP Diagnostics enables the ability for the CUP to proactively notify the z System that there are abnormal conditions in the fabric that might impact either the performance or reliability of the I/O traffic traversing through the fabric.
This paper reviews the basic functionality of FICON CUP and then introduces the new FICON Management Server (FMS) and CUP Diagnostics capabilities now available to be used by the IBM Health Checker for z/OS to gain better insight into the health and robustness of FICON Storage Area Network (SAN) fabrics. Introduction ESCON Directors. The IBM 9032-5 credit starvation) and many other ESCON Directors had an in-band analysis capabilities. The Control Unit Port (CUP) function management capability that utilized an allows a z/OS system to communicate RMF provides online, interactive embedded port in the control processing with the FICON Director through performance monitoring and long-term cards to provide a communications path standard channel program protocols. overview reporting with post-processor to an MVS console. This was used for This includes control functions like reports. Some RMF reports that can three primary purposes: first, reporting blocking and unblocking ports, assist in analysis of fabric problems are: hardware (Field-Replaceable Unit [FRU]) performance monitoring, and error errors up to MVS (helpdesk), second, Channel path activity report reporting functions. •• allowing and prohibiting ports (the ••Device activity report The CUP device is implemented as a world’s first “zoning” mechanism) by I/O queueing activity report standard device control unit via firmware using Prohibit Dynamic Connectivity •• implementation on a Brocade switch that Mask (PDCM), and third, basic ••FICON Director activity report allows a z/OS system to issue appropriate performance monitoring. When switched Enterprise Disk Systems (ESS) Link channel program Channel Command •• FICON was being introduced, IBM Statistics report Word (CCW) commands to it. The CUP wanted to make certain that its mainframe device is defined in the I/O configuration customers would have a consistent However, RMF is not so effective at as a switch device and is brought online management look and feel between helping a user troubleshoot problems to z/OS. The control unit definition for ESCON and FICON, so CUP was carried such as fabric congestion, device the CUP device consists of one or more forward to FICON. congestion, bad cables, failing optics, channel paths attached to the switch with or FRU problems in a switched fabric. Today, FICON CUP support is provided the reserved port address destination What is really needed is broader by all mainframe storage and SAN of 0xFE, which is defined by the FICON architecture that defines a richer set vendors, but customers must implement architecture as the address of the CUP. of data that can be supplied from the optional FMS license to make it Therefore, I/O requests routed to this fabric components to the z Systems functional on their FICON switches and destination port are directed to the CUP. components. That can be accomplished directors. Today’s more advanced CUP is through a tighter integration of CUP on The FICON CUP provides an in-band still used for the three primary functions switching devices and enhanced host- management interface defined by IBM listed above, but it has been enhanced based management programs. that defines the CCWs and data to also provide RMF reporting for the payloads that the FICON host can use FICON switching devices and to help Host-based management programs can for managing the switch. The protocol implement FICON Dynamic Channel Path manage FICON switches by sending used is the IBM version of the ANSI Management (DCM). CCW commands to the switch control FC-SB4 single-byte command code unit defined in the I/O Configuration Data specification, which defines the protocol Traditional Uses of the Set (IOCDS) and Hardware Configuration used for transporting CCWs to the FICON Cup Definition (HCD). A FICON switching switch, and for the switch to direct data device (that is, Director-class or fixed FICON CUP provides a defined CCW and status back. FICON CUP becomes configuration switch) that supports CUP and payload interface for host in-band available when the FMS optional license is can be controlled by one or more host- management and collection of FICON installed and enabled on FICON switching based management programs or switch switch performance metrics, resulting devices such as the Brocade® DCX® 8510 consoles. Control of the FICON switches in the collection of RMF 74 subtype 7 Backbone and the Brocade DCX 6510 can be shared between these options. records, more commonly known as Switch, regardless of which vendor sells CUP commands, which are channel the FICON Director Activity Report. these devices. command words (CCWs), monitor and Through the use of this record and report control FICON switch functions. There FICON CUP is a direct architectural a user can gain a good understanding are 42 CUP commands, or CCWs, for descendant of the CUP that ran on about frame pacing delay (that is, buffer monitoring and control of FICON switch
2 device functions. CUP commands including both the Computer Electronics Brocade FOS v7.2 initiated the CUP have generally been oriented towards Complex (CEC) and its peripherals. The Diagnostics function by providing some management of a single switch even IBM Health Checker for z/OS reports basic, but limited, responses to the z though the use of CUP in a cascaded problems to various system operations Systems host. FICON environment is fully supported. components (that is, automation components) and initiates further analysis First Wave of CUP It has become very apparent over time actions and remedial actions. that as fabrics become more complex it Diagnostics Enhanced is essential that they become proactively FICON switching devices (that is, a Capabilities with monitored by a z System. Although very storage networking fabric) sit in the Brocade FOS 7.2 mature and robust by most standards, middle, between the computing and In March 2014, two new functions became there are a few challenges in the way storage components, so switches have a available in Brocade FOS v7.2 that were that z/OS has traditionally defined and critical role in transporting data between engineered into the CUP functionality. worked with fabrics. For example, the z/ the CEC and its peripherals. But, as ROUTE and HEALTH keywords were OS IOCDS definitions include F_Ports, described earlier, the preponderance provided, in order to be functional with relative to Control Units and Devices, of fabric operations are invisible to z/ the Display Matrix z/OS command. but there are no definitions for Inter- OS, which cannot quickly react to or While z/OS has historically provided Switch Links (ISLs), routing topologies, or troubleshoot those issues. sophisticated I/O monitoring and distance extension facilities. Subsequently, Errors within the fabric can be caused by recovery, this modified command these fabric components are essentially either software or hardware problems or exposes newly available fabric diagnostic invisible to z/OS and z/OS can do nothing by frame traffic rates that exceed available data, provided directly from the switch, to react to specific problems in the fabric. capacity. Some of the potential issues that and makes some of the fabric operations Rather, z/OS just suffers performance can cause problems in the fabric are: become evident that previously were degradations and reliability exposures until invisible to z/OS. the problems manifest in a secondary ••Defective cables or optics manner, such as a serious degradation ••Congestion within the fabric The ROUTE function provides important of I/O performance or reliability, that it is information back to the command issuer I/O or switch configuration problems, capable of detecting. CUP Diagnostics is •• regarding the specific path that frames are such as incorrect cabling or routing a facility that enables the CUP to notify traversing between a mainframe Channel the IBM Health Checker for z/OS host ••Channel or control unit hardware or Path ID (CHPID) and a device (that is, management program about fabric events firmware issues the to device TODEV keyword) or from a device to a mainframe CHPID (that is, while also supporting subsequent queries I/O configuration definition errors by the host to provide detailed information •• the FROMDEV keyword). In a parallel about its fabric topologies (that is, Now IBM, in a cooperative effort development effort, two new IBM z/OS domains, ISLs, routing groups) and the with Brocade, has defined the CUP Health Checks were added so that z/OS location of, and details of, errors within Diagnostics function, a new architectural could recognize some specific problems those topologies. component to be implemented in the within the fabric. These functions are Brocade Fabric OS® (Brocade FOS) FMS supported on all switches beginning IBM and Brocade Team Up code. CUP Diagnostics are the newest with Brocade FOS v7.1.0c and Brocade to Create CUP Diagnostics CUP CCWs and payloads that enable Network Advisor 12.0.2 software levels the z Systems host to receive direct or later. The Health Checker for z/OS is an IBM alerts about fabric problems and then software component that is used to On the z Systems side, z/OS 1.12 and later, proactively gather additional, detailed monitor and detect performance and plus some required Authorized Program information about the fabric (for example, Reliability, Availability and Serviceability Analysis Reports (APARs) and Program components, topologies, ISL routing, and (RAS) issues in data center components Temporary Fixes (PTFs), are required with fabric problems). Brocade FOS v7.1.0c to support the CUP
3 Diagnostics and IBM Health Checker for To make good use of CUP Diagnostics, performed only when the channel is z/OS features added at that firmware level. the user interface is often the ROUTE connected to a switch and the control unit enhancement to the z/OS Display Matrix definition for the channel path is defined in Important Note: Even if the customer command. The DISPLAY M=DEV the I/O configuration with a two-byte link has no plans to use these new CUP command allows a user to display the address (that is, cascaded FICON). features, the IBM Health Checker for z/ route through the SAN fabric for a specific OS feature uses a previously undefined See the example in Figure 1. Here, a user device and channel path, by specifying bit in the Sense data. If the user does wants to see how frames are being sent the ROUTE keyword. The routing not implement these APARs and PTFs, from the mainframe (CHPID 09) through information returned includes all of the then any time the channel issues a Sense the fabric in order to reach one of the domains and ports that are in the path command to the CUP, and there is health disk arrays, A000. The user also wants from the channel to the device. Since the information to report, an error message a report about the health of the fabric usual intent is to find the frame routing indicating an invalid command code is along that path. The user issues the that is being used in a cascaded fabric, posted to the z/OS console. This might Display Matrix command with a ROUTE the reporting of routing information is become annoying to the user. parameter and a HEALTH parameter. You can see the output that is displayed when the CUP Diagnostics function is executed on their behalf via the Display Matrix command.
(Aggregate-F8) In the routing portion of the display output, 4 ISL Links CUP CUP information for each switch in the path is F8, F9, FA, FB 8, 9, A, B displayed, which includes its domain and 22 E0, E1, E2, E3 10, 11, 12, 13 B3 switch type. The definition of the switch
Port (Aggregate-E0) Port type depends upon the number 45 4 ISL Links 60 of switches in the path between the CHPID 09 channel and device, and the position of UNITADD the switch within the path. If there is only A000 one switch, the switch type is shown as Only Director. If there is more than one switch, the first switch is known as the Figure 1 Source Director, and the last switch is Figure 1: CHPID 9 using entry point 45 on CUP22 to send and receive frames to CU A000. known as the Destination Director. If there are any switches between the source director and the destination director (see Refer to Figure 1: Issuing the command: D M=DEV(A000,(09)),ROUTE=TODEV, Figure 2), each of those switches is known HEALTH IEE583I hh.mm.ss DISPLAY M 058 DEVICE sdddd STATUS=ONLINE as an Intermediate Director.
From the example output, you can see Source to destination routing information follows: that the route from the channel to the device travels through two switches, Switch Domain=22, Type=Source Director Domain ID 22 (that is, source director) Group Port Type From To Agg Dyn Speed Misc and Domain ID B3 (destination director). 45 Entry Chan Agg-F8 .. .. 16G Static The channel is connected to entry port 45 80 Exit A000 B3OA .. .. 16G on switch Domain ID 22, as indicated by
the “From” column for that port. The “To” Switch Domain=B3, Type=Source Director Group column for the same port indicates that Port Type From To Agg Dyn Speed Misc I/O requests originating from this port are 0A Entry 22FA B-360 F8 .. 16G Static 60 Exit A000 B3OA .. .. 16G .....
4 second ISL aggregation for Domain ID 22 (Aggregate-F8) aggregation-E0. 4 ISL Links If the command “D M=DEV(A000,(09)),ROUTE=FROMDEV, F8, F9, FA, FB 08, 09, 0A, 0B HEALTH” is issued, then the trunking 22 B3 aggregations are for connections from E0, E1, E2, E3 Domain ID B3 flowing frames to Domain CUP CUP 10, 11, 12, 13 ID 22. Its first ISL aggregation is Domain ID B3 aggregation-08, and the second (Aggregate-E0) ISL aggregation is Domain ID B3 Port 4 ISL Links Port 45 60 aggregation-10.
UNITADD The speed listed is the negotiated speed, CHPID A000 09 in gigabits per second. (Reports only the Master ISL Link You can interpret the information for from the trunking aggregation) the destination director B3 in a similar manner. Entry port 0A is within the aggregation group F8, and it routes the I/O requests to port 60 on the same switch. Once again, there is no distinction Figure 2: CHPID 9 could use a link on one of two hardware trunks to send and receive frames to between trunked and nontrunked ISL CU A000. connections. Therefore, only a single ISL link is shown in the output chosen from routed to ISL aggregate FA (that is, four For exit ports, the logical connection the FSPF list. trunked links, as in Figure 2). Brocade appears under the “From” column and Although port 60 has data routed to FOS v7.2 did not make any distinction always represents a single entry port on it from multiple ISL entry ports on the between trunked and nontrunked ISL the same switch. The physical connection switch, it is not possible to indicate that to connections. Therefore, only a single appears under the “To” column and the user in this version of Brocade FOS. ISL link is shown in the output. Fabric represents either a channel, control unit, or Port 60 is not part of an aggregate group, Shortest Path First (FSPF) provided CUP a single port on another switch. therefore it contains “..” in that column. The Diagnostics with a list of possible egress Brocade FOS v7.3 and beyond provide storage device control unit is connected to ports, and CUP Diagnostics simply chose additional information about individual port 60. the first egress port shown in that list as trunk ISL links that Brocade FOS 7.2 does Note that in this static routing example the master ISL link. not provide. If you examine Figure 2, you (that is, Port-Based Routing [PBR]), the For entry ports, the physical connection see that there are two trunks of four ISL information in the “Misc” column indicates appears under the “From” column and links each. Each trunk is an aggregation of that I/Os are being statically routed. In represents either a channel, a control ISL links and is identified using the lowest the version of CUP Diagnostics for unit, or a single port on another switch. port number in the group. In this case, the Brocade FOS v7.2, there is no recognition The logical connection appears under source switch is Domain ID 22, so the of Device-Based Routing (DBR) or the “To” column and represents a single aggregation designations are for Domain FICON Dynamic Routing (that is, port or a group of ports on the same ID 22 flowing frames to Domain ID B3. Exchange-Based Routing) capabilities on switch, depending on the routing and That makes the first ISL aggregation for ISL connections. grouping methods. Domain ID 22 aggregation-F8 and the
5 The following output shows the “health” display for the example above: In the previous version of CUP Diagnostics (that is, Brocade FOS v7.2) Health information follows: only the first FSPF link of an ISL
Fabric Health=No health issues trunking aggregation was listed in the report that was provided in response Switch Domain=22, Health=No health issues to a CUP Diagnostics query. With the %Util %Delay Error Count Opt Signal Port Health Trn/Rcv Trn/Rcv Trn/Recv Trn/Recv release of Brocade FOS v7.3, CUP 45 Port Normal 99/99 0/0 0/0 -4.8/-6.6 Diagnostics was capable of providing a FA Port Normal 99/99 0/0 0/0 -4.9/-6.6 much more robust display of all of the Switch Domain=B3, Health=No health issues members of an ISL aggregation/trunk. %Util %Delay Error Count Opt Signal Port Health Trn/Rcv Trn/Rcv Trn/Recv Trn/Recv A common Aggregation Group Number 0A Port Normal 99/99 0/0 0/0 -2.1/-2.4 (AGN) is assigned to indicate which 60 Port Normal 99/99 0/0 0/0 -2.4/-2.1 ISL links are contained within the same
! trunk aggregation. Each port in the trunk aggregation is displayed with its For the health portion of the display, a is switch-vendor-specific. If any data own statistics. description of the health of the fabric, each is not valid, then ”..” is displayed in the switch, and each port is displayed, as well appropriate column. As an example, referring to Figure 3, a as the following information for each port: user would like to see how frames are Brocade FOS 7.3 being sent from the mainframe (CHPID The % transmit/receive utilization •• 09) through the fabric, in order to reach indicates the percent utilization of the Enhancements to the CUP one of their disk arrays, A000, similar to total transmit or receive bandwidth Diagnostic Capability what was done at Brocade FOS v7.2. The available at the port. In October 2014, Brocade FOS v7.3.0b user also wants a report about the health became available for mainframe ••The % transmit delay is the percent of of the fabric along that path. The user enterprises, and it hosted a plethora of time that the frame transmission was issues the same Display Matrix command new capabilities for CUP Diagnostics. delayed because no buffer credits were with a ROUTE and a HEALTH parameter. Among its enhancements was the ability available on the port. The % receive You can view the enhanced output to provide information about complex delay is the percent of time that the port displayed from CUP Diagnostics, via the routing topologies. This section describes was unable to receive frames because Display Matrix command, as follows. all receive frames were utilized. these changes. ••The error count is the number of errors detected on the port that affect the transmission or receipt of frames on the port. This is a sum of the errors counted (Aggregate-F8) over the fabric diagnostic interval, which 4 ISL Links CUP CUP is set to 30 seconds by z/OS. F8, F9, FA, FB 8, 9, A, B 22 B3 ••The optical signal column indicates the E0, E1, E2, E3 10, 11, 12, 13 signal strength of the fiber optic signal Port (Aggregate-E0) Port being transmitted/received by this 45 4 ISL Links 60 CHPID port, in units of dBm (that is, denotes 09 an absolute power level measured in UNITADD decibels and referenced to 1 milliwatt). A000
This example of the health data reveals that there are no health problems within the fabric. The text provided in the fabric Figure 3: Trunk (that is, aggregation) groups are assigned the lowest port in the trunk as the health, switch health, and port health group number.
6 Referring to Figure 3: Issuing the command: D M=DEV(A000,(09)),ROUTE=TODEV, From the example output, you can see HEALTH that the route from the channel to the device travels through two switches, IEE583I hh.mm.ss DISPLAY M 058 Domain IDs 22 and B3. Domain ID 22 DEVICE sdddd STATUS=ONLINE is the source director, and Domain ID B3 is the destination director. The channel Source to destination routing information follows: is connected to entry port 45 on switch Switch Domain=22, Type=Source Director Domain ID 22, as indicated by the “From” Group column for that port. The “To” column for Port Type From To Agg Dyn Speed Misc 45 Entry Chan Agg-F8 .. .. 8G Static Alt=1 the same port indicates that I/O requests F8 Exit A000 B3O8 F8 .. 16G originating from this port are routed to ISL F9 Exit A000 B3O9 F8 .. 16G aggregate FA (that is, four trunked links; FA Exit A000 B3OA F8 .. 16G see Figure 4). From the “Agg” column, FB Exit A000 B3OB F8 .. 16G you can also determine that aggregate F8 consists of exit ports F8, F9, FA, and Switch Domain=B3, Type=Source Director FB on the switch. Each of these exit ports Group is connected to a different port on switch Port Type From To Agg Dyn Speed Misc 08 Entry A001 B360 .. .. 8G Static Alt=1 domain B3, as shown in the “To” column 09 Exit A002 B360 F8 .. 16G Static Alt=1 for the ports. For example, port F9 routes 0A Exit A003 B360 F8 .. 16G Static Alt=1 I/O to port 09 on domain B3. 0B Exit A004 B360 F8 .. 16G Static Alt=1 60 Exit Mult CU F8 .. 16G ..... When a port is connected to a channel, ! “Chan” appears under the “From” or “To” columns of the display, depending on which direction was specified. Likewise, (Aggregate-F8) 4 ISL Links when a port is connected to a control unit, “CU” appears in the display.
F8, F9, FA, FB When an entry or exit port is connected 08, 09, 0A, 0B to a single switch port, the domain and 22 B3 port number are displayed as a four-digit E0, E1, E2, E3 number under the “From” or “To” columns. CUP 10, 11, 12, 13 CUP When an entry port is connected to multiple ports on the current switch, a (Aggregate-E0) Port dynamic or aggregate group number 4 ISL Links Port 45 60 is displayed under the “To” column, UNITADD depending on the routing and grouping CHPID A000 09 methods used. When a set of entry or exit ports are part of an aggregate group of ports, those (Better detail on the aggregated links of a hardware trunk) ports are assigned an aggregate group number. This group number appears in two places. First, it appears under the “Agg” column, to show which ports make Figure 4: A more detailed look at the ISL links contained in the Aggregation Groups. up the aggregate group. Second, if I/O requests are being statically routed from an entry port to this aggregate set of ports, then the aggregate group
7 number appears under the “To” column of When an entry port uses static routing, data is not valid,”..” is displayed in the the entry port. The value that is assigned the exit port or aggregate group number appropriate column. for the aggregate group number is switch- assigned to the port and the numbers of Note: The errors above were vendor-specific. alternate paths are displayed. Detailed manufactured by the author so that the information about the alternate paths is Note that the definition of what is reader could see some of the possible not displayed. considered an entry port or exit port error indications that a user might depends on the direction of the display Note that, in this static routing example experience during actual operations. request: channel to device (TODEV) (that is, PBR or DBR), the data for or device to channel (FROMDEV). For aggregate E0 is not displayed; however, Blind Director Enhancement to example, when TODEV is specified, the the information in the “Misc” column CUP Diagnostics port that is connected to the channel is indicates that I/Os are statically routed. A Blind Director is a domain where the considered an entry port, and the port One alternate route is defined, which is CUP is not capable of processing Director connected to the control unit is considered the aggregate E0 routing path. FICON Diagnostics Query messages, because an exit port. However, if FROMDEV is Dynamic Routing (that is, Exchange- its firmware level is prior to Brocade FOS specified, the roles are reversed. Based Routing) is not supported at v7.1 or it does not have FMS mode (CUP) Brocade FOS v7.2, so only static routing enabled. See Figure 5 to view these two If the command, “D is reported by v7.3 CUP Diagnostics. conditions. If a CUP Diagnostics query, M=DEV(A000,(09)),ROUTE=FROMDEV, where a domain after the Source domain HEALTH” is issued, then the trunking This example of the health data shows is blind and incapable of responding to aggregations are for connections from that there are some health problems a CUP Diagnostic query request, it will Domain ID B3 flowing frames to Domain within the fabric on both switches, as return a code indicating the domain was ID 22. Its first ISL aggregation is Domain shown by the fabric and switch text Port down-level or that FMS mode was off. ID B3 aggregation-08, and the second Error. You see a combination of bottleneck ISL aggregation is Domain ID B3 detected errors, slow draining device Brocade FOS v7.3 implemented several aggregation-10. detected errors, and some ports that improvements to the ability to handle and have been fenced. The text provided in report a Blind Director in the routing path The speed listed is the negotiated speed, the fabric health, switch health, and port of a Director Diagnostic Query. in gigabits per second. health is switch-vendor-specific. If any The information for the destination director B3 can be interpreted in a similar The following output shows the “health” display for the example above: manner. Entry ports 8, 9, A, and B are all within aggregate group F8, and they route the I/O requests to port 60 on the same Health information follows: switch (that is, Domain ID B3). Notice Fabric Health=Port Error that port 60 can have data routed to it Switch Domain=22, Health=BRD020-Port Error from multiple entry ports on the switch. %Util %Delay Error Count Opt Signal Because there is not a single originating Port Health Trn/Rcv Trn/Rcv Trn/Recv Trn/Recv 45 BRP031-Bottleneck Detect 99/0 0/0 0/0 -4.8/-6.6 port to identify, the “From” column F8 No health issues 32/44 4/0 0/0 -4.9/-6.6 contains “Mult”. Port 60 is not part of an 09 BRP030-Port Fenced 0/0 0/0 0/0 ../-6.6 FA No health issues 32/44 4/0 0/0 -4.9/-6.6 aggregate group and therefore contains “..” FB No health issues 32/44 4/0 0/0 -4.9/-6.6 in that column. The storage device control unit is connected to port 60. Switch Domain=B3, Health=Port Error One of the major changes in this release %Util %Delay Error Count Opt Signal Port Health Trn/Rcv Trn/Rcv Trn/Recv Trn/Recv of CUP Diagnostics is that both PBR and 08 No health issues 0/0 0/0 0/0 -2.5/-3.8 BRP030-Port Fenced DBR are noted as “Static” under “Misc” 09 0/0 0/0 0/0 ../-3.3 0A No health issues 33/0 7/0 0/0 -2.4/-2.2 in the report. DBR was not supported in 0B No health issues 33/0 4/0 0/0 -3.9/-2.7 BRP033-SlowDrain Detect CUP Diagnostics at Brocade FOS v7.2. 60 0/0 95/0 0/0 -2.5/-3.8 !
8 FOS FOS FOS FOS 22 7.3 7.0 B3 22 7.3 7.3 B3
CUP CUP CUP NO CUP
Port Port Port Port Source 45 Destination Source Destination switch switch 60 switch 45 switch 60 CHPID UNITADD CHPID 09 09 UNITADD A000 A000
• Blind Director due • Blind Director due to to down-level FMS disabled on the Brocade FOS destination switch
Figure 5: Blind Director is not capable of processing Director Diagnostics Query messages due to inadequate firmware levels (Left) and not having CUP enabled (Right).
Important Note: A user might have switches at or close to the same Brocade specifically to enable the reporting of the FMS license installed on all FICON FOS level. this routing information. It has Fabric switching devices. Then during an Diagnostic Queries that are initiated in two upgrade from a previous version of Diamond Topology Enhancement ways: by a console operator, in response Brocade FOS to v7.3, High Integrity to CUP Diagnostics to a Health Summary Code (HSC) Fabric (HIF) does not get set. At Brocade Previous versions of CUP Diagnostics unsolicited alert received by the host, FOS 7.3, the FMS mode is dependent were limited to query paths with domains or by z/OS workload manager, when it upon HIF to be enabled. If for any reason in a linear configuration, as shown in detects a performance problem that might HIF is not enabled on a Brocade FOS 7.3 Figure 6 on the following page. be attributable to fabric performance. switch then FMS will be disabled and a In multidomain fabrics, typically there Brocade FOS v7.3 supports CUP query by CUP Diagnostics will treat it as a is FICON traffic that traverses the ISLs Diagnostic queries where the shortest “Blind Director”. between the domains (that is, Fibre path from the Source domain to the For a cascaded CUP Diagnostics query, Channel or Fibre Channel over IP (FCIP) Destination domain can contain multiple where the Source and Destination ISL links). The Brocade 78xx Series of subpaths that all have an equal path cost. domains and the Source port are valid, Switches, which are distance extension So a “Diamond” topology (see Figure 7), and one of the domains after the Source and business continuance switches, is one where there are two ISL paths from domain is down-level (Brocade FOS v7.0 always place intermediate domains (for the Source domain (that is, the CHPID or earlier) or FMS mode is disabled, CUP example, 51/71 in Figure 6) between the attached to an F_Port) to the Destination Diagnostics indicates that the domain is two endpoint switch domains. As stated domain (that is, F_Port leading to the incapable of allowing the query. Brocade before, the z/OS software has no visibility storage control unit), and both of the ISL FOS v7.3 handles domains that do not into the routing information (that is, routing paths have an equal cost. support CUP Diagnostics by supplying policy, number of ISL paths available, It is prudent, at this point, to provide a an appropriate reason code (for example, link utilization, or link degradation). The brief refresher of ISL equal-cost paths. lower firmware level or FMS mode being IBM Health Checker for z/OS, through An important benefit of Fibre Channel off). Customers typically have cascaded the use of CUP Diagnostics, is designed
9 Brocade 7840 Brocade 7840
SID = Source ID DID = Destination ID
Figure 6: Linear Configurations. is the possibility of using multiple ISLs the link and the number of hops between Linux on z Systems, doing SCSI I/O for between switches. There is no need the source and the destination switching example, are allowed to make more than to worry about loops and no blocked device. So, in order to have equal-cost ISL one hop to its storage. links, just additional ISLs that provide paths, every ISL must be enabled to run at A Diamond topology is also relevant bandwidth and redundancy. If a user adds the same data rate and to make the same to fabrics that utilize Exchange-Based more ISLs between switches, FSPF cares number of hops. Routing (that is, FICON Dynamic Routing about this topology change. FSPF is a It is also important to note that even [FIDR]). The reason for this is that with routing protocol used in Fibre Channel though FICON is currently qualified static routing, there is only one path that fabrics to establish routes across the to make use of only a single hop, the is always chosen through the fabric, fabric and to recalculate these routes if a one-hop restriction is basically a FICON and CUP Diagnostics does not have to topology change occurs (for example, link qualification issue. There is nothing in the provide any path forking reports unless failures). Each switch is a hop, and each Fibre Channel standard, or FICON itself, FIDR is in use. This enhancement was ISL between switches has a cost. The cost that precludes the use of multiple hops. made in Brocade FOS v7.3 to prepare for of an ISL is dependent on the line rate of the advent of FIDR at Brocade FOS v7.4.
Brocade 7840 Brocade 7840
SID = Source ID DID = Destination ID
Figure 7: Diamond Configurations.
10 Brocade FOS 7.4 conservative_policy. MAPS also supports CUP uses any triggered MAPS events to Enhancements to the CUP the FMS action in custom rules. See Table create Health Check Alerts. These Health 1 for an example. Some of the default rules Check Alerts are sent to z Systems hosts Diagnostics Capability are configured with the action FMS along by using the FICON Unsolicited Alert Brocade FOS v7.4 became available in with typical default actions of RASLOG, Status mechanism, to notify the hosts June 2015, and this version included SNMP, and EMAIL. The FMS action can of MAPS-detected conditions. The IBM a very significant change to CUP also be configured at the MAPS global Health Checker for z/OS can then issue Diagnostics. CUP Diagnostics was action level. (See Table 1.) subsequent FICON CCW commands enhanced to include an interface to back to the CUP to retrieve detailed and In the active policy, if the FMS notification Brocade FOS Fabric Vision™ Monitoring contextual information about the alert. action is configured for any triggered and Alerting Policy Suite (MAPS) feature events, then MAPS sends the notification Here is an example of a user creating so that MAPS can notify FMS (that is, to CUP with event information. The a custom rule that will send event CUP) when an appropriately configured following information is sent to CUP: information to CUP: MAPS event is triggered. To support this CUP notification feature, MAPS enables a ••Triggered event rule name ••mapsrule --create myRule -group ALL_F_PORTS -monitor CRC setting that is called FMS mode. Object and its type on which the event •• -timebase MIN -op ge -value 0 -action The rules that have an FMS notification was triggered raslog,snmp,FMS action are part of all three of the default ••Severity of the event MAPS policies: dflt_aggressive_policy, The Brocade FOS Fabric Vision Condition on which the event was dflt_moderate_policy and dflt_ •• MAPS module exposes a new action triggered
Table 1: An example of the FMS-supported default rules for dflt_aggressive_policy.
Policy Rule Name Condition Action
Aggressive defALL_HOST_PORTSC3TXTO_2 ALL_HOST_PORTS(C3TXTO/MIN>2) RASLOG, SNMP, EMAIL, FMS defALL_HOST_PORTSC3TXTO_4 ALL_HOST_PORTS(C3TXTO/MIN>4) FENCE, DECOM, SNMP, EMAIL, FMS defALL_OTHER_F_PORTSC3TXTO_2 ALL_OTHER_F_PORTS(C3TXTO/MIN>2) RASLOG, SNMP, EMAIL, FMS defALL_OTHER_F_PORTSC3TXTO_4 ALL_OTHER_F_PORTS(C3TXTO/MIN>4) FENCE, DECOM, SNMP, EMAIL, FMS defALL_BE_PORTSCRC_5M_10 ALL_BE_PORTS(CRC/5MIN>10) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSCRC_D_100 ALL_BE_PORTS(CRC/DAY>100 RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSITW_5M_10 ALL_BE_PORTS(ITW/5MIN>10) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSITW_D_100 ALL_BE_PORTS(ITW/DAY>100) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSLR_5M_10 ALL_BE_PORTS(LR/5MIN>10) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSLR_D_100 ALL_BE_PORTS(LR/DAY>100) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSBAD_OS_5M_10 ALL_BE_PORTS(BAD_OS/5MIN>10) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSBAD_OS_D_100 ALL_BE_PORTS(BAD_OS/DAY>100) RASLOG, SNMP, EMAIL, FMS efALL_BE_PORTSFRM_LONG_5M_10 ALL_BE_PORTS(FRM_LONG/5MIN>10) RASLOG, SNMP, EMAIL, FMS defALL_BE_PORTSFRM_LONG_D_100 ALL_BE_PORTS(FRM_LONG/DAY>100) RASLOG, SNMP,EMAIL, FMS defALL_BE_PORTSFRM_TRUNC_5M_10 ALL_BE_PORTS(FRM_TRUNC/5MIN>10) RASLOG, SNMP,E MAIL, FMS
defALL_BE_PORTS_FRM_TRUNC_D_100 ALL_BE_PORTS(FRM_TRUNC/DAY>100) RASLOG, SNMP, EMAIL, FMS
11 in the mapsconfig command. Modified column of the MVS display. Each node Traffic congestion can occur on either definition of this command is given here: can have two different types of events F_Ports or E_Ports. Bottlenecks are represented by HSC codes: detected by special Brocade FOS mapsConfig --actions
12 MAPS Trigger Values and HSC Codes and Text That Are Sent to the Z/OS Console MAPS MS Values
------MAPS MS Values ------
Monitored Element HSDB MSG MAPS - Type HSC Code ------CRC (BRP100-CRC ) ( Port) 0x11000100 ITW (BRP101-ITW ) ( Port) 0x11000101 LOSS_SYNC (BRP102-LossSync ) ( Port) 0x11000102 LF (BRP103-LostFrame ) ( Port) 0x11000103 LOSS_SIGNAL (BRP104-Signal Loss ) ( Port) 0x11000104 PE (BRP105-Proto Errors ) ( Port) 0x11000105 STATE_CHG (BRP106-State Chg ) ( Port) 0x11000106 LR (BRP107-LinkReset ) ( Port) 0x11000107 C3TXTO (BRP108-C3 Tx TO ) ( Port) 0x11000108 RX (BRP109-Rx Perf ) ( Latency) 0x11000109 TX (BRP10A-Tx Perf ) ( Latency) 0x1100010A UTIL (BRP10B-Util ) ( Latency) 0x1100010B CRED_ZERO (BRP10C-CredDelayAtZero ) ( Traffic) 0x1100010C BN_SECS (BRP10D-Bottleneck ) ( Port) 0x1100010D CRC (BRD200-BE CRC ) ( BE Port) 0x21000200 ITW (BRD201-BE ITW ) ( BE Port) 0x21000201 LR (BRD202-BE LinkReject ) ( BE Port) 0x21000202 BAD_OS (BRD203-BE Bad OrdSet ) ( BE Port) 0x21000203 FRM_LONG (BRD204-BE Frame Long ) ( BE Port) 0x21000204 FRM_TRUNC (BRD205-BE Frame Truncat ) ( BE Port) 0x21000205 GE_CRC (BRP170-GE Port CRC ) ( unknown) 0x11000170 GE_INV_LEN (BRP171-GE Port InvLen ) ( unknown) 0x11000171 GE_LOS_OF_SIG (BRP172-GE Port LOS ) ( unknown) 0x11000172 CRED_ZERO (BRD209-BE VTAP VC CRDZ ) ( Traffic) 0x21000209 DEV_LATENCY_IMPACT (BRP10E-DevImpact ) ( Latency) Sticky State State Desc( IO_LATENCY_CLEAR) - See Sticky State List State Desc( IO_PERF_IMPACT) - See Sticky State List State Desc( IO_FRAME_LOSS) - See Sticky State List
THPUT_DEG_PER (BRP10F-ThruputDegr ) ( Traffic) 0x1100010F FTX (BRP110-TxFrames ) ( Unknown) 0x11000110 DEV_NPIV_LOGINS (BRP111-NPIV Logins ) ( Port) 0x11000111 TX_Q_LATENCY (BRP112-TX Q Latency ) ( Unknown) 0x11000112 CIR_STATE (BRP131-CircuitChange ) ( FCIP Health) 0x11000131 CIR_UTIL (BRP132-CircuitUtil ) ( FCIP Health) 0x11000132 CIR_PKTLOSS (BRP133-CircuitPktLoss ) ( FCIP Health) 0x11000134 FLASH_USAGE (BRD100-HW_Flash ) (Switch Resource) 0x21000100 - (TreatAsThreshold) CPU (BRD101-HW_CPU Util ) (Switch Resource) 0x21000101 - (TreatAsThreshold) MEMORY_USAGE (BRD102-HW_MemUsage ) (Switch Resource) 0x21000102 - (TreatAsThreshold) TEMP (BRD103-HW_Temperature ) (Switch Resource) Sticky State - (TreatAsThreshold) State Desc( IN_RANGE) - See Sticky State List State Desc( OUT_OF_RANGE) - See Sticky State List
PS_STATE (BRD104-HW_PS_StChg ) ( FRU) Sticky State - (TreatAsThreshold) State Desc( OUT) - See Sticky State List State Desc( ON) - See Sticky State List State Desc( FAULTY) - See Sticky State List
FAN_STATE (BRD105-HW_Fan ) ( FRU) Sticky State - (TreatAsThreshold) State Desc( OUT) - See Sticky State List State Desc( ON) - See Sticky State List State Desc( FAULTY) - See Sticky State List
PORT (BRD106-HW_Port ) ( Unknown) Sticky State - (TreatAsThreshold) State Desc( DOWN) - See Sticky State List State Desc( UP) - See Sticky State List
SFP_STATE (BRD107-HW_GBIC ) ( FRU) Sticky State - (TreatAsThreshold) State Desc( OUT) - See Sticky State List State Desc( IN) - See Sticky State List State Desc( FAULTY) - See Sticky State List
13 MAPS MS Values (continued)
BLADE_STATE (BRD108-HW_Blade ) ( FRU) Sticky State - (TreatAsThreshold) State Desc( OUT) - See Sticky State List State Desc( ON) - See Sticky State List State Desc( OFF) - See Sticky State List State Desc( FAULTY) - See Sticky State List
WWN (BRD109-HW_WWN ) ( FRU) Sticky State - (TreatAsThreshold) State Desc( OUT) - See Sticky State List State Desc( ON) - See Sticky State List State Desc( FAULTY) - See Sticky State List
SEC_TELNET (BRD120-SecTelnet ) ( Sec) 0x21000120 SEC_HTTP (BRD121-SecHTTPViolatn ) ( Sec) 0x21000121 SEC_SCC (BRD122-SecSWLoginViolat ) ( Sec) 0x21000122 SEC_DCC (BRD123-SecDeviceViolatn ) ( Sec) 0x21000123 SEC_LV (BRD124-SecLoginViolatn ) ( Sec) 0x21000124 SEC_CERT (BRD125-SecInvalidCerts ) ( Sec) 0x21000125 SEC_TS (BRD126-SecTimerOOSync ) ( Sec) 0x21000126 SEC_AUTH_FAIL (BRD127-SecSLAPFailures ) ( Sec) 0x21000126 SEC_FCS (BRD128-SecNoFCSViolatn ) ( Sec) 0x21000128 SEC_IDB (BRD129-SecIncompatDB ) ( Sec) 0x21000129 SEC_CMD (BRD12A-SecIllegalCmdVio ) ( Sec) 0x2100012A DAYS_TO_EXPIRE No HSC Tbl Entry ( Sec) ------EXPIRED_CERTS No HSC Tbl Entry ( Sec) ------RULESTATS (- ) ( Unknown) 0x41000102 SFP_TEMP (BRP140-SFP Temperature ) ( Port) 0x11000140 VOLTAGE (BRP141-SFP Voltage ) ( Port) 0x11000141 CURRENT (BRP142-SFP Current ) ( Port) 0x11000142 RXP (BRP143-SFP Rcv Power ) ( Port) 0x11000143 TXP (BRP144-SFP TX Power ) ( Port) 0x11000144 PWR_HRS (BRP145-SFP PowerOn Hrs ) ( Port) 0x11000145 BAD_TEMP (BRD161-HW TmpSensor ) ( Unknown) 0x21000161 - (TreatAsThreshold) BAD_PWR (BRD162-HW PwrMarginal ) ( Unknown) 0x21000162 - (TreatAsThreshold) BAD_FAN (BRD163-HW Fan ) ( Unknown) 0x21000163 - (TreatAsThreshold) FAULTY_PORTS (BRD164-HW Port ) ( Unknown) 0x21000164 - (TreatAsThreshold) MISSING_SFP (BRD165-HW MissGBIC ) ( Unknown) 0x21000165 - (TreatAsThreshold) FAULTY_BLADE (BRD166-HW Blade ) ( Unknown) 0x21000166 - (TreatAsThreshold) MARG_PORTS (BRD167-HW Marg Port ) ( Unknown) 0x21000167 - (TreatAsThreshold) ERR_PORTS (BRD168-HW ErrorPort ) ( Unknown) 0x21000168 - (TreatAsThreshold) EPORT_DOWN (BRF110-E_Port Down ) ( Fabric) 0x41000100 FAB_CFG (BRF111-Config Change ) ( Fabric) 0x41000101 DID_CHG (BRF112-DID Change ) ( Fabric) 0x41000102 FAB_SEG (BRF113-Segmentation ) ( Fabric) 0x41000103 ZONE_CHG (BRF114-Zone Change ) ( Fabric) 0x41000104 FLOGI (BRF115-Flogi ) ( Fabric) 0x41000105 L2_DEVCNT_PER (BRF116-L2 Dev Cnt Pctg ) ( Fabric) 0x41000106 LSAN_DEVCNT_PER (BRF117-LSAN Dev Cnt Pct ) ( Fabric) 0x41000107 ZONE_CFGSZ_PER (BRF118-Zone Cfg Siz Pct ) ( Fabric) 0x41000108 BB_FCR_CNT (BRF119-BB FCR Cnt ) ( Fabric) 0x41000109 DOWN_CORE (BRD170-HW Core Blade ) ( Unknown) 0x21000170 - (TreatAsThreshold) HA_SYNC (BRD171-HW HA Sync ) ( Unknown) 0x21000171 - (TreatAsThreshold) WWN_DOWN (BRD172-HW WWN Down ) ( Unknown) 0x21000172 - (TreatAsThreshold) SFP_HEALTH (BRD173-HW SPF Health ) ( Unknown) 0x21000170 - (TreatAsThreshold) TX_FCNT (BRD181-Flow Tx Cnt ) ( Traffic) 0x21000181 RX_FCNT (BRD182-Flow Rx Cnt ) ( Traffic) 0x21000182 TX_THPUT (BRD183-Flow Tx Thpt ) ( Traffic) 0x21000183 RX_THPUT (BRD184-Flow Rd Thpt ) ( Traffic) 0x21000184 IO_RD (BRD185-Flow Rd Cnt ) ( Traffic) 0x21000185 IO_WR (BRD186-Flow Wr Cnt ) ( Traffic) 0x21000186 IO_RD_BYTES (BRD187-Flow Rd Bytes ) ( Traffic) 0x21000187 IO_WR_BYTES (BRD188-Flow Wr Bytes ) ( Traffic) 0x21000188 ETH_MGMT_PORT_STATE (BRD189-Flow Eth MgtPort ) (Switch Resource) Sticky State State Desc( hh) 0x11000163 RTT (BRP164-VE Rtt ) ( FCIP Health) 0x11000164 JITTER (BRP165-VE Jitter ) ( FCIP Health) 0x11000165 JITTER_INTERNAL (BRP166-JitterInternal ) ( Unknown) 0x11000166 CIR_QOS_UTIL (BRP167-CQOS_Util ) ( FCIP Health) 0x11000167
14 MAPS MS Values (continued)
CIR_QOS_PKTLOSS (BRP168-CQOS_PktLoss ) ( FCIP Health) 0x11000168 BE_LATENCY_IMPACT No HSC Tbl Entry ( Latency) ------State Desc( IO_LATENCY_CLEAR) ------NOT FOUND IN TABLE ------State Desc( IO_PERF_IMPACT) ------NOT FOUND IN TABLE ------State Desc( IO_FRAME_LOSS) ------NOT FOUND IN TABLE ------
Sticky State Events HSC Code HSDB Description ------0x11000800 BRP800-IO_LATENCY_CLEAR 0x11000801 BRP801-IO_LATENCY_DETECT 0x11000803 BRP802-IO_FRAME_LOSS 0x21000800 BRD800-TEMP IN RANGE 0x21000801 BRD801-TEMP OUTOF RANGE 0x21000810 BRD810-PwrSuply Out 0x21000811 BRD811-PwerSuply On 0x21000812 BRD812-PwerSuply Faulty 0x21000820 BRD820-Fan Out 0x21000821 BRD82 1 -Fan On 0x21000822 BRD822-Fan Faulty 0x21000830 BRD830-Port Down 0x21000831 BRD831-Port Up 0x21000840 BRD841-GBIC Out 0x21000842 BRD842-GBIC On 0x21000843 BRD843-GBIC Faulty 0x21000850 BRD850-Blade Out 0x21000851 BRD851-Blade On 0x21000852 BRD852-Blade Off 0x21000853 BRD853-Blade Faulty 0x21000860 BRD860-WWN Out 0x21000861 BRD861-WWN On 0x21000862 BRD862-WWN Faulty 0x21000870 BRD870-EthMgtPort Down 0x21000871 BRD871-EthMgtPort Up
15 Summary A switch that has CUP Diagnostics About Brocade support signals to z/OS that there is a When a FICON user installs and enables Brocade networking solutions help health problem. The z/OS system the optional FMS license on a Fibre organizations achieve their critical initiates monitoring for the path, Channel switching device, the user can business initiatives as they transition requesting diagnostic data from the utilize new CUP enhancements to provide to a world where applications and switch at regular intervals. The problem a more robust fabric environment. information reside anywhere. Today, might require intervention such as Brocade is extending its proven data Switch (CUP) Diagnostics provides new additional z/OS system commands or I/O center expertise across the entire network fabric-wide diagnostic command channel configuration changes. After no errors or with open, virtual, and efficient solutions programs to enable z/OS to obtain fabric health issues are detected by the switch built for consolidation, virtualization, topology, collect diagnostic information for at least two hours, the monitoring and cloud computing. Learn more at such as performance data, determine the of the route is stopped and no longer www.brocade.com. health of the fabric, generate a diagnostic appears in the report. log on the switch, and help users resolve The IBM Health Checker for z/OS problems in the fabric. reports if any switches that support A switch device can provide an unsolicited CUP Diagnostics capabilities indicate unit check, called a Health Summary unusual health conditions on connected unit check, which indicates that one or channel paths. more switches or ports in the fabric are While z/OS has historically provided operating at less than optimal capability. sophisticated I/O monitoring and This check triggers z/OS to retrieve the recovery, these Health Checker reports diagnostic information from the switch to expose newly available diagnostic data further diagnose the problem. The sense provided directly from the switch. These data identifies the source and destination health checks enable additional insight ports that are used for the query. into problems with a user’s fabric, such as hardware errors, I/O misconfigurations, or fabric congestion.
Corporate Headquarters European Headquarters Asia Pacific Headquarters San Jose, CA USA Geneva, Switzerland Singapore T: +1-408-333-8000 T: +41-22-799-56-40 T: +65-6538-4700 [email protected] [email protected] [email protected]
© 2016 Brocade Communications Systems, Inc. All Rights Reserved. 03/16 GA-WP-2103-01
Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, Vplane, and Vyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of others.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.