Device-Alias Database

#CLUS Advanced Storage Area Network Design Edward Mazurek Technical Lead Data Center Storage Networking [email protected] @TheRealEdMaz BRKSAN-2883

#CLUS Agenda

• Introduction

• Design Principles

• Design Principles for Slow Drain and Congestion Isolation

• Design Principles for SAN Analytics • Q&A

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How 1 Find this session in the Cisco Live Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKSAN-2883 by the speaker until June 18, 2018.

• Assumptions: • Most SANs are reliable and have few problems

• Move from: • Occasional problems that sometimes cause outages

• Move to: • Infrequent problems and almost no outages

How can your SAN be more reliable, robust and less prone to errors?

• Cisco FC/FCoE SAN switches provide a host of advanced features that can make your SANs more

• Robust

• Scalable

• Fault tolerant

• High performance

• Easy to Manage

• Easy to investigate / troubleshoot

• VSANs

• Zoning and Smart Zoning

• N-Port Virtualization

• Trunking and Port-channeling

• MDS Internal CRC handling

• Device-alias

• SAN Security

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 VSANs Introduced in 2002 • A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric and Create Virtual Fabrics

• Analogous to VRFs in Ethernet Per Port Allocation • Virtual fabrics created from larger cost-effective redundant physical fabric

• Reduces wasted ports of a SAN island approach

• Fabric events are isolated per VSAN which gives further isolation for High Availability

• FC Features can be configured on a per VSAN basis.

• ANSI T.11 committee and is now part of Fibre Channel standards as Virtual Fabrics

• Logically separate fabrics

• Hardware enforced

• Prevents fabric disruptions • RSCN sent within fabric only

• Each fabric service (zone server, name server, login server, etc.) operates independently in each VSAN

• Each VSAN is configured and managed independently

phx2-9513# show fspf vsan 43 vsan database FSPF routing for VSAN 43 vsan 2 interface fc1/1 FSPF routing administration status is enabled vsan 2 interface fc1/2 FSPF routing operational status is UP vsan 4 interface fc1/8 It is an intra-domain router vsan 4 interface fc1/9 Autonomous region is 0 MinLsArrival = 1000 msec , MinLsInterval = 2000 phx2-9513# show zoneset active vsan 43 msec zoneset name UCS-Fabric-B vsan 43 Local Domain is 0xe6(230) zone name UCS-B-VMware-Netapp vsan 43 Number of LSRs = 3, Total Checksum = 0x00012848 #CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Zoning & VSANs

VSAN 2 1. Assign physical ports to VSANs

Disk2 Disk3 2. Configure zones within each VSAN Zone A Host1 Disk1 Zone C • A zone consists of multiple zone members Zone B Disk4 Host2 3. Assign zones to zoneset Zoneset 1 • Each VSAN has its own zoneset

VSAN 3 4. Activate zoneset in VSAN Zone A Host4 • Members in a zone can access each other; Zone B Host3 Disk5 members in different zones cannot access Disk6 each other Zoneset 1 • Devices can belong to more than one zone

• Non-zoned devices are members of the default zone zone name AS01_NetApp vsan 42 member pwwn 20:03:00:25:b5:0a:00:06 • A physical fabric can have a member pwwn 50:0a:09:84:9d:53:43:54 maximum of 16,000 zones (9700-only network) device-alias name AS01 • Attributes can include pWWN, FC pwwn 20:03:00:25:b5:0a:00:06 alias, FCID, FWWN, Switch Interface device-alias name NTAP fc x/y, Symbolic node name, Device member pwwn 50:0a:09:84:9d:53:43:54 zone name AS01_NetApp vsan 42 alias member device-alias AS01 member device-alias NTAP • Recommended: Device-alias and/or PWWN

. Standard zoning model just Number of ACLs has “members” 10,000 . Any member can talk to any 8,000 other member 6,000 . Recommendation: 1-1 zoning 4,000 . Each pair consumes two ACL entries in TCAM 2,000

. Result: n*(n-1) entries per 0

NumberofACLEntries

40 60 80 20 30 50 70 90

zone 10 100 Number of Members

Operation Today – 1:1Operation Zoning Today – ManyOperation - Many Smart Zoning 8 x I Zones Cmds ACLs Zones Cmds ACLs Zones Cmds ACLs 4 x T Create 32 96 Create64 1 13 Create132 1 13 64 zones(s) zones(s) zones(s) Add an +4 +12Add +8an +1 Add+24 an +1 +8 initiator initiator initiator Add a +8 +24 Add+16 a +1 Add+24 a +1 +16 target target target

• Feature added in NX-OS 5.2(6)

• Allows storage admins to create larger zones while still keeping premise of single initiator & single target

• Dramatic reduction SAN administrative time for zoning

• Utility to convert existing zone or zoneset to Smart Zoning

• zone mode enhanced • Acquires lock on all switches while zoning changes are underway • Enables full zoneset distribution

• zone smart-zoning enable • Allows for more efficient, easier zoning

• zone confirm-commit • Causes zoning changes to be displayed during zone commit

• zoneset overwrite-control – New in NX-OS 6.2(13) • Prevents a different zoneset than the currently activated zoneset from being inadvertently activated

• Do not use fWWN, sWWN or interface zoning for NPV switches • All devices on that NPV link will be zoned together!

• Check ACLTCAM usage after zoning changes • show system internal acl tcam-usage • show system internal acltcam-soc tcam-usage

9710-2# show interface fc1/8 • Allows for the correction of some frame fc1/8 is trunking errors … Port mode is TE • Almost zero latency penalty Port vsan is 1 Speed is 16 Gbps • Can prevent SCSI timeouts/aborts Rate mode is dedicated Transmit B2B Credit is 500 • Applies to 9700 FC, 9396S, 9132T Receive B2B Credit is 500 • 16G – Applies fixed speed FC ISLs only B2B State Change Number is 14 Receive data field Size is 2112 • 32G – On by default Beacon is turned off admin fec state is up • Configured via: oper fec state is up Trunk vsans (admin allowed and active) (1-2,20,237) • switchport fec tts

• No reason not to use it!

MDS9710-1# show interface fc9/1 counters details fc9/1 These are corrected and not 861181 frames, 75211536 bytes received dropped

… 0 fec corrected blocks These are dropped as CRC 0 fec uncorrected blocks errors

Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%

FEC corrected blocks can be used as a warning of some frame corruption

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Trunking & Port Channels Trunking Single-link ISL or PortChannel ISL can be configured to become EISL – (TE_Port) Trunk

VSAN1 VSAN1 Traffic engineering with pruning VSANs on/off the trunk

VSAN2 VSAN2 Efficient use of ISL bandwidth VSAN3 VSAN3

TE Port TE Port Up to 16 links can be combined into a PortChannel increasing the aggregate bandwidth by distributing traffic Port Channel granularly among all functional links in the channel Port TE Port Channel TE Port Load balances across multiple links and maintains optimum bandwidth utilization. Load balancing is based on the source ID, destination ID, and exchange ID

If one link fails, traffic previously carried on this link is switched to the remaining links. To the upper protocol, E Port E Port the link is still there, although the bandwidth is diminished. The routing tables are not affected by link failure

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 N-Port Virtualization Scaling Fabrics with Stability • N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like a server/HBA performing multiple fabric logins through a single physical link

• Physical servers connect to the NPV switch and login to the upstream NPIV core switch

• No local switching is done on an FC switch in NPV mode

• FC edge switch in NPV mode does not take up a domain ID MDS9513 (config)# feature npiv • Helps to alleviate domain ID exhaustion in large fabrics Blade Server NPV Switch FC NPIV Core Switch F-Port Server1 F-Port FC1/1 N_Port_ID 1 Server2 FC1/2 N_Port_ID 2 NP-Port Server3 FC1/3 N_Port_ID 3 F_Port

F-Port Port Channel F-Port Port Channel w/ NPV .Bundle multiple ports in to 1 logical link F-Port Port Core Channel Director Storage Any port, any module

Blade N . High-Availability (HA)

System Blade 2 Blade Servers are transparent if a cable, port, or line

Blade 1 cards fails Blade . Traffic Management NP-Ports F-Ports Higher aggregate bandwidth Hardware-based load balancing F-Port Trunking F-Port Trunking w/ NPV Core Director .Partition F-Port to carry traffic for multiple VSANs F-Port Trunking . Extend VSAN benefits to Blade VSAN1 Blade N Servers Blade 2 VSAN2 Separate management domains Blade 1 VSAN3 Blade System Separate fault isolation domains NP-Port F-Port Differentiated services: QoS, Security

phx2-5548-3# show flogi database 5548 ------INTERFACE VSAN FCID PORT NAME NODE NAME ------fc2/9 12 0x020000 20:41:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 fc2/9 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 D2 fc2/9 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 fc2/10 12 0x020020 20:42:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 fc2/1 fc2/10 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 0 fc2/10 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00

Total number of flogi = 7 fc2/1 fc2/2 phx2-5548-3#

Fabric Interconnect

phx2-5548-3# show flogi database 5548 ------INTERFACE VSAN FCID PORT NAME NODE NAME ------San-po3 12 0x020040 24:0c:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 San-po3 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02 San-po3 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 D2 San-po3 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01 San-po3 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 2/9 2/1 San-po3 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00 san-po-0 3 Total number of flogi = 6 All devices associated with 2/2 phx2-5548-3# port-channel now 2/1

Fabric Interconnect

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Port-channel design considerations All types of switches • Name port-channels the same on both sides (for clarity)

• Common port allocation in both fabrics

• ISL speeds should be >= edge device speeds

• Maximum 16 members per port-channel allowed

• Multiple port-channels to same adjacent switch should be equal cost

• Member of VSAN 1 + trunk other VSANs

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Port-channel design considerations All types of switches • Use channel mode active • This ensures misconfigurations and cabling are checked

• Check TCAM usage on NPIV core switch • show system internal acl tcam-usage • show system internal acltcam-soc tcam-usage

• When possible use same port on each LC (for clarity): • Ex. fc1/5, fc2/5, fc3/5, fc4/5, etc.

• If multiple members per linecard distribute across Fwd Engines and port-groups • show port-resources module x

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Port-channel design considerations Fabric switches – 9250i, 9148S • Ensure enough credits for distance • Can “rob” buffers from other ports in port-group that are “out-of-service”

• Split PC members across different FWD engines to distribute ACLTCAM • For F port-channels to NPV switches (like UCS FIs) • Each device’s zoning ACLTCAM programming will be repeated on each member • For E port-channels(or just ISLs) using IVR • Each host/target session that gets translated will take up ACLTCAM on each member • Ex. On a 9148S a 6 member port-channel could be split across the 3 fwd engines as follows: fc1/1, fc1/2, fc1/17, fc1/18, fc1/33 and fc1/34

• Split large F port-channels into two separate port-channels each with half members

• Consider MDS 9396S and MDS 9132T for larger scale deployments

Fabric switches – 9250i, 9148S Zoning region is • Check TCAM usage after major zoning operations the most likely to 9148S has 3 FWD be exceeded engines MDS9148s-1# show system internal acltcam-soc tcam-usage

TCAM Entries: 98 entries in use ======due to zoning Region1 Region2 Region3 Region4 Region5 Region6 Mod Fwd Dir TOP SYS SECURITY ZONING BOTTOM FCC DIS FCC ENA Eng Use/Total Use/Total Use/Total Use/Total Use/Total Use/Total ------1 1 INPUT 19/407 1/407 98/2852 * 4/407 0/0 0/0 1 1 OUTPUT 0/25 0/25 0/140 0/25 0/12 1/25 1 2 INPUT 19/407 1/407 0/2852 * 4/407 0/0 0/0 1 2 OUTPUT 0/25 0/25 0/140 0/25 0/12 1/25 1 3 INPUT 19/407 1/407 0/2852 * 4/407 0/0 0/0 1 3 OUTPUT 0/25 0/25 0/140 0/25 0/12 1/25 ------

Fwd Fwd-Eng Zoning Region Switch Type Port Range(s) Engines Number Entries fc1/25-36 & MDS 9148 3 1 2852 fc1/45-48 fc1/5-12 & 2 2852 fc1/37-44 1-4 & 13-24 3 2852 fc1/5-12 & MDS 9250i 4 1 2852 eth1/1-8 fc1/1-4 & fc1/13-20 & 2 2852 fc1/37-40 fc1/21-36 3 2852 ips1/1-2 4 2852

Ports are allocated to fwd-engines according the following table:

Fwd Fwd-Eng Zoning Region Switch Type Port Range(s) Engines Number Entries MDS 9148S 3 fc1/1-16 1 2852 fc1/17-32 2 2852 fc1/33-48 3 2852 MDS 9132T 2 fc1/1-16 0 49136 fc1/17-32 1 49136

For others see the Zoning Configuration Guide

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 F port-channel design considerations Fwd Fwd-Eng Zoning Region Switch Type Port Range(s) Engines Number Entries MDS 9396S 12 1-8 0 49136 9-16 1 49136 17-24 2 49136 25-32 3 49136 33-40 4 49136 41-48 5 49136 49-56 6 49136 57-64 7 49136 65-72 8 49136 73-80 9 49136 81-88 10 49136 89-96 11 49136

• New feature in 6.2(13) to handle frames internally corrupted due to bad HW • Frames that are received corrupted and cannot be FEC corrected are dropped at the ingress port • These frames are not included in this feature • In rare cases frames can get corrupted internally due to bad hardware • These are then dropped • Sometimes difficult to detect • New feature detects the condition and isolates hardware • 5 possible stages where frames can get corrupted

Stages of Internal CRC Detection and Isolation

The five possible stages at which internal CRC errors may occur in a switch: 1. Ingress buffer of a module 2. Ingress crossbar of a module 3. Crossbar of a fabric module 4. Egress crossbar of a module 5. Egress buffer of a module

• The modules that support this functionality are: • Cisco MDS 9700 48-Port 16-Gbps Fibre Channel Switching Module • Cisco MDS 9700 48-Port 32-Gbps Fibre Channel Switching Module • Cisco MDS 9700 24-Port 16Gbps / 10-port SAN Extension Module • Cisco MDS 9700 48-Port 10-Gbps Fibre Channel over Ethernet Switching Module • Cisco MDS 9700 24-Port 40-Gbps Fibre Channel over Ethernet Switching Module • Cisco MDS 9700 Fabric Module 1 • Cisco MDS 9700 Supervisors

• Enabled via the following configuration command: • hardware fabric crc threshold 1-100

• When detected failing module is powered down

• Threshold is per 24 hour period

• New in NX-OS 6.2(13)

• Port-monitor can also alert on these types of errors

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 CFS Distribution over IP • Cisco Fabric Services(CFS) is used to distribute various features • callhome Callhome server • device-alias DDAS Daemon • fctimer Fibre Channel timer • ivr Inter-VSAN Routing • ntp Network Time Server • port-security Port Security Manager • radius Radius Daemon • role Role • syslogd System Logger Daemon • tacacs Tacacs Daemon • And several others…

• Topologies with NPV switches do not get this distribution

• To distribute to NPV switches distribute via IP multicast

• Multicast addresses should be unique per fabric show cfs status Distribution : Enabled Distribution over IP : Enabled - mode IPv4 IPv4 multicast address : 239.255.70.83

• Problems can arise when multicast address are non-unique per fabric

• Can lead to high CPU and CFS distribution problems if enabled

• Ensure “show cfs peers…” shows appropriate per-fabric peers only!

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 Device-alias • device-alias(DA) is a way of naming PWWNs • DAs are distributed on a fabric basis via CFS • device-alias database is independent of VSANs • If a device is moved from one VSAN to another no DA changes are needed • device-alias can run in two modes: • Basic – device-alias names used but PWWNs are substituted in config • Enhanced – device-alias names exist in configuration natively – Allows rename without zoneset re-activations • device-alias are used in zoning, IVR zoning and port-security • copy running-config startup-config fabric after making changes!

• device-alias confirm-commit • Displays the changes and prompts for confirmation Enabling this gives this when commit is MDS9710-2(config)# device-alias confirm-commit enable done MDS9710-2(config)# device-alias database MDS9710-2(config-device-alias-db)# device-alias name edm pwwn 1000000011111111 MDS9710-2(config-device-alias-db)# device-alias commit The following device-alias changes are about to be committed + device-alias name edm pwwn 10:00:00:00:11:11:11:11 Do you want to continue? (y/n) [n]

• To prevent problems the same device-alias is only allowed once per commit. • Example: MDS9148S(config)# device-alias database MDS9148S(config-device-alias-db)# device-alias name test pwwn 1122334455667788 MDS9148S(config-device-alias-db)# device-alias rename test test1 Command rejected. Device-alias reused in current session :test Please use 'show device-alias session rejected' to display the rejected set of commands and for the device-alias best-practices recommendation.

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 SAN Security Secure management access • Role-based access control – CLI, SNMP, Web FC CT Management security Device/SAN New in NX-OS 6.2(9) Secure management protocols Management Security Via SSH, Port-security • SSH, SFTP, and SNMPv3 SFTP, SNMPv3, and • Disable feature telnet User Roles RADIUS or TACACS+ or LDAP Server for Secure switch control protocols Authentication • TrustSec • FC-SP (DH-CHAP) SAN AAA - RADIUS, TACACS+ and LDAP Protocol • User, switch and iSCSI host authentication Security (FC-SP) Fabric Binding iSCSI- VSANs • Prevent unauthorized switches from joining fabric Attached Provide Servers Secure Port-security Hardware-Based Isolation Zoning Via Port and • Ensure only approved devices login to fabric WWN FC CT Management Security • Ensure only approved devices send FC CT cmds Shared Physical Storage

• FC servers can send management requests into switches via FC

• By default the MDS does not prohibit this

• Two possibilities 1. Zone Generic Services zone service via inband management Defaults to allowing queries and zoning changes 2. Fibre Channel Common Transport (FC-CT) Management Security Prevents unauthorized FC-CT queries into a network Defaults to allow inband access to management information

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46 FC Management Security Zone Generic Services • Defaults to read-write to allow local devices to make zoning changes • To disable configure: • no zone gs read-write vsan x • Or Switch default • no system default zone gs read-write for all VSANs

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 FC Management Security Fibre Channel Common Transport (FC-CT) Management Security • Attached FC devices can send in Common Transport commands to query the Name Server • Ensure only approved devices send FC CT cmds • Introduced in NX-OS 6.2(9) • To enable • fc-management enable • If devices are to be permitted to query the name server then they must be added into the database: • fc-management database vsan • pwwn feature operation

• FLOGI scale • FLOGI quiesce

Feature Parameters Old Limits New Limits (MDS 9700) (MDS 9718) Login (FLOGI / FLOGIs per Module 1000 2000 FDISC) FLOGIs per Switch 4000 8000

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 FLOGI Scale Enhancements Flogi scale • On a switch or module reload, routing information(RIB) is updated based on prior fcdomain manager information. This is done even prior to devices logging in. This will reduce the time taken to generate the ELS_ACC(FLOGI).

• CLI: flogi scale enable – default is “enabled” on all switch types except for 9148S and 9250i

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 FLOGI Scale Enhancements Flogi quiesce • When a device logs out of a port the Fport-server will send the ELS_ACC(LOGO) normally but will delay notifying other applications and sending any RSCNs for a default 2 seconds.

• If the device re-logs in to the same interface then there is very little the Fport-server needs to do.

• If the device does not re-log in to the same interface within 2 seconds or logs into a different interface on the same switch then FLOGI notifies all of the other applications and sends the appropriate “offline” RSCNs.

• If there are devices sharing PWWNs by logging into separate switches in failover conditions then this should be disabled

• CLI: flogi quiesce timeout 0-20000 0 disables it. • NX-OS 8.1 and 8.2 default is 2000(2 seconds) • NX-OS 8.3(1) and later the default was changed to 0 (no delay)

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 Design Principles for Slow Drain and Congestion Isolation SAN Congestion What is SAN congestion? • SAN congestion is when some part of the SAN has frames that cannot be immediately transmitted

• Caused by two main reasons

1. “Traditional” slow drain • Devices purposely withholding buffer to buffer credits • Lost credits • Easy to spot

2. Overutilization / Oversubscription • Devices requesting more data than they can receive at their link rate • More difficult and tricky to spot

SCSI Read SCSI Data 1. Server SCSI Read sends SCSI Read 16Gb FC multiple SCSI Read port- 8Gb reads to channel Data different FC FC 8Gb RRDY targets switch1 switch2 No R_Rdy Data Data Sent FC Data Data Data Data RRDY RRDY 8Gb server No/slow No R_Rdy Data RRdy sent Sent RRDY No R_Rdy Sent 5. Server SCSI Data 6. FC 3. Data 7. Data stops or 4. Data 8. Left FC switch must arriving at builds up 9. Data slows sent by switch 10. Right frequently a on ingress builds up R_RDYs the FC stops FC Switch stop maximum due on ingress switch sending stops sending due combined excessive due R_Rdys to sending to 0 Tx 16 Gbps data being excessive right FC R_Rdys to credits received data being switch received arrays Both arrays and all devices utilizing ISLs are affected!

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 2. Data sent at line Over Utilization - Example rate 8Gbps by both Multiple SCSI reads cause congestion w/o withholding R_RDYs arrays 1. Server sends SCSI Read SCSI Data SCSI Read multiple SCSI Read reads to 16Gb FC SCSI Read different port- 8Gb targets FC channel FC Data closely 8Gb RRDY switch1 switch2 No R_Rdy spaced Data Data Sent FC Data Data Data Data Data Data RRDY RRDY RRDY RRDY RRDY 8Gb server No R_Rdy Data Sent RRDY No R_Rdy Sent 5. Server 4. Data SCSI Data 6. FC switch 7. Data sending sent at line 3. Data 8. Left FC 9. Data receiving builds up R_RDYs w/o rate arriving at switch builds up 10. Right excess data on ingress delay 8Gbps by a stops on ingress FC Switch due to due the FC combined sending due stops sheer excessive switch 16 Gbps R_Rdys to excessive sending volume data being right FC data being R_Rdys to received switch received arrays Not strictly “slow drain” but the effects are exactly the same! #CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 SAN Congestion – Alerting / Prevention Alerting - Port-monitor • Port-monitor is the primary mechanism for alerting

• Port-monitor functions by running at periodic “poll-intervals”

• Each “poll-interval” it checks a configured counter

• If a counter >= a configured “rising-threshold” an SNMP alert/syslog is generated

• As long as counter continues above the “falling-threshold” no further SNMP alerts or syslogs are generated

• If a counter <= a configured “falling-threshold” SNMP alert/syslog is generated

• counter poll-interval delta rising-threshold event falling-threshold event warning- threshold • poll-interval – Seconds - How often should this counter be checked? • delta – Compare the current value with the value at the previous poll interval • absolute – Match the actual value • rising-threshold – How much the counter must increase in this poll interval to trigger • event – Indicates severity of alert - info, warning, error, etc. • warning-threshold – Optional - A lower value than rising-threshold to issue warning syslog message (no alert)

• falling-threshold - How much the counter must decrease in this poll interval to reset • portguard – Optional – Action to take when rising-threshold is reached • errordisable – Place put in error-disable state. Requires manual shut/no shut to re- activate • flap – shut/no shut port • cong-isolate – For congestion-isolation – Only valid on these four counters: • TXWait • TX-credit-not-available • credit-loss-reco • tx-slowport-oper-delay

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59 Slow Drain Alerting and Prevention Alerting - Port-monitor – RMON event severities • Event indicates severity in alert • 1 – Fatal • 2 – Critical mds9513(config-port-monitor)# show rmon events Event 1 is active, owned by PMON@FATAL • 3 – Error Description is FATAL(1) Event firing causes log and trap to community public, last fired never • 4 – Warning Event 2 is active, owned by PMON@CRITICAL Description is CRITICAL(2) • 5 – Informational Event firing causes log and trap to community public, last fired never Event 3 is active, owned by PMON@ERROR Description is ERROR(3) Suggestions: Event firing causes log and trap to community public, last fired never Event 4 is active, owned by PMON@WARNING 2 – Link failure events Description is WARNING(4) Event firing causes log and trap to community public, last fired 3 – Packet loss events 2014/02/21-17:13:11 Event 5 is active, owned by PMON@INFO 4 – Delay type events Description is INFORMATION(5) Event firing causes log and trap to community public, last fired 2014/03/08-08:25:19

Counter name Slow Drain Event Description Level credit-loss-reco 3 2 Credit loss recovery counter – 1-1.5 secs @ 0 Tx credits lr-rx 3 2 The number of link resets received by the fc-port lr-tx 3 2 Link resets transmitted by the fc-port timeout-discards 2 3 Timeout discards counter tx-discards 2 3 Tx discards counter – all reasons tx-credit-not-available 1 4 Credit not available counter 10% (100ms) increments tx-slowport-oper-delay 1 4 Slowport operational delay (9500 gen4 & 16G) txwait 1 4 Percentage of time at 0 Tx credits and packets Q’d tx-datarate 1 4 Tx data rate as a percentage of link speed

Counter name Slow Drain Event Description Level Link-loss NA 2 Link loss counter Sync-loss NA 2 Sync loss counter Signal-loss NA 2 Signal loss counter Invalid-words NA 3 Invalid words counter Invalid-crc NA 3 Frames received with invalid CRC counter

Note link-loss, sync-loss and signal-loss are very similar. Probably only need link-loss.

Counter name Slow Drain Event Description Level err-pkt-from-port NA 3 Internal CRC Error packets from port to XBAR err-pkt-from-xbar NA 3 Internal CRC error packets from XBAR to module err-pkt-to-xbar NA 3 Internal CRC error packets from module to XBAR

These will alert for the same indications of “hardware fabric crc” command

Note these are only valid in one active policy

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 Slow Drain Alerting and Prevention Alerting – Port-monitor – tx-datarate counter • tx-datarate counter is used for detecting “over utilization”

• Configure it as follows: • counter tx-datarate poll-interval 10 delta rising-threshold 80 event 4 falling-threshold 79 event 4

• New in NX-OS 8.2(1) this is added into logging onboard

• show logging onboard datarate

• This is critical to being able to identify “over utilization”!

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 Slow Drain Alerting and Prevention Alerting – Port-monitor – tx-datarate counter MDS9710-1# show logging onboard datarate ------Module: 1 datarate ------DATARATE INFORMATION FROM FCMAC

------| Interface | Speed | Alarm-types | Rate | Timestamp | ------| fc1/13 | 4G | TX_DATARATE_FALLING | 1% | Thu Nov 2 09:16:11 2017 | | fc1/13 | 4G | TX_DATARATE_RISING | 83% | Thu Nov 2 08:56:10 2017 | | fc1/13 | 4G | TX_DATARATE_FALLING | 73% | Thu Nov 2 11:19:46 2017 | | fc1/13 | 4G | TX_DATARATE_RISING | 83% | Wed Nov 1 08:49:04 2017 |

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 Slow Drain Alerting and Prevention Event 2 – Critical Alerting - Port-monitor – AllPorts Example no portguard Event 3 – Error Event 4 - Warning port-monitor name AllPorts logical-type all Policy applies to all ports counter link-loss poll-interval 60 delta rising-threshold 5 event 2 falling-threshold 0 event 2 counter invalid-crc poll-interval 60 delta rising-threshold 5 event 3 falling-threshold 0 event 3 counter tx-discards poll-interval 60 delta rising-threshold 50 event 3 falling-threshold 10 event 3 counter lr-rx poll-interval 60 delta rising-threshold 5 event 2 falling-threshold 1 event 2 counter lr-tx poll-interval 60 delta rising-threshold 5 event 2 falling-threshold 1 event 2 counter timeout-discards poll-interval 60 delta rising-threshold 50 event 3 falling-threshold 10 event 3 counter credit-loss-reco poll-interval 60 delta rising-threshold 1 event 2 falling-threshold 0 event 2 counter tx-credit-not-available poll-interval 1 delta rising-threshold 10 event 4 falling-threshold 0 event 4 counter tx-datarate poll-interval 10 delta rising-threshold 80 event 4 falling-threshold 79 event 4 counter err-pkt-from-port poll-interval 300 delta rising-threshold 5 event 3 falling-threshold 0 event 3 counter err-pkt-to-xbar poll-interval 300 delta rising-threshold 5 event 3 falling-threshold 0 event 3 counter err-pkt-from-xbar poll-interval 300 delta rising-threshold 5 event 3 falling-threshold 0 event 3 counter tx-slowport-oper-delay poll-interval 1 absolute rising-threshold 80 event 4 falling-threshold 0 event 4 counter txwait poll-interval 1 delta rising-threshold 10 event 4 falling-threshold 0 event 4 monitor counter err-pkt-from-port monitor counter err-pkt-to-xbar Hardware internal CRC monitor counter err-pkt-from-xbar errors are monitored no monitor counter sync-loss no monitor counter signal-loss no monitor counter invalid-words no monitor counter state-change These counters are not no monitor counter rx-datarate monitored port-monitor activate AllPorts

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 Slow Drain Alerting and Prevention Alerting - Port-monitor – Sample output MDS9710-1# show port-monitor active

Policy Name : AllPorts Admin status : Active Oper status : Active Port type : All Ports ------Counter Threshold Interval Rising Threshold event Falling Threshold event Warning Threshold PMON Portguard ------Link Loss Delta 60 5 2 0 2 Not enabled Not enabled Invalid CRC's Delta 60 5 3 0 3 Not enabled Not enabled TX Discards Delta 60 50 3 10 3 Not enabled Not enabled LR RX Delta 60 5 2 1 2 Not enabled Not enabled LR TX Delta 60 5 2 1 2 Not enabled Not enabled Timeout Discards Delta 60 50 3 10 3 Not enabled Not enabled Credit Loss Reco Delta 60 1 2 0 2 Not enabled Not enabled TX Credit Not Available Delta 1 10% 4 0% 4 Not enabled Not enabled TX Datarate Delta 10 80% 4 79% 4 Not enabled Not enabled ASIC Error Pkt from Port Delta 300 5 3 0 3 Not enabled Not enabled ASIC Error Pkt to xbar Delta 300 5 3 0 3 Not enabled Not enabled ASIC Error Pkt from xbar Delta 300 5 3 0 3 Not enabled Not enabled TX-Slowport-Oper-Delay Absolute 1 80ms 4 0ms 4 Not enabled Not enabled TXWait Delta 1 10% 4 0% 4 Not enabled Not enabled ------

• <= indicates falling alert

• Rising alert generated once when rising-threshold condition is met

• Additional rising alerts are not generated until a falling alert occurs

• Falling alert generated once the falling-threshold condition is met show rmon logs Event 4 2179 WARNING(4) Rising alarm 65500, fired at 2016/11/12-03:30:35 iso.3.6.1.2.1.31.1.1.1.10.18415616=453507726 >= 400000000 2181 WARNING(4) Falling alarm 65500, fired at 2016/11/12-03:31:07 iso.3.6.1.2.1.31.1.1.1.10.18415616=20996824 <= 300000000

Note: 18415616 is the interface index(ifindex) for fc4/27

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 Slow Drain Alerting and Prevention Alerting – Port-monitor – Rising/Falling-threshold Rising threshold Alert Falling threshold Alert counter timeout-discards poll-interval 60 delta rising-threshold 10 falling-threshold 5… At +360 seconds Timeout discards delta is 0 < falling-threshold – Alert!

50 45 40 At +240 and +300 seconds Timeout discards 35 delta is 8 > falling-threshold – No action 30 25 20 At +180 seconds Timeout discards delta 15 is 15 - hits rising-threshold – Alert!

10 Timeout Timeout discards 5

60 120 180 240 300 360 420

Counter checked every 60 seconds

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 Slow Drain Alerting and Prevention Alerting – Port-monitor - Advanced • check-interval • Optional • Small polling internals sometimes “miss” counter increments if they span different intervals • Small rising thresholds can make port-monitor too sensitive • Allows for larger polling-intervals but frequent checking

• warning-threshold • Optional • A lower value than rising-threshold to issue warning syslog message (no alert)

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70 Slow Drain Alerting and Prevention Alerting – Port-monitor - Advanced • Check-interval • Small polling intervals can “miss” counter increments if they span different intervals • Small rising thresholds can make port-monitor too sensitive • Allows for larger polling-intervals but frequent checking port-monitor check-interval 5 counter timeout-discards poll-interval 60 delta rising-threshold 50 … 50 45 40 Timeout discards hits rising- 35 threshold at T+15 seconds 30 25 Alert generated 45 seconds early! 20 15

10 Timeout Timeout discards 5

5 10 15 30 45 60

Counter checked every 5 seconds Counter checked at poll-interval

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 Slow Drain Alerting and Prevention Alerting – Port-monitor - DCNM 10.4(1) event log Events are visually distinguished by event severity

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72 Slow Drain Alerting and Prevention Prevention – FC - Adjust Congestion Drop Threshold Lower • Lowering congestion drop timeout value from 500ms to 200ms system timeout congestion-drop 200 logical-type edge

• Frees up ingress buffer space quicker 0 sec -- Frame Check Timestamp • Can be set differently on F and E ports Frame Frame of each frame • Congestion timeout for mode F should be smaller than(or equal to) mode E. 200ms -- • Global command for switch Drop the Frames • Recommended for F ports from the queue

• Do not go below 200ms! Credit Frame

• No-credit-drop causes frames to be system timeout no-credit-drop 200 logical-type edge dropped immediately if the destination port is at 0 Tx credits for the time specified

• Should be used in conjunction with lowering congestion-drop threshold

• Recommended for F ports

• Can drastically improve ISL performance under slow drain conditions

• xxx_FORCE_TIMEOUT_ON/OFF counter

• By default no-credit-drop is not enabled

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 Slow Drain Alerting and Prevention Prevention - FCoE - Adjust Congestion-Drop Threshold Lower • Lowering congestion-drop timeout value system timeout fcoe congestion-drop 200 mode edge from 500ms to 200ms

• Do not go below 200ms! Resume Frame

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 Slow Drain Alerting and Prevention Frame Frame Frames in Rx queue Pause 0 sec -- Frame PausePause from other ports Prevention - FCoE - Setting the PausePause Frame -- . Frame pause-drop threshold . Pause- . -- drop • Pause-drop causes frames to be dropped if Pause 100ms 100ms-- the destination port is continuously paused for the time specified -- -- Drop frames in Rx queue • Should be used in conjunction with lowering congestion timeout threshold

• Recommended for F ports(edge) Drop any new arriving Frames immediately • Can drastically improve ISL performance under slow drain conditions Once Resume arrives Resume -- • By default pause-drop is not enabled resume sending Frame • Checked by SW – minimum 100ms Frame Frame system timeout fcoe pause-drop 100 mode edge #CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Slow Drain Alerting and Prevention Prevention – SAN congestion - VSANs and ISLs

• ISLs trunking multiple VSANs will all experience congestion if one VSAN is experiencing it

• Separate physical ISLs for each Single interface or port-channel VSAN provides better isolation Individual interfaces or port- channels per VSAN • With separate per-VSAN ISLs congestion on one VSAN will not affect the other VSANs

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77 Slow Drain Alerting and Prevention Congestion-Isolation of slow devices • NX-OS 8.1(1) added the ER_RDY and Congestion-Isolation feature

• ISLs now have a new FC flow control mechanism – ER_RDY

• This internally partitions the physical link into 4 virtual links: • VL0 – Control traffic – 15 B2B credits • VL1 – High-priority traffic – 15 B2B credits • VL2 – Slow traffic – 40 B2B credits • VL3 – Normal traffic – 430 B2B credits

• ER_RDYs are now sent/managed separately by VL

• Initially all end device traffic is sent over the “Normal” VL - VL3

• Once a slow device is detected, it is put into the slow virtual link – VL2

Note: B2B credit numbers are default for a 500 B2B credit ISL

3 steps to enable

1. ISLs must be put into ER_RDY Mode • MDS-9710(config)# system fc flow-control er_rdy • ISLs flapped to change mode • Port-channel members can be flapped one at a time to be non-disruptive

2. Feature congestion-isolation configured • MDS9710(config)# feature congestion-isolation

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 Slow Drain Alerting and Prevention Congestion-Isolation of slow devices 3. Port-monitor counters with portguard cong-isolate – 4 available 1. credit-loss-reco 2. tx-credit-not-available 3. tx-slowport-oper-delay 4. Txwait Example: MDS9710(config-port-monitor)# counter txwait poll-interval 1 delta rising- threshold 40 event 4 falling-threshold 0 event 4 portguard cong-isolate The above will congestion isolate a device if it has 400ms of TxWait in a 1 second interval

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 Slow Drain Alerting and Prevention Congestion-Isolation – Normal Traffic Flow VL3 All traffic Initially all traffic is normal and sent through VL3

Physical link(ISL) in ER_RDY mode

Host Target1 1

MDS1 MDS2 Target2 Active Zoning Virtual link - VL3(Normal VL) Host Zone1 2 Host1 <-> Target1 VL0 – Control traffic VL1 – High-priority traffic Zone2 VL2 – Slow traffic Host2 <-> Target2 VL3 – Normal traffic

Congestion-Isolation of slow devices – Scenario 1 All traffic to Once slow device is detected by port-monitor traffic to it is isolated to VL2 Host 1 sent across VL2 MDS2 limits traffic by not sending R_RDYs to Target1 Host1 Slow device detected! FCNS “slow-dev” No R_Rdy Sent Target1 No R_Rdy Sent MDS1 No R_Rdy MDS2 Host2 Sent(VL2) VL2 – Slow traffic Target2 VL3 – Normal traffic Active zoning All traffic to Zone1 Host 2 still sent across Host1 <-> Target1 Host2 <- Target2 traffic is not impacted! VL3 Zone2 Host2 <-> Target2 #CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82 Slow Drain Alerting and Prevention

Congestion-Isolation of slow devices - Scenario 2 All traffic to All traffic to the slow device is isolated to VL2 impacting Host 2 as well Host 1 sent Zoning overlaps cause traffic from common targets to be affected across VL2 Host1 Slow device detected! FCNS “slow-dev” No R_Rdy Target1 Sent No R_Rdy Sent MDS1 No R_Rdy MDS2 Host2 Sent(VL2) VL2 – Slow traffic Active Zoning Target2 VL3 – Normal traffic Zone1 All traffic to Host1 <-> Target1 Target1 and Target2->Host2 traffic is impacted! Host 2 still Host1 <-> Target2 sent across Target1 and Target2->Host1 traffic is impacted! VL3 Zone2 Host2 <-> Target1 Host2 <-> Target2

• Only works for “traditional” slow drain

• Only works across ISLs so has no effect in single switch fabrics

• FCoE is currently not supported

• If slow port is NPV connection all devices on port will be isolated

• No automatic de-isolation currently

• Several congestion-isolation commands available • Manually configure devices as slow • Display devices that are congestion-isolated

• See the Configuration Guide for more details

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 Slow Drain Alerting and Prevention Summary Proactive • Configure a lower congestion-drop on F ports system timeout congestion-drop 200 logical-type edge System timeout fcoe congestion-drop 200 mode edge Don’t go below 200ms!

• Configure no-credit-drop on F ports system timeout no-credit-drop 100 logical-type edge 200ms – safe, 100ms – aggressive, 50ms – Very aggressive

• Configure pause-drop on F ports system timeout fcoe pause-drop 100 mode edge 200ms – safe, 100ms – aggressive

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 Slow Drain Alerting and Prevention Summary Proactive • Configure port-monitor policy(s) Use samples included in port-monitor section Make sure to include tx-datarate at 80%/79% polling-interval 10 seconds

• Configure switchport logical-type core on interfaces to NPV switches MDS9710(config-if)# switchport logical-type core

• Consider separate physical ISL topology for older/slower VSANs

• Consider implementing congestion-isolation feature

4/8/16/32 G Fibre Channel module for MDS 9700

Pervasive | No appliance | No probes | Always on

High Performance End to End Onboard analytics Visibility for Scale with MDS 9700 engine for data trouble shooting SCSI level flow data Director Platform collection

Hardware is available in April 2017. Analytics#CLUS functionality© 2018available Cisco and/or its affiliates. in AllNX rights -reserved.OS Cisco 8.2(1) Public 32G FC Module for MDS 9700 - DS-X9648-1536K9 SCSI Performance Exchange Metrics Exchange Completion time Write CMD Data Access Latency MDS MDS Write Outstanding IO XFER_RDY Data Access Latency Exchange Data… Completion IOPS & other flow level counters Time STATUS Per flow timeout drop frames Read Failed exchanges, IO retransmissions CMD

MDS MDS Read SCSI error conditions (Aborts, Rejects, Data Access etc) Exchange Data… Latency Completion IO block size & other detailed flow Time STATUS level stats To be measured at SID-DID-LUN level

1 2 3

Cisco Existing Recommended

Possible • Native switch-integrated fabric-wide analytics • 32G module should “see” all frames at least once! • Investment protection of 16G FC module on MDS 9700 • Seamless & non-disruptive insertion of 32G FC module • High speed ISL  Increase performance with fewer links

• To enable feature • feature analytics

• To enable a specific port • analytics type fc-scsi

• Note analytics is a licensed feature • Traditional license – 3 year term • Smart License – Subscription based

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91 SAN Analytics Query Types • 2 query types • Pull Query • A one-time query used to extract the flow information that is stored in a database at the instant the query is executed. The output is in JSON format. • Overlay CLI—A predefined pull query that displays the flow metrics in a user- friendly tabular format.

• Periodic-export(push) Query • A recurring query installed to periodically extract the flow metrics that are stored in a database. The output is in JSON format.

• Query Syntax • select all | column1[,column2…] • from analytics_type.view_type • [where filter_list1 [[and | or ] filter_list2 …]] • [sort column] • [limit number]

Pull Query Examples MDS9710-1# show analytics query 'select all from fc-scsi.scsi_initiator' { "values": { "1": { "port": "fc9/16", "vsan": "1", "initiator_id": "0xc0100", ... 50 metrics total! "read_io_timeouts": "129", "write_io_timeouts": "0" }, "2": { "port": "fc9/32", "vsan": "1", "initiator_id": "0xc00e0", ... "read_io_timeouts": "2", "write_io_timeouts": "4" } }} #CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 SAN Analytics Pull Query Examples – Query with “where” key filter MDS9710-1# show analytics query 'select all from fc-scsi.scsi_initiator where port=fc9/16' { "values": { "1": { "port": "fc9/16", "vsan": "1", "initiator_id": "0xc0100", … "write_io_initiation_time_max": "1277985", "read_io_inter_gap_time_min": "26", "read_io_inter_gap_time_max": "33834735", "write_io_inter_gap_time_min": "26", "write_io_inter_gap_time_max": "56612131", "read_io_aborts": "0", "write_io_aborts": "0", "read_io_failures": "0", "write_io_failures": "0", "read_io_timeouts": "43346", "write_io_timeouts": "4237"

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95 SAN Analytics showanalytics Overlay CLI - Syntax MDS9710-1# showanalytics --help usage: analytics [-h] [--version] [--info] [--errors] [--initiator-itl] [--target-itl] [--interface INTERFACE] [--vsan VSAN] [--target TARGET] [--initiator INITIATOR] [--lun LUN] optional arguments: -h, --help show this help message and exit --version version --info --info | --errors mandatory --errors --info | --errors mandatory --initiator-itl --initiator-itl | --target-itl mandatory --target-itl --initiator-itl | --target-itl mandatory --interface INTERFACE fc interface --vsan VSAN vsan --target TARGET target FCID --initiator INITIATOR initiator FCID --lun LUN lun

showanalytics - Overlay CLI – Example - Info

MDS9710-1# showanalytics --initiator-itl --info

Interface fc9/32 +------+------+------+------+ | VSAN|I|T|L | Avg IOPS | Avg Thput (B/s) | Avg ECT (usec) | +------+------+------+------+ | | Read|Write | Read|Write | Read|Write | | | | | | | 1|0xc00e0|0x560020|0000-0000-0000-0000 | 0|0 | 0|0 | 10208|6032 | +------+------+------+------+

Interface fc9/16 +------+------+------+------+ | VSAN|I|T|L | Avg IOPS | Avg Thput (B/s) | Avg ECT (usec) | +------+------+------+------+ | | Read|Write | Read|Write | Read|Write | | | | | | | 1|0xc0100|0x560001|0000-0000-0000-0000 | 2311|1199 | 302940160|39305216 | 1936|2300 | | 1|0xc0100|0x560020|0000-0000-0000-0000 | 2409|3661 | 78946304|119980032 | 219|5238 | +------+------+------+------+

showanalytics - Overlay CLI - Example - Errors

MDS9710-1# showanalytics --interface fc9/16 --initiator-itl --errors

Interface fc9/16 +------+------+------+ | VSAN|I|T|L | Total SCSI Failures | Total FC Aborts | +------+------+------+ | | Read|Write | Read|Write | | | | | | 1|0xc0100|0x560001|0000-0000-0000-0000 | 0|0 | 0|0 | | 1|0xc0100|0x560020|0000-0000-0000-0000 | 0|0 | 0|2 | +------+------+------+

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98 SAN Analytics showanalytics - Overlay CLI – python source • “source copy-sys” command copies all scripts to bootflash:scripts

MDS9710-1# show file bootflash:/scripts/analytics.py #!/usr/bin/env python

############################################################## # Copyright (c) 2017 by Cisco Systems, Inc. # ############################################################## import sys import argparse import json from prettytable import * import cli …

Analytics congestion-drops

MDS9706-2# show analytics type fc-scsi flow congestion-drops |======| | | | Source |Destination| Congestion | Timestamp | | INTF | VSAN | FCID | FCID | Drops(delta) | | |======| | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC99 | 00000002 | 1. 02/08/18 15:27:18.051 | | fc6/1 | 0237 | 0x350040 | 0xCC0060 | 00000192 | 2. 02/08/18 15:27:08.053 | | fc6/1 | 0237 | 0xFFFC35 | 0xFFFCB1 | 00000002 | 3. 02/08/18 15:27:03.050 | | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC99 | 00000002 | 4. 02/08/18 15:27:03.050 | | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC99 | 00000002 | 5. 02/08/18 15:26:48.039 | | fc6/1 | 0237 | 0x350040 | 0xCC0060 | 00000116 | 6. 02/08/18 15:26:48.039 | | fc6/1 | 0237 | 0x350040 | 0xCC0060 | 00000103 | 7. 02/08/18 15:26:33.031 | | fc6/1 | 0237 | 0x350040 | 0xCC0060 | 00000170 | 8. 02/08/18 15:26:28.028 | | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC99 | 00000002 | 9. 02/08/18 15:26:18.019 | | fc6/1 | 0237 | 0xFFFC35 | 0xFFFCB1 | 00000002 |10. 02/08/18 15:26:08.012 | | fc6/1 | 0237 | 0xFFFC35 | 0xFFFCED | 00000001 |11. 02/08/18 15:26:08.012 | | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC0C | 00000001 |12. 02/08/18 15:26:08.012 | | fc6/1 | 0001 | 0xFFFC56 | 0xFFFC99 | 00000001 |13. 02/08/18 15:26:08.012 | | fc6/1 | 0237 | 0xFFFC35 | 0xFFFCB1 | 00000002 |14. 02/08/18 15:26:03.009 | |======|

Clear all statistics in view fc-scsi.scsi_initiator

• clear analytics 'select all from fc-scsi.scsi_initiator’

Clear all statistics in view fc-scsi.scsi_initiator only for port fc9/16

• clear analytics 'select all from fc-scsi.scsi_initiator where port=fc9/16'

• Understand and implement the many features provided

• MDS release notes have information on new features

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103 Additional Relevant Sessions Storage Networking – Cisco Live Orlando 2018 • BRKDCN-1008 - Managing Data Center Networks using Cisco Data Center Network Manager(DCNM) Art Scrimo Monday, Jun 11, 08:30 a.m. - 10:00 a.m.

• BRKDCN-2494 - NVMe, and NVMe over Fabrics Deep Dive J Metz Tuesday, Jun 12, 01:30 p.m. - 03:30 p.m.

• CTHDCN-2598 - Cisco SAN Insights: Knowledge is power! Art Scrimo Wednesday, Jun 13, 08:30 a.m. - 9:00 a.m.

• BRKDCN-1121 - Fiber Channel Networking for the IP Network Engineer and SAN Core Edge Design Chad Hintz Wednesday, Jun 13, 10:30 a.m. - 12:00 p.m.

• PSODCN-1130 - Cisco Storage Networking: Linking Technology and Innovation to Solve Business Pr Adarsh Viswanathan Wednesday, Jun 13, 03:30 p.m. - 04:30 p.m.

• BRKDCN-2010 - Designing Storage Networks for next decade in an All Flash Data Center Mark Allen Thursday, Jun 14, 10:30 a.m. - 12:00 p.m.

• BRKNMS-2002 - Management, Monitoring, and Automation of Data Center Deployments with DCNM 11 Ahmed Abeer Thursday, Jun 14, 10:30 a.m. - 12:00 p.m.

Give us your feedback to be entered into a Daily Survey Drawing. Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

#CLUS BRKSAN-2883 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105 Continue Demos in Walk-in Meet the Related your the Cisco self-paced engineer sessions education campus labs 1:1 meetings

#CLUS #CLUS