Dell PowerConnect and FTOS - Flow Control and Network Performance Dell Engineering September 2014 A Dell EMC Technical White Paper Revisions Version Date Authors 2.0 September 2014 Ed Blazek, Mike Matthews, Sulaimon Odunmbaku 1.0 Initial Release April 2011 Dell PowerConnect Team THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Copyright © 2011 - 2016 Dell Inc. or its subsidiaries. All Rights Reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. Except as stated below, no part of this document may be reproduced, distributed or transmitted in any form or by any means, without express permission of Dell. You may distribute this document within your company or organization only, without alteration of its contents. THIS DOCUMENT IS PROVIDED “AS-IS”, AND WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED. IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE SPECIFICALLY DISCLAIMED. PRODUCT WARRANTIES APPLICABLE TO THE DELL PRODUCTS DESCRIBED IN THIS DOCUMENT MAY BE FOUND AT: http://www.dell.com/learn/us/en/vn/terms-of-sale-commercial-and-public-sector-warranties Performance of network reference architectures discussed in this document may vary with differing deployment conditions, network loads, and the like. Third party products may be included in reference architectures for the convenience of the reader. Inclusion of such third party products does not necessarily constitute Dell’s recommendation of those products. Please consult your Dell representative for additional information. Trademarks used in this text: Dell™, the Dell logo, Dell Boomi™, Dell Precision™ ,OptiPlex™, Latitude™, PowerEdge™, PowerVault™, PowerConnect™, OpenManage™, EqualLogic™, Compellent™, KACE™, FlexAddress™, Force10™ and Vostro™ are trademarks of Dell Inc. Other Dell trademarks may be used in this document. Cisco Nexus®, Cisco MDS®, Cisco NX-0S®, and other Cisco Catalyst® are registered trademarks of Cisco System Inc. EMC VNX®, and EMC Unisphere® are registered trademarks of EMC Corporation. Intel®, Pentium®, Xeon®, Core® and Celeron® are registered trademarks of Intel Corporation in the U.S. and other countries. AMD®is a registered trademark and AMD Opteron™, AMD Phenom™ and AMD Sempron™ are trademarks of Advanced Micro Devices, Inc. Microsoft®, Windows®, Windows Server®, Internet Explorer®, MS-DOS®, Windows Vista® and Active Directory® are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat® and Red Hat® Enterprise Linux® are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell® and SUSE® are registered trademarks of Novell Inc. in the United States and other countries. Oracle® is a registered trademark of Oracle Corporation and/or its affiliates. Citrix®, Xen®, XenServer® and XenMotion® are either registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries. VMware®, Virtual SMP®, vMotion®, vCenter® and vSphere® are registered trademarks or trademarks of VMware, Inc. in the United States or other countries. IBM® is a registered trademark of International Business Machines Corporation. Broadcom® and NetXtreme® are registered trademarks of Broadcom Corporation. QLogic is a registered trademark of QLogic Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others. 2 Dell Networking - Flow Control and Network Performance | version 2.0 Table of contents Revisions ............................................................................................................................................................................ 2 1 Introduction ................................................................................................................................................................... 4 2 Types of Flow Control .................................................................................................................................................. 5 2.1 Ethernet Pause Frame Flow Control .................................................................................................................. 5 2.2 Priority-Based Flow Control ................................................................................................................................ 6 3 Ethernet Pause Frame Flow Control and TCP ............................................................................................................. 7 3.1 TCP implementation of Flow Control .................................................................................................................. 7 3.2 Ethernet Pause Frame Flow Control interoperability with TCP Flow Control..................................................... 7 4 Industry Flow Control Implementations ........................................................................................................................ 9 4.1 Dell PowerConnect Ethernet Pause Frame Flow Control Implementation ........................................................ 9 4.2 Dell FTOS Ethernet Pause Frame Flow Control Implementation .................................................................... 10 4.2.1 On Dell FTOS Switches .................................................................................................................................... 10 4.2.2 On Dell M I/O Aggregator Modular Switch ....................................................................................................... 11 4.2.2.1 Standalone Mode ......................................................................................................................................... 11 4.2.2.2 Stacking Mode ............................................................................................................................................. 12 4.2.2.3 VLT Mode .................................................................................................................................................... 12 4.2.2.4 Programmable MUX (PMUX) Mode ............................................................................................................ 12 4.3 Other Vendors Flow Control Implementation ................................................................................................... 13 5 Ethernet Pause Frame Flow Control Implementation Issues and Recommendations .............................................. 15 5.1 External Head of Line Blocking ........................................................................................................................ 15 5.2 Congestion Spreading ...................................................................................................................................... 16 5.3 Recommendations ............................................................................................................................................ 16 5.3.1 Asymmetric Flow Control .................................................................................................................................. 17 5.3.2 Asymmetric with Symmetric Flow Control ........................................................................................................ 17 5.3.3 Flow Control on iSCSI Ethernet Network ......................................................................................................... 17 5.3.4 Disable Flow Control on Network Core ............................................................................................................ 18 5.3.5 Global Disable of Flow Control ......................................................................................................................... 18 6 Ethernet Pause Frame Flow Control on Stacking and VLT Interfaces ...................................................................... 19 6.1 Dell PowerConnect and FTOS Stacking .......................................................................................................... 19 6.2 Dell FTOS Virtual Link Trunking ....................................................................................................................... 19 7 Summary .................................................................................................................................................................... 20 A References ................................................................................................................................................................. 21 Support and Feedback ..................................................................................................................................................... 22 3 Dell Networking - Flow Control and Network Performance | version 2.0 1 Introduction Flow control is defined in Annex 31B “MAC Control PAUSE operation” of the IEEE 802.3 Standard [1]. Transmission of Annex 31B PAUSE frames may be useful when deployed at the edge of a network for certain specific situations, but is generally considered harmful in the network core due to possible poor network performance when flow control is enabled. Due to this, Annex 31B flow control is not considered a viable method