DECUSATIS LAYOUT_Layout 1 8/23/12 12:38 PM Page 26 CLOUD COMPUTING: NETWORKING AND COMMUNICATIONS CHALLENGES Communication within Clouds: Open Standards and Proprietary Protocols for Data Center Networking Carolyn J. Sher DeCusatis and Aparico Carranza, New York City College of Technology Casimer M. DeCusatis, IBM Corporation BSTRACT A available on affordable commodity x86 servers Cloud computing and other highly virtualized within the last decade or so. In recent years, data center applications have placed many new many equipment vendors have contributed to and unique requirements on the data center net- the hardware and software infrastructure, which work infrastructure. Conventional network pro- has made enterprise-class virtualization widely tocols and architectures such as Spanning Tree available. This, in turn, enables new designs for Protocol and multichassis link aggregation can cloud computing, including hosting multiple limit the scale, latency, throughput, and virtual independent tenants on a shared infrastructure, machine mobility for large cloud networks. This rapid and dynamic provisioning of new features, has led to a multitude of new networking proto- and implementing advanced load balancing, cols and architectures. We present a tutorial on security, and business continuity functions some of the key requirements for cloud comput- (including multisite transaction mirroring). This ing networks and the various approaches that has brought about profound changes in many have been proposed to implement them. These aspects of data center design, including new include industry standards (e.g., TRILL, SPB, requirements for the data center network. software-defined networking, and OpenFlow), Modern cloud data centers employ resource best practices for standards-based data center pooling to make more efficient use of data center networking (e.g., the open datacenter interoper- appliances and to enable dynamic reprovisioning in able network), as well as vendor proprietary response to changing application needs. Examples approaches (e.g., FabricPath, VCS, and Qfabric). of this include elastic workloads where application components are added, removed, or resized based on the traffic load; mobile applications relocating INTRODUCTION to different hosts based on distance from the host or hardware availability; and proactive disaster Cloud computing is a method of delivering com- recovery solutions, which relocate applications in puting services from a large, highly virtualized response to a planned site shutdown or a natural data center to many independent end users, disaster. It has been shown [2] that highly virtual- using shared applications and pooled resources. ized servers place unique requirements on the data While there are many different definitions for center network. Cloud data center networks must cloud computing [1], it is typically distinguished contend with huge numbers of attached devices by the following attributes: on-demand self-ser- (both physical and virtual), large numbers of isolat- vice, broad network access, resource pooling, ed independent subnetworks, multitenancy (appli- rapid and elastic resource provisioning, and cation components belonging to different tenants metered service at various quality levels. are collocated on a single host), and automated Implementation of these attributes as part of creation, deletion, and migration of virtual a large enterprise-class cloud computing service machines (facilitated by large layer 2 network that provides continuous availability to a large domains). Furthermore, many cloud data centers number of users typically requires significantly now contain clusters or pods of servers, storage, more server, networking, and storage resources and networking, configured so that the vast majori- than conventional data centers (up to an order ty of traffic (80–90 percent in some cases) flows of magnitude more in many cases). This is only between adjacent servers within a pod (so-called achievable through extensive use of virtualiza- east-west traffic). This is a very different traffic pat- tion. While server virtualization has existed since tern from conventional data center networks, the 1960s, when it was first implemented on which supported higher levels of traffic between IBM mainframes, it has only become widely server racks or pods (so-called north-south traffic). 26 0163-6804/12/$25.00 © 2012 IEEE IEEE Communications Magazine • September 2012 DECUSATIS LAYOUT_Layout 1 8/23/12 12:38 PM Page 27 MC-LAG and LAG STP and LAG To help overcome LAG LAG the limitations of STP, several enhancements have MC-LAG MC-LAG been standardized. These include Multi- LAG STP blocked LAG LAG ple STP, which con- figures a separate spanning tree for Figure 1. MC-LAG configuration without STP (left) and with STP (right). each virtual local area network group To cope with these problems, many attempts EtherChannel [MCEC], also referred to as Vir- and blocks all but have been made to develop best practices for net- tual Port Channels [vPC] for Cisco switches.) working design and management. Several new The changing requirements of cloud data center one of the possible industry standards and proprietary network archi- networks are forcing designers to reexamine the alternate paths tectures have recently been proposed. Many net- role of STP. One of the drawbacks of a spanning work designers, users, and administrators have tree protocol is that in blocking redundant ports within each span- long expressed a desire for standardization and and paths, a spanning tree reduces the aggregate ning tree. multivendor interoperability in the data center, to available network bandwidth significantly. Addi- simplify management, improve reliability, and tionally, STP can result in circuitous and subop- avoid being locked into one particular vendor’s timal communication paths through the network, proprietary product offerings and pricing struc- adding latency and degrading application perfor- ture. These conclusions were supported by a mance. A spanning tree cannot easily be segre- recent analyst report [3], which determined that gated into smaller domains to provide better multisourcing of network equipment is not only scalability, fault isolation, or multitenancy. Final- practical, but can reduce total cost of ownership ly, the time taken to recompute the spanning by 15–25 percent. Furthermore, a recent survey of tree and propagate the changes in the event of a 468 business technology professionals on their failure can vary widely, and sometimes becomes data networking purchasing preferences [4] quite large (seconds to minutes). This is highly showed that adherence to industry standards was disruptive for elastic applications and virtual their second highest requirement, behind virtual- machine migrations, and can lead to cascaded ization support. Standardization also encourages system-level failures. future-proofing of the network and helps promote To help overcome the limitations of STP, sev- buying confidence. Despite these advantages, eral enhancements have been standardized. These many large network equipment providers have include Multiple STP (MSTP), which configures a advocated for proprietary network protocols. A separate spanning tree for each virtual local area recent study [5] showed that five out of the six network (VLAN) group and blocks all but one of largest network equipment manufacturers include the possible alternate paths within each spanning proprietary features in their products, and only tree. Also, the link aggregation group (LAG) three of the six claimed interoperability with standard (IEEE 802.3ad) allows two or more other vendors’ access layer switches. physical links to be bonded into a single logical In this article, we present a tutorial on cloud link, either between two switches or between a networking design practices, including both server and a switch. Since a LAG introduces a industry standard and vendor proprietary alter- loop in the network, STP has to be disabled on natives. It should be noted that although modern network ports using LAGs. It is possible for one data centers will almost certainly require some end of the link aggregated port group to be dual- version of these new protocols, many of these homed into two different devices to provide approaches are far less mature than convention- device-level redundancy. The other end of the al network designs. Early adopters should use group is still single-homed and continues to run caution when evaluating the best choices for normal LAG. This extension to the LAG specifi- their data center needs. cation is called multichassis link aggregation (MC-LAG), and is standardized as IEEE 802.1ax (2008). As shown in Fig. 1, MC-LAG can be used SPANNING TREE PROTOCOL AND to create a loop-free topology without relying on ULTICHASSIS INK GGREGATION STP; because STP views the LAG as a single link, M L A it will not exclude redundant links within the Spanning Tree Protocol (STP) is a layer 2 switch- LAG. For example, it is possible for a pair of net- ing protocol used by classical Ethernet that work interface cards (NICs) to be dual-homed ensures loop-free network topologies by always into a pair of access switches (using NIC team- creating a single path tree structure through the ing), and then use MC-LAG to interconnect the network. In the event of a link failure or recon- access switches with a pair of core switches. figuration, the network halts all traffic while the Most MC-LAG systems allow dual homing spanning tree algorithm