Generic and Automatic Address Configuration for Data Center Networks¤ Kai Chen?y, Chuanxiong Guoy, Haitao Wuy, Jing Yuanz?, Zhenqian Feng]y, Yan Chen?, Songwu Lux, Wenfei Wu\y ?Northwestern University, yMicrosoft Research Asia, zTsinghua University, ]NUDT, xUCLA, \BUAA ?{kchen,ychen}@northwestern.edu, y{chguo,hwu,v-zhfe,v-wenfwu}@microsoft.com, [email protected], [email protected] ABSTRACT 1. INTRODUCTION Data center networks encode locality and topology information into 1.1 Motivation their server and switch addresses for performance and routing pur- poses. For this reason, the traditional address configuration protocols Mega data centers [1] are being built around the world to pro- such as DHCP require huge amount of manual input, leaving them vide various cloud computing services such as Web search, online error-prone. social networking, online office and IT infrastructure out-sourcing In this paper, we present DAC, a generic and automatic Data cen- for both individual users and organizations. To take the advantage of ter Address Configuration system. With an automatically generated economies of scale, it is common for a data center to contain tens or blueprint which defines the connections of servers and switches la- even hundreds of thousands of servers. The current choice for build- beled by logical IDs, e.g., IP addresses, DAC first learns the physical ing data centers is using commodity servers and Ethernet switches topology labeled by device IDs, e.g., MAC addresses. Then at the for hardware and the standard TCP/IP protocol suite for inter-server core of DAC is its device-to-logical ID mapping and malfunction de- communication. This choice provides the best performance to price tection. DAC makes an innovation in abstracting the device-to-logical trade-off [2]. All the servers are connected via network switches to ID mapping to the graph isomorphism problem, and solves it with low form a large distributed system. time-complexity by leveraging the attributes of data center network Before the servers and switches can provide any useful services, topologies. Its malfunction detection scheme detects errors such as however, they must be correctly configured. For existing data centers device and link failures and miswirings, including the most difficult using the TCP/IP protocol, the configuration includes assigning IP case where miswirings do not cause any node degree change. address to every server. For layer-2 Ethernet, we can use DHCP [3] We have evaluated DAC via simulation, implementation and exper- for dynamic IP address configuration. But servers in a data center iments. Our simulation results show that DAC can accurately find all need more than one IP address in certain address ranges. This is be- the hardest-to-detect malfunctions and can autoconfigure a large data cause for performance and fault tolerance reasons, servers need to center with 3.8 million devices in 46 seconds. In our implementa- know the locality of other servers. For example, in a distributed file tion, we successfully autoconfigure a small 64-server BCube network system [4], a chunk of data is replicated several times, typically three, within 300 milliseconds and show that DAC is a viable solution for to increase reliability. It is better to put the second replica on a server data center autoconfiguration. in the same rack as the original, and the third replica on a server at another rack. The current practice is to embed locality information into IP addresses. The address locality can also be used to increase Categories and Subject Descriptors performance. For example, instead of fetching a piece of data from C.2.1 [Network Architecture and Design]: Network communica- a distant server, we can retrieve the same piece of data from a closer tions; C.2.3 [Network Operations]: Network management one. This kind of locality based optimization is widely used in data General Terms center applications [4, 5]. The newly proposed data center network (DCN) structures [6–9] Algorithms, Design, Performance, Management go one step further by encoding their topology information into their Keywords logical IDs. These logical IDs can take the form of IP address (e.g., in VL2 [9]), or MAC address (e.g., in Portland [8]), or even newly Data center networks, Address configuration, Graph isomorphism invented IDs (e.g., in DCell [6] and BCube [7]). These structures then leverage the topological information embedded in the logical IDs for ¤This work was performed when Kai, Zhenqian and Wenfei were interns at Microsoft Research Asia. scalable and efficient routing. For example, Portland switches choose a routing path by exploiting the location information of destination PMAC. BCube servers build a source routing path by modifying one digit at one step based on source and destination BCube IDs. Permission to make digital or hard copies of all or part of this work for For all the cases above, we need to configure the logical IDs, which personal or classroom use is granted without fee provided that copies are may be IP or MAC addresses or BCube or DCell IDs, for all the not made or distributed for profit or commercial advantage and that copies servers and switches. Meanwhile, in the physical topology, all the bear this notice and the full citation on the first page. To copy otherwise, to devices are identified by their unique device IDs, such as MAC ad- republish, to post on servers or to redistribute to lists, requires prior specific dresses. A naïve way is to build a static device-to-logical ID mapping permission and/or a fee. SIGCOMM’10, August 30–September 3, 2010, New Delhi, India. table at the DHCP server. Building such a table is mainly a manual Copyright 2010 ACM 978-1-4503-0201-2/10/08 ...$10.00. effort which does not work for the following two reasons. First of all, the scale of data center is huge. It is not uncommon that a mega data center can have hundreds of thousands of servers [1]. Secondly, switch manual configuration is error-prone. A recent survey from 100 data center professionals [10] suggested that 57% of the data center out- ages are caused by human errors. Two more surveys [11, 12] showed server 50%-80% of network downtime is due to human configuration er- rors. In short, “the vast majority of failures in data centers are caused, (a). Blueprint: (b). Physical network topology: triggered or exacerbated by human errors." [13] Each node has a logical ID Each device has a device ID Figure 1: An example of blueprint, and physical topology con- 1.2 Challenges and Contributions structed by following the interconnections in blueprint. Automatic address configuration is therefore highly desirable for data center networks. We envision that a good autoconfiguration sys- our solution is fast and is able to detect all the hardest-to-detect mal- tem will have the following features which also pose challenges for functions. building such a system. We have studied our DAC design via extensive experiments and ² Generality. The system needs to be applicable to various network simulations. The experimental results show that the time of our device- topologies and addressing schemes. to-logical ID mapping scales in proportion to the total number of de- vices in the networks. Furthermore, our simulation results show that ² Efficiency and scalability. The system should assign a logical ID DAC can autoconfigure a large data center with 3.8 million devices to a device quickly and be scalable to a large number of devices. in 46 seconds. We have also developed and implemented DAC as an ² Malfunction and error handling. The system must be able to han- application on a 64-server testbed, where the 64 servers and 16 mini- dle various malfunctions such as broken NICs and wires, and hu- switches form a two level BCube [7] network. Our autoconfiguration man errors such as miswirings. protocols automatically and accurately assign BCube logical IDs to ² Minimal human intervention. The system should require minimal these 64 servers within 300 milliseconds. manual effort to reduce human errors. The rest of the paper is organized as follows. Section 2 presents the system overview. Section 3 introduces the device-to-logical ID map- To the best of our knowledge, there are very few existing solutions ping. Section 4 discusses how DAC deals with malfunctions. Sec- and none of them can meet all the requirements above. In this pa- tion 5 and Section 6 evaluate DAC via experiments, simulations and per, we address these problems by proposing DAC – a generic and implementations. Section 7 discusses the related work. Section 8 automatic Data center Address Configuration system for all the ex- concludes the paper. isting and future data center networks. To make our solution generic, we assume that we only have a blueprint of the to-be-configured data center network, which defines how the servers and switches are con- 2. SYSTEM OVERVIEW nected, and labels each device with a logical ID. The blueprint can be One important characteristic shared by all data centers is that a automatically generated because all the existing data center network given data center is owned and operated by a single organization. structures are quite regular and can be described either recursively or DAC takes advantage of this property to employ a centralized auto- iteratively (see [6–9] for examples). configuration manager, which we call DAC manager throughout this Through a physical network topology learning procedure, DAC paper. DAC manager deals with all the address configuration intel- first automatically learns and stores the physical topology of the data ligences such as physical topology collection, device-to-logical ID center network into an autoconfiguration manager. Then, we make mapping, logical ID dissemination and malfunction detection.