Taking Control of SDN-based Cloud Systems via the Data Plane Kashyap Thimmaraju Bhargava Shastry Tobias Fiebig Felicitas Hetzelt Security in Security in Faculty of Technology, Security in Telecommunications Telecommunications Policy and Management Telecommunications TU Berlin TU Berlin TU Delft TU Berlin Berlin, Germany Berlin, Germany Delft, Netherlands Berlin, Germany kash@sect:tu-berlin:de bshastry@sect:tu-berlin:de t:fiebig@tudelft:nl file@sect:tu-berlin:de Jean-Pierre Seifert Anja Feldmann Stefan Schmid∗† Security in Internet Network Faculty of Computer Telecommunications Architectures Science TU Berlin TU Berlin University of Vienna Berlin, Germany Berlin, Germany Vienna, Austria jpseifert@sect:tu-berlin:de anja@inet:tu-berlin:de schmiste@univie:ac:at ABSTRACT KEYWORDS Virtual switches are a crucial component of SDN-based cloud sys- Network Isolation; Network Virtualization; Data Plane Security; tems, enabling the interconnection of virtual machines in a flexible Packet Parsing; MPLS; Virtual Switches; Open vSwitch; Cloud Se- and “software-defined” manner. This paper raises the alarm onthe curity; OpenStack; Attacker Models; ROP; SDN; NFV security implications of virtual switches. In particular, we show that virtual switches not only increase the attack surface of the cloud, 1 INTRODUCTION but virtual switch vulnerabilities can also lead to attacks of much higher impact compared to traditional switches. Modern cloud systems such as OpenStack [1], Microsoft Azure [2] We present a systematic security analysis and identify four de- and Google Cloud Platform [3] are designed for programmabil- sign decisions which introduce vulnerabilities. Our findings moti- ity, (logically) centralized network control and global visibility. vate us to revisit existing threat models for SDN-based cloud setups, These tenets also lie at the heart of Software-defined Networking and introduce a new attacker model for SDN-based cloud systems (SDN) [4, 5] which enables cloud providers to efficiently utilize their using virtual switches. resources [6], manage their multi-tenant networks [7], and reason We demonstrate the practical relevance of our analysis using a about orchestration [8]. case study with Open vSwitch and OpenStack. Employing a fuzzing The data plane of Software-Defined Networks in the cloud are methodology, we find several exploitable vulnerabilities in Open highly virtualized [7]: Virtual switches (running on the servers) are vSwitch. Using just one vulnerability we were able to create a worm responsible for providing connectivity and isolation among virtual that can compromise hundreds of servers in a matter of minutes. machines [9]. Prominent virtual switches today are: Open vSwitch Our findings are applicable beyond virtual switches: NFV and (OvS) [10], Cisco Nexus 1000V [11], VMware vSwitch [12] and high-performance fast path implementations face similar issues. Microsoft VFP [2]. This paper also studies various mitigation techniques and discusses Virtual switches are typically not limited to provide traditional how to redesign virtual switches for their integration. switching but support an increasing number of network and middle- box functionality [2, 13], e.g., routing, firewalling, network address translation and load-balancing. Placing such functionality at the virtualized edge of the network (i.e., the servers) is attractive, as it ∗ Also with, Internet Network Architectures, TU Berlin. allows to keep the network fabric simple and as it supports scala- †Also with, Dept. of Computer Science, Aalborg University. bility [2, 9]. However, the trend to move functionality from the network fabric to the edge (virtual switch) also comes at the price of increased Permission to make digital or hard copies of all or part of this work for personal or complexity. For example, the number of protocols that need to be classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation parsed and supported by virtual switches (Open vSwitch and Cisco on the first page. Copyrights for components of this work owned by others than the Nexus 1000v) and OpenFlow [4] have been growing steadily over author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or the last years [14] (see Fig. 1). republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. The trend towards more complex virtual switches is worrisome SOSR’18, March 28-29, 2018, Los Angeles, CA, USA as it may increase the attack surface of the virtual switch. For © 2018 Copyright held by the owner/author(s). Publication rights licensed to the example, implementing network protocol parsers in the virtual Association for Computing Machinery. ACM ISBN ......$15.00 switch is non-trivial and error-prone [15, 16]. These observations https://doi:org/::: lead us in this paper to conduct a security study of virtual switches. integrated the fixes. Ubuntu, Redhat, Debian, Suse, Mirantis, and OF 1.3 ovs-0.90.4OF 1.0ovs-1.0.0 OF 1.1 ovs-1.2.0OF 1.2 ovs-1.7.0 ovs-1.11.0 ovs-2.3.0 ovs-2.4.0 ovs-2.6.0ovs-2.7.0 other stakeholders have applied these fixes in their stable releases. 50 Furthermore, CVE-2016-2074 and CVE-2016-10377 were assigned 40 to the discovered vulnerabilities. 30 Structure: We provide necessary background information on SDN 20 in the cloud and virtual switches in Section 2. Section 3 introduces Parsed Protocols 10 and discusses our security analysis of virtual switches and existing 0 threat models. Based on this analysis we propose a new attacker Jul-2009 Jul-2010 Jul-2011 Jul-2012 Jul-2013 Jul-2014 Jul-2015 Jul-2016 Jul-2017 model. Section 4 presents a proof-of-concept case study attack on Jan-2009 Jan-2010 Jan-2011 Jan-2012 Jan-2013 Jan-2014 Jan-2015 Jan-2016 Jan-2017 Time OvS in OpenStack. We then investigate how our findings on OvS Open vSwitch OpenFlow vNexus relate to other virtual switches, high performance fast paths and SDN/NFV in Section 5. Subsequently, we discuss possible software Figure 1: The total number of parsed high-level protocols in mitigations and their performance impact in Section 6, and design two popular virtual switches and OpenFlow from 2009-2017. countermeasures in Section 7. After discussing related work in Section 8, we conclude in Section 9. Our contributions: 2 BACKGROUND • We present a systematic security analysis of virtual switches. This section reviews the background necessary to understand the We find that virtual switches not only increase the attack remainder of this paper. surface of an SDN-based cloud system (compared to their traditional counterparts), but can also have a much larger 2.1 SDN in the Cloud impact on cloud systems. Centralized network control and network-wide view are key design • Our analysis reveals four main factors that cause security goals for state-of-the-art cloud operating systems such as Open- issues: The co-location of virtual switches with the server’s Stack [1] and other commercial cloud providers, e.g., Google Cloud virtualization layer (in user- and kernel-space); centralized Platform [3]. Data plane isolation is typically ensured using separate control; complex packet parsing (and processing) of attacker physical/logical networks (guest, management and external) and controlled data. tunneling technologies [17] such as VLAN, GRE, VXLAN, MPLS, • Our findings motivate us to revisit current threat models. We etc. A cloud network generally comprises of a physical network observe that existing models do not encompass the security consisting of physical switches interconnecting virtualized servers issues identified in this paper leading us to introduce anew and an overlay (virtual) network interconnecting the VMs. The attacker model for the operation of virtualized data plane centralized control is attractive as it reduces the operational cost components in a Software-defined Network as well as in the and complexity of managing cloud networks [18]. It also provides context of Network Function Virtualization (NFV): A low- flexibility for managing and using cloud services, including VM budget attacker can cause significant harm on SDN-based migration. In addition, the servers in the cloud are connected to cloud systems. external networks (e.g., Internet) via a network node (router). In • We demonstrate the practical feasibility of our attacks on SDN, OpenFlow [4] is the de facto standard south bound protocol. OvS, a popular open-source virtual switch implementation Via the OpenFlow API, the controller can add, remove, update, and used in SDN-based cloud systems. This case study shows monitor flow tables and their flows. that commonly used virtual switch implementations are not resilient against our attacker model. Indeed, such an attacker 2.2 Virtual Switches can successfully exploit a whole SDN-based cloud setup The network’s data plane(s) can either be distributed across virtual- within minutes. ized servers or across physical (hardware) switches. OvS, VMware • We extend our study by surveying high performance fast vSwitch, and Cisco Nexus 1000V are examples of the former and paths, other virtual switch implementations, and related SDN are commonly referred to as virtual switches. Cisco VN-Link [19] and NFV technologies. We find that they are also susceptible and Virtual Ethernet Port Aggregator (VEPA) [20] are examples of to the same design issues. Furthermore, we find that soft- the latter. ware mitigations are commonly not considered during the Virtual switches have the advantage that inter-VM traffic, i.e., evaluation of new data plane components. traffic between VMs on the same server, does not leave that server. • We find that software mitigations for the vulnerabilities we The performance overhead of software-only switching (e.g., OvS) exploited could be adopted with a small performance penalty can be alleviated by hardware-offloading features: While tradition- for real-world traffic scenarios.