Nemu: a Distributed Testbed for the Virtualization of Dynamic, Fixed and Mobile Networks Vincent Autefage, Damien Magoni

NEmu: A distributed testbed for the virtualization of dynamic, fixed and mobile networks Vincent Autefage, Damien Magoni

To cite this version:

Vincent Autefage, Damien Magoni. NEmu: A distributed testbed for the virtualization of dynamic, fixed and mobile networks. Computer Communications, Elsevier, 2016, 80, pp.33-44. 10.1016/j.comcom.2016.01.005. hal-01436511

HAL Id: hal-01436511 https://hal.archives-ouvertes.fr/hal-01436511 Submitted on 16 Jan 2017

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. NEmu: a Distributed Testbed for the Virtualization of Dynamic Fixed and Mobile Networks

Vincent Autefage, Damien Magoni∗ University of Bordeaux, LaBRI 351, Cours de la Liberation 33400 Talence, France

Abstract Experimentation is typically the last step before launching a network application on a large production scale. However, it is often diﬃcult to gather enough hardware resources for experimenting with a reasonably sized distributed application inside a controlled environment. Virtualization is thus a handy technique for creating such an experimentation testbed. We propose a tool called NEmu designed to create virtual dynamic networks by using emulation for testing and evaluating prototypes of networked or distributed applications with a complete control over the network topology and link parameters. NEmu leverages system emulators such as QEMU for virtualizing the hosts and the routers. It uses vnd for virtualizing components such as links and switches. In addition, NEmu allows users to create such customized topologies with limited hardware resources and without any administrative rights. We validate NEmu by replicating two network experiments and by showing that NEmu gives results very similar to the original ones. Keywords: Emulation, mobile, network, testbed, virtualization.

1. Introduction for reducing the equipment and space costs as well as the energy consumption of using physical hosts [3]. Experimentation is important to realistically and ac- Our solution to overcome the above hardware constraints curately test and evaluate network applications. Exper- is thus to build a test bed able to set up virtualized net- imentation on algorithms is usually made by simulation. works. A virtual network uses virtual machines instead This technique is available through well known software of physical hosts and connects them with virtual links in like ns [1] or OMNeT++ [2] and enables to evaluate the order to build a virtual network topology. The virtual ma- efficiency and the scalability of algorithms or protocols. chines of a virtual network can be hosted on one or several Experimentation on a real program, i.e. implementation, physical hosts depending on the number of virtual ma- is different to the extent that it is more focused on execu- chines needed and the resources capacities of the physical tion time, processor usage, memory consumption, network ones. properties, etc. We propose a tool designed to create virtual networks It can be a difficult task when trying to experiment for testing and evaluating prototypes of applications on with a network application involving dozens of machines or the top of static, dynamic or mobile networks with a com- more. Moreover, the mobility or the dynamics of scenarios plete control over the network topology and link properties can drastically increase the difficulty of experimentation. (bandwidth, delay, bit error rate, etc.) and the mobility Using the Internet as a test bed is impractical as no pa- of nodes. The goal of our tool is to enable the creation rameters can be controlled. Setting up a hardware test of reasonably sized virtual networks while minimizing the bed is expensive and cumbersome. Furthermore, network number of necessary physical hosts and network equip- applications can have very different ways of connecting ment needed. It can build host-based overlay networks hosts to each others and changing the network topology by using emulators such as QEMU [4]. We have called and network parameters of a hardware test bed is time our tool NEmu which stands for Network Emulator for consuming and error prone. Virtualization techniques for mobile universes because it is able to create both fixed creating such an experimentation test bed can save re- and mobile emulated networks. It is also a tribute to the sources and ease manipulations. It is a proven method name of the QEMU software which is a powerful machine emulator heavily used by NEmu. The contributions of our ∗ work are as follows: Corresponding author: phone: +33-5-4000-3540, fax: +33-5- 4000-6669 • Email addresses: [email protected] (Vincent Autefage), A detailed description of our NEmu software which [email protected] (Damien Magoni) is able to manage a distributed set of virtual nodes

Preprint submitted to Computer Communications January 14, 2016 and links for emulating any arbitrary static, dynamic In addition, NEmu includes three important extra prop- or mobile network topology (Section 2). erties:

• A detailed description of our nemo tool which im- • The accessibility which means that NEmu can be plements the mobility inside NEmu and enables to fully executed without any administrative rights on create a complete autonomous mobile network by the physical infrastructure. Indeed, the major part following a predefined scenario (Section 4). of public infrastructures, like universities and lab- oratories, does not provide administrative access to • A performance validation experiment by making a their users in order to ensure the security and the in- comparison between NEmu and Mininet [5] (Section tegrity of the whole domain. Therefore, the user ex- 5). ecution would allow most people to use NEmu freely. • A state of the art on related and previous work tar- • The dynamicity of the topology enables node hot- geted at networking emulation and a comparison of connections which means that a virtual node can join NEmu the features provided by with the ones offered or leave the topology dynamically without perturb- by similar alternative virtual networking testbeds ing the overall virtual network. (Section 6). • The mobility of nodes provides a way to create a self This paper is an extended and revised version of our predefined topology evolution through time and space. vious work published in [6]. In other words, it is possible to create an autonomous connectivity scenario. 2. Description of NEmu • The community aspect of the virtual network pro- 2.1. Overall Design vides the possibility for several people to supply vir- NEmu is a python program consisting of 6000 lines tual sub-networks in order to build a community net- of code which allows to build a dynamic and distributed work like the Internet is. virtual network infrastructure. It is based on the concept of Network Virtualization Environment (NVE) introduced 2.2. Network Elements by N. Chowdhury and R. Boutaba in [7]. The main charac- NEmu is a distributed virtual network environment teristic of a NVE is that it hosts multiple Virtual Networks which allows users to create arbitrary and dynamic topolo- (VN) that are firstly not aware of one another, and that gies. To this end NEmu is based on different building are secondly completely independent of each other. A VN blocks. NEmu uses virtual nodes connected by virtual links is a set of virtual nodes connected by virtual links in order in order to create a virtual network topology. A virtual to form a virtual topology. NEmu provides the possibility topology can be hosted by one or several physical hosts. of creating several virtual network topologies with the cen- The part of the virtual topology laying on a given physical tral property that a VN is strictly disjoint from another in host represents a NEmu session which is configured by the order to ensure the integrity of each VN. NEmu manager. Thus, NEmu integrates characteristics that are funda- mental to a NVE: First, the flexibility and heterogeneity 2.2.1. Virtual Node allows the user to construct a customized topology, with A virtual node for NEmu is an emulated machine that custom virtual nodes and virtual links. The scalability al- requires a hard disk image to work. This image is typically lows different virtual nodes to be hosted by different phys- provided as a regular file on the physical host machine. ical hosts in order to avoid limitations of a unique physical Two types of virtual nodes currently exist in NEmu: machine. The isolation decouples the different virtual net- • works which run on the same infrastructure. It also guar- A VHost is a virtual host machine (i.e., end-user ter- anties a strict separation between the host and the virtual minal) on which the hardware properties and the op- networks. The stability ensures that faults in a virtual erating system can be fully configured by the user. network would not affect another one. The manageabil- • A VRouter is a virtual router directly configured by ity ensures that the virtual network and the physical in- NEmu and provides ready-to-use network services. frastructure are completely independent. Therefore, a VN created on an infrastructure A can be deployed on another Each virtual node uses a virtual storage which can be infrastructure B. The legacy support ensures that the NVE either a real media (cdrom, hard drive, etc.), a raw file or can emulate former devices and architectures. Finally, the a host directory. A raw file can be privately dedicated or programmability provides some optional network services shared by several other virtual nodes. A modification of to simplify the use of the virtual network (such as DHCP, a shared file by one virtual machine will affect the others DNS, etc.). It also implies that the user can develop and which may be troublesome if the file contains the operating integrate his own additional services. system. To solve the problem, NEmu uses Copy-on-Write (CoW) operations on the original file. A CoW file (also 2 known as a sparse file) only stores the differences with its • A VSwitch is a virtual multi-point switch emulat- original file. The advantage, compared to a regular copy, ing a physical Ethernet switch and interconnecting is that the CoW file is much smaller. In addition, NEmu several nodes. can use a regular directory on the physical host (without Virtual links building a CoW), as a storage media in four different ways: typically carry Ethernet frames from one virtual Network Interface Card (NIC) to one or more other • by making a Sparse file which only stores the differ- virtual NICs. This Ethernet traffic is tunneled between ences with its original file; virtual nodes by using TCP or UDP connections. NEmu can also use a VDE [13], a virtual switch which inter- • by making a Squash file system which is a read-only connects virtual machines through the shared memory sys- raw image; tem inside the Linux kernel, in order to create a local switch NEmu vnd • by using a FAT16 emulated interface which enables multi-point . Alternatively, can use our vnd a direct access to a host’s file system; program to emulate a network component. stands for virtual network device. The vnd is a C++ program which • by using a Virtio interface [8] which also enables a consists in 6k lines of code that can emulate a VLine, a direct access to a host’s file system; VHub or a VSwitch (defined as modes). The advantages of using a vnd is that the user can set the bandwidth, de- • by using a Network Block Device which enables a lay, jitter and bit error rate on any interface in any mode virtual node to remotely access to a block device whereas QEMU offers no control over its hub emulation. through the real IP network [9]; In addition, NEmu also provides a Slirp which is a spe- • by using a SSH tunnel which enables a virtual node cial type of link whose purpose is to provide an Internet to remotely access to a block device through a se- access to the virtual node. It is an emulation of a NATed cured connection. access to the real Internet by using the physical host NIC. Also, NEmu is able to interconnect a virtual NIC to a As said before, a VHost needs a disk image which must TUN/TAP kernel interface or to any UNIX socket. be supplied by the user. This image must be prepared Figure 1 shows an example of a NEmu managed virtual prior to creating the virtual network. Furthermore, one network. On the left side, two VHosts are connected to a image can be used by many VHosts by using sparse files. VRouter through a VSwitch by using UDP tunnels. On the NEmu provides a network topology visualization option right side, two VHosts are connected to the above VRouter by processing its topology data file through Graphviz [10]. through a VHub by using TCP tunnels. Here, virtual links A VRouter is directly configured by NEmu and pro- are created and managed inside vnd processes. vides several services to simplify the virtual network management: DHCP, DNS, NFS, HTTP, SSH, NTP, Netfil- 2.3. Management of Virtual Networks ter, dynamic routing protocols (RIP and OSPF), and QoS We explain below the notion of a NEmu session and management with Traffic Control [11]. Moreover, it is we describe the physical resources needed to run a virtual easily possible to add some new services through a plug-in network. system available in NEmu. A router is running a customized image version of TinyCore which is a lightweight 2.3.1. Session and highly configurable Linux distribution [12]. Such a As already said above, a NEmu session represents a system typically requires about ∼30 MBytes on disk and complete configuration of a network topology which lays ∼100 MBytes in memory with all services running. Ser- on a physical host (storages, virtual nodes configurations vices provided by a VRouter are optional and can be en- and links). A distributed virtual network on n physical abled or disabled before or during runtime. hosts consists in n NEmu sessions at least. A session is represented by an auto-generated directory in order to be 2.2.2. Virtual Link saved and re-used. A session can be saved as a sparse A virtual link for NEmu is an emulated network con- archive which compresses all elements and which is com- nection between virtual nodes. This emulated connection patible with sparse files unlike traditional archives. can either be performed inside the machine emulator of a node (the link thus being attached to this node) or be 2.3.2. Manager performed by a dedicated emulation program (not running The NEmu manager is the command line user interface any system image in this case). Three types of virtual links to manipulate a session. Sessions are independent even if currently exist in NEmu: they are part of the same network topology. The manager can be used in three ways : • A VLine is a virtual point-to-point link interconnecting two nodes. • as a python module to be integrated in another script or program; • A VHub is a virtual multi-point hub emulating a physical Ethernet hub and interconnecting several nodes. • as a dynamic python interpreter; 3 VHost VHost VSwitch VRouter VHostVHub VHost

Debian linux Debian linux TCP between virtual hosts Debian linux Debian linux

QEMU QEMU QEMU QEMU system system Tinycore linux system system emulation emulation emulation emulation vnd switch mode Ethernet over QEMU Ethernet over UDP unicast TCP unicast system vnd Ethernet over TCP unicast Ethernet over UDP unicast emulation Ethernet over hub mode Ethernet over TCP unicast TCP unicast NEmu managed network elements

Figure 1: Network elements in action.

• as a python script launcher. The NEmu manager provides a remote accesses, through SSH connections, to manipulate NEmu sessions laying on other distant hosts. The python language is upgraded in # Creates a new session order to interact with other distant sessions. InitNemu()

# New switch with 3 ports 2.4. Example of a Topology VSwitch(’switch’, niface=3)

We present in Figure 2 the Python script that generates # New hub with 3 ports the network topology previously shown in Figure 1. VHub(’hub’, niface=3)

# New router with DHCP and SSH services 2.5. Accuracy and Scalability VRouter("router", nics=[VNic(), VNic()], services=[ Service("ipforward"), NEmu does not currently provide any speciﬁc primi- Service("ifup", "0:192.168.1.1", "1:192.168.2.1"), tives or tools for measuring backend side performances or Service("dnsmasq", domain="local1", net="192.168.1.0/24", start="192.168.1.10", end="192.168.1.20"), for collecting results from all the nodes in the virtual net- Service("dnsmasq", domain="local2", net="192.168.2.0/24", work. The measurement tools available are all the usual start="192.168.2.10", end="192.168.2.20"), Service("sshd")]) system tools that can be installed and run either inside the virtual machines or on the physical hosts. Typical network # New host configuration (French keyboard, SDL display and 512MB of RAM) VHostConf(’chost’, sdl=None, k=’fr’, m=512) performance evaluation tools include: iperf, netperf, etc. The accuracy of NEmu is mainly limited by the underly- # New hosts with the chost common configuration and 2 NICs VHost(’a’, conf=’chost’, hds=[VFs(’debian.img’, type=’cow’], nics=[VNic()]) ing network characteristics (e.g., bandwidths, delays, error VHost(’b’, conf=’chost’, hds=[VFs(’debian.img’, type=’cow’], nics=[VNic()]) rates, etc) of the backend connections between the phys- VHost(’c’, conf=’chost’, hds=[VFs(’debian.img’, type=’cow’], nics=[VNic()]) VHost(’d’, conf=’chost’, hds=[VFs(’debian.img’, type=’cow’], nics=[VNic()]) ical hosts. It is obvious that the virtual bandwidth set between two virtual nodes will never be able to be above # Connects nodes to links Link(’router:0’, ’switch:0’) the real physical bandwidth available between the physical Link(’router:1’, ’hub:0’) machines hosting those virtual nodes. Finally, the scala- Link(’a’, ’switch:1’) Link(’b’, ’switch:2’) bility of NEmu is mainly limited by QEMU’s requirements Link(’c’, ’hub:1’) for the virtual machines. The needs for each VM is typi- Link(’d’, ’hub:2’) cally at least one dedicated core (if possible) and at least # Starts the virtual network a few hundred megabytes of RAM per VM. Thus, on a StartNemu() single regular machine, a virtual network could scale to a dozen nodes. On a group of regular machines or on a big Figure 2: A example of a topology script written for NEmu. server or cluster, it could scale to a hundred nodes.

4 VND VND

INPUT QUEUE INPUT QUEUE INPUT QUEUE INPUT QUEUE

FORWARDING INTERFACE 1 INTERFACE 2 INTERFACE 1 INTERFACE 2 ENGINE FORWARDING ENGINE OUTPUT QUEUE OUTPUT QUEUE OUTPUT QUEUE OUTPUT QUEUE

INPUT QUEUE INPUT QUEUE vnd Figure 3: Architecture of a . link mode INTERFACE 3 INTERFACE 4

OUTPUT QUEUE OUTPUT QUEUE

3. The Virtual Network Device (vnd)

This section presents our software program called vnd. It is a program which is able to emulate network devices Figure 4: Link mode. such as a link, hub, switch or an access point from a high level point of view. This is by far not the first software A vnd can be set to one of six different working modes able to emulate network devices but it has some unique depending on the network component that it emulates. features which may prove useful in the network virtualiza- The first four modes are typical network components which tion domain: are independent from any virtual machine. The last two • it runs as a lightweight stand-alone process and can modes are used to emulate a Wireless Interface Card (WIC) fail without killing virtual machines, in either infrastructure or ad hoc mode. Thus in the last two modes, the vnd is not used as a separate network com- • it can support dynamic connections and reconnec- ponent but it is used in conjunction with a virtual machine tions as well as disconnections and is immune to the to form a mobile node. When the vnd is used as a wireless failures of virtual machines, card emulator, it is connected to its virtual mobile node by a specific and unique interface called a direct inter- • it provides many networking backends, such as the face. If the mobile node is considered using infrastructure sockets API, which is available on any platform, to mode (BSS or ESS), then the vnd also has another spe- connect to the virtual machines, cific and unique interface called an access interface which • it can dynamically set the link properties such as is connected to the access point that the mobile node is bandwidth, delay, jitter and bit error rate, currently associated with. A vnd can be set to one of the six possible modes: • it can coarsely emulate wireless interface cards in 1. link: each interface is directly bound to another in- infrastructure and ad hoc modes as well as access terface, which means that any data going into the points. input of the first interface is forwarded to the output of the second interface in this given direction 3.1. Architecture (i.e., it is one way as shown on Figure 4), A vnd contains an engine and several interfaces. It can 2. hub: each interface is bound to all others, which contain any number of interfaces as long as system mem- means that any data going into the input of an in- ory is available. Interfaces can be created and destroyed terface is forwarded to the output of all the other at runtime. Each interface owns an input queue and an interfaces except itself as shown on Figure 5, output queue. Each queue has a number of buffers which 3. switch: any frame going into the input of an inter- can be set at runtime. Interfaces are internally connected face is forwarded to the switch engine which uses a through the engine. Figure 3 shows the architecture of a forwarding table to determine the output interface vnd with two interfaces. Data coming in or out of a vnd leading to the device having the same address as the can be interpreted in two ways: frame’s destination address as shown on Figure 6, • raw: data is considered as an uninterpreted flow of 4. access point: any frame going into the input of an bytes and each buffer can contain data bytes up to interface is forwarded to the switch engine which uses its maximum size, a forwarding table to determine the output interface leading to the device having the same address as the • Ethernet: data is considered as Ethernet frames and frame’s destination address (see Section 4), each buffer can contain only one frame whose size 5. infrastructure interface: any frame going into the shall be less or equal than the buffer’s maximum size. input of the access interface is forwarded to the output of the direct interface leading to the mobile 5 VND VND

INPUT QUEUE INPUT QUEUE INPUT QUEUE read INTERFACE 1 FORWARDING INTERFACE 2 ENGINE OUTPUT QUEUE OUTPUT QUEUE END POINT 1 bind INTERFACE 1

OUTPUT QUEUE write

INPUT QUEUE INPUT QUEUE

hub mode INTERFACE 3 INTERFACE 4

OUTPUT QUEUE OUTPUT QUEUE Figure 7: Bind between an emulated interface and a network backend in a vnd.

Figure 5: Hub mode. tination MAC address of the frame in the forwarding table and forward the frame to the interface associated with that VND address. Currently, the forwarding table does not remove

INPUT QUEUE INPUT QUEUE entries depending on a given lifetime and thus the table must be manually cleared if needed. The vnd supports INTERFACE 1 FORWARDING INTERFACE 2 ENGINE port-based VLANs in hub and switch modes. The vnd OUTPUT QUEUE OUTPUT QUEUE does not yet implement the Spanning Tree Protocol, thus

FORWARDING it is up to the user to avoid making loops in the topology TABLE of the virtual network.

INPUT QUEUE INPUT QUEUE

switch mode 3.2. Implementation INTERFACE 3 INTERFACE 4 In the domain of virtualization, the term network back- OUTPUT QUEUE OUTPUT QUEUE end is often used to designate the software part of an emulator that enables the connection of the emulator to the other emulators either on the same physical machine or on different ones. Network backends on UNIX are usu- Figure 6: Switch mode. ally implemented with TAP interfaces, VDE [13], sockets or slirp (which provides a full TCP/IP stack implementing a node itself, and any frame going into the input of virtual NATed network). The vnd currently provides Internet and UNIX local the direct interface is forwarded to the output of sockets backends as well as TAP and VDE backends. All the access interface leading to the access point (see Section 4), these backends are implemented in an object called endpoint. To be useful, a network backend must be tied to a virtual 6. ad hoc interface: any frame going into the input network interface in a machine or a vnd. This tie is im- of any interface which is not the direct interface is plemented in the code of emulators in more or less flexible forwarded to the output of the direct interface lead- ways. In order to support the dynamic features presented ing to the mobile node itself, and any frame going at the beginning of this section, the vnd implements the tie into the input of the direct interface is forwarded in a flexible way by separating the virtual interface from to all the other output interfaces except itself (see the endpoint. This tie can be dynamically created or de- Section 4). stroyed by using the bind command as shown on Figure 7. The last four modes only make sense when the data is in- Thus the failure of a network backend connection does not terpreted as Ethernet frames as MAC addresses are needed. impact a virtual interface except for the loss of traffic. An In order to emulate the IEEE 802.11 protocols, a pseudo endpoint can also be rewired to another interface if needed header is added to any frame coming from an access point although data can be lost in the process. or emulated WIC. As NEmu currently only uses QEMU for system em- The forwarding table is filled as in a hardware switch ulation, we show on Figure 8 an example of a TCP con- having auto-learning capability. When a frame is received nection between the network backends of a QEMU virtual by an interface, the vnd checks if the source MAC address machine and a vnd. QEMU defines a local VLAN object is associated with this interface. If yes nothing is done, if to associate the virtual eth0 interface to the socket back- no, the vnd stores this association in the forwarding table. end called socket.0. The difference between the vnd bind When a frame is transmitted, the engine looks up the des- and the QEMU VLAN is that a bind creates a bijection 6 OS Table 1: Throughput of the switch emulators Network Emulation Measured eth0 vnd Component Program Throughput virtual Baseline nc to nc link 910 Mbps NIC vnd (raw connections) 682 Mbps interface QEMU vlan 1 VLine QEMU to QEMU direct link 272 Mbps vnd (link mode) 240 Mbps socket.0 endpoint VHub QEMU hub 160 Mbps vnd (hub mode) 240 Mbps TCP connection VSwitch Dynamips switch 12 Mbps vnd (switch mode) 156 Mbps

Figure 8: TCP connection between the network backends of a QEMU and a vnd. of the vnd by the user) and the value of the measured bandwidth does not exceed 2%. between an interface and an endpoint whereas a VLAN can connect several backends to the same interface thus 4. The Network Mobilizer (nemo) actually acting as a simple VLAN inside the emulator (ev- erything is broadcasted inside though). 4.1. Design NEmu can emulate mobile networks. Thus it is pos- 3.3. Performances sible to create a virtual network topology that evolves in In order to be useful and realistic, the vnd program time. In order to manage mobility, NEmu uses a special must have acceptable performances. We have carried out mobility engine called nemo. nemo [14] is a lightweight several measurements to evaluate its performances and C++ program which can generate connectivity scenarios compare them to other emulators, namely QEMU and Dy- for mobile networks. A connectivity scenario is a time namips, as they also provide socket-based network back- stamped list of wireless link connection and disconnection ends. The scenario was interconnecting two QEMU vir- events between mobile nodes. Indeed, nemo is based on tual machines via a virtual network device (line, hub and a specific use of the vnd [15] software, which can on-the- switch) emulated by either a QEMU, a Dynamips or a fly create virtual links having dynamically set character- vnd process. All network backend connections were TCP istics. nemo is able to send orders to NEmu in real time connections made on the loop-back interface of the physi- which enables to emulate the changes of connectivity be- cal machine which was an Intel Core 2 equipped desktop tween mobile nodes by creating, destroying or changing PC. An FTP session was established between the two vir- the characteristics of the links at the appropriate time. tual machines and a 1GB file was transferred and timed to nemo works behind the scene and is entirely controlled compute the bandwidth. Table 1 shows the throughput of by NEmu which acts as the user interface. nemo is im- the various possible emulations of virtual network devices. plemented in C++, contains around 3000 lines of code, We can see that our vnd program performs at least as well and is using some Boost [16] libraries including the pow- as the others when emulating any device. We can also see erful asynchronous input/output library called asio. It is that QEMU limits the throughput of the virtual machines a lightweight program using around 1MB in RAM and it at around 34MB/s. is portable thanks to Boost (on the majority of UNIX and To measure the maximum throughput of the vnd itself, Windows variants). nemo is composed of two parts : one we have used two netcat processes interconnected by a vnd part based on a simulated time scheduler and another part in raw mode. The throughput amounts to 682 Mbps to be based on a real time scheduler. The source code is avail- compared with the 910 Mbps obtained by a direct connec- able at [14]. tion between the two nc. Thus the vnd only achieves 75% of the throughput achieved by the direct connection. This 4.2. Simulated Time Scheduler loss is due to the pipelining of the two TCP connections The simulated time scheduler is the heart of the simu- as well as the queueing and the processing time inside the lation part of nemo. It can generate connectivity scenarios vnd. for the real time scheduler. Three steps are necessary to Finally we have also done measurements to evaluate generate a connectivity scenario: the accuracy of the vnd bandwidth and delay parameters. • Concerning the bandwidth, we can deduce from the above generate a map; results that the vnd can at most emulate 100 Mbps speeds. • generate a mobility scenario on this map; We have observed by varying the bandwidth parameter from 100 kbps to 100 Mbps that the difference between • generate a connectivity scenario from the mobility the value of the bandwidth parameter (set on the interface scenario. 7 load load load

P (t), V (t), A (t) generate generate 1 1 1 generate

P2(t), V2(t), A2(t) bps13(t), d13(t)

mapping tool mobility modeling tool nRT scheduler bps23(t), d23(t)

P3(t), V3(t), A3(t) bps14(t), d14(t) bps34(t), d34(t)

P4(t), V4(t), A4(t)

save save save

STEP 1: MAP DEFINITION STEP 2: MOBILITY SCENARIO STEP 3: CONNECTIVITY SCENARIO

Figure 9: Generation of connectivity scenarios with nemo.

At each step, the results of the step can be saved on 4.3. Real Time Scheduler disk in order to be loaded at a later time to avoid recom- The real time scheduler is the heart of the emulation putation. The simulated time scheduler runs the mobility part of nemo. It executes the connectivity events at their scenario and at each time interval (set by the user), it com- exact time stamp, set with respect to the start of the sce- putes the distances and the possible wireless connections nario. The temporal precision used in the real time sched- between all the pairs of mobile nodes. The steps are il- uler is equal to the one set during the processing of the lustrated on Figure 9. Being able to generate connectivity mobility scenario by the simulated time scheduler. The scenarios is an advantage over using a network simulator interaction of the real-time scheduler with NEmu is illus- interconnecting real applications with taps, because the trated on Figure 10. It shows that NEmu plays the role latter must compute the mobility at every run and this of a central controller for the other processes. In a virtual computation could be too heavy to enable the real-time mobile network, one vnd is used to emulate each wireless execution of the applications. Up to now, nemo generate network interface card (WIC). Thus there is one vnd per rectangular maps and purely random mobility scenarios. virtual mobile node and inside it are instanciated the real This is useful for carrying functional tests. nemo is also ca- network backend links (i.e., TCP or UDP tunnels). NEmu pable of importing ns-2 formatted mobility files produced transmits the orders of the user (e.g., start, stop, etc) to by tools such as Bonnmotion [17] providing realistic mobil- nemo. When the real time scheduler is running, NEmu ity models such as the Gauss-Markov Mobility Model [18] also recovers the connectivity events generated by nemo or the Reference Point Group Mobility Model [19]. In the and retransmit them to the various vnd corresponding to future, nemo will be able to load more elaborate 2D or 3D the WICs of the virtual mobile nodes in order to make the maps containing attenuation information. network topology evolve. The real time scheduler can be The simulated time scheduler suffers from several lim- paused and resumed at any moment by the user. itations: The real time scheduler suffers from several limitations: • it may require intensive computation of the order of • as opposed to NEmu, nemo is centralized; O(n2) (that is why it is written in C++); • it is better to execute all the sessions and nodes on • it requires the user to make a tradeoff between the a unique physical host to avoid reducing the perfor- temporal precision (the time interval between each mances; connectivity evaluation), the computation time and the number of events detected. • the sockets used to connect the vnd introduce delays and reduce the temporal precision. The latter will be at best of the order of the millisecond.

8 NEmu

NEmu RT orders commands control connections

nemo VM1 RT scheduler NEMO direct VND1 link adhoc mode wireless link direct direct link VND2 wireless VND3 link VM2 adhoc adhoc VM3 mode link mode

Figure 10: Emulation of virtual mobile networks with nemo. Figure 11: Original Mosh results obtained by mininet.

100% 5. Experimentation 90% We present the results of two experiments that we repli- 80% cated in order to show the accuracy of NEmu. The first 70% experiment was initially done with the Mininet container- 60% based emulator, the second was done with the JBotSim 50% simulator. In both cases, we were able to accurately repli- CDF 40% cate the results with NEmu. 30% 20% 5.1. Mosh Experiment Replication 10% Mosh SSH In order to validate the accuracy of experimentation 0% results obtained with NEmu, we reproduce a performance 0 0.5 1 1.5 2 2.5 benchmark of Mosh [20]. Mosh is a remote terminal appli- Keystroke Response Time (seconds) cation which is more tolerant to connectivity break than SSH by using the SSP protocol and a predictive algorithm. Figure 12: Mosh results obtained by NEmu. The experimentation consists in measuring the average keystrokes response time for Mosh and SSH. This experi- Thanks to our vnd program, we configure the network ment has been previously carried out on Mininet [5], an- properties as detailed above. Original results are presented other network emulator which is well known for its degree in Figure 11. Our results are illustrated in Figure 12. We of realism in experimental conditions [21]. can notice that both results are nearly identical. Those re- We reproduce the exact experimentation described in a sults imply that NEmu can offer a similar degree of realism Stanford network lecture [22] and which has been officially than Mininet. supported by Mosh developers. In this experimentation the client is connected to a switch through an emulated 5.2. AMiRALE Experiment Replication 3G network, and the server through an emulated Wi-Fi network. Authors consider the following experimental net- In order to validate the accuracy of experimentation work conditions: with mobile devices performed with NEmu, we replicate performance results obtained by simulation of a multi- • 3G: agents system called AMiRALE [23]. Emulation is performed with NEmu while simulations are carried out with – packet loss rate: 0.01; JBotSim [24], a Java library which enables the design of – bandwidth: 1 Mbps; low and high level communication scenarios and behaviors – delay: 450ms. of heterogeneous mobile nodes. AMiRALE is a distributed system which enables sev- • Wi-Fi: eral autonomous vehicles to perform common tasks col- laboratively. The application scenario consists in a team – packet loss rate: 0.08; of ground robots which has to collect a given number of – bandwidth: 25 Mbps; garbage in a park, each one being a target for the clean- – delay: 30ms. ing robots. We evaluate the output data rates generated by AMiRALE as a function of the number of targets to 9 100 to the system which means that it requires administrative Theoretical rights to be configured properly. Simulation Virtualization systems such as VirtualBox [31] and Hyper- 80 Emulation V [32] are also limited to x86 and x64 architecture emulation. 60 Regarding UML [33], the software is now unmaintained and only can emulate Linux operating systems. 40 OpenVZ [34] can be see as the successor of UML but also only supports Linux operating systems. Datarates (kbit/s) 20 LXC [35] is a Linux kernel system which can encap- sulate several process and a sub-file system in a virtual 0 container. This solution cannot be consider as a real vir- 180 360 720 1440 tualization system and does not enable any hardware con- Number of targets figuration. Dynamips [36] is an emulation system dedicated to Figure 13: AMiRALE results obtained by JBotSim and NEmu. CISCO systems. Therefore, it cannot emulate standard user machines. process. Each robot is specialized which means that it 6.2. Link Emulation Systems can only clean one kind of garbage. When a robot finds 6.2.1. Virtual Switches a garbage which it is not able to collect itself, it generates a new mission in order to inform other robots of the NEmu uses a program called vnd in order to emulate existence of this garbage. This strategy enables a robot customized virtual links. Nevertheless, other systems en- to clean a garbage which has been discovered by another able to inter-connect several virtual machines. robot. VDE [13] is a virtual switch which inter-connects vir- Figure 13 shows the results of our experiment as a func- tual machines through the shared memory system inside tion of the number of targets. Standard deviation values the Linux kernel. Such a system cannot be distributed are also shown on the plots. Since all missions are broad- on several physical machines. Moreover, VDE does not cast without any restriction, we can calculate the theoret- include any mechanism in order to manipulate link prop- ical data rates by multiplying the number of missions by erties like bandwidth, delay, etc. the size of an unique mission and divide the result by the Open vSwitch [37] is an open source project which en- frequency of broadcasts. This result is provided by the ables to instantiate virtual switches with a high customiza- theoretical plot. The emulation plot shows results from tion of virtual links. However, this software relies on vir- the NEmu experiments while the simulation plot shows tual network interfaces inside the Linux kernel which can results from the JBotSim simulations. This figure shows only be created by a system administrator of the physi- that theoretical, simulation and emulation results are very cal infrastructure. The accessibility defined in Section 2 similar which implies that NEmu provides coherent per- would be impossible with Open vSwitch. formance results for this mobile devices’ scenario. Vnet [38] is a distributed inter-connection system which enables to link several virtual machines which lay on different physical hosts. Even if the system is distributed, it 6. Related Work does not provide any link customization mechanism. Click [39] is a Linux kernel framework which allows to 6.1. Node Emulation Systems create software defined routers (i.e. at network layer 3). Currently, NEmu uses QEMU virtual machines as vir- Therefore, this solution is not suitable for our needs. tual network nodes. Despite the fact that a lot of solutions of host virtualization exist, we chose QEMU which 6.2.2. Link Properties Manipulation is a generic and open source machine emulator and virtu- Our program vswitch includes the link properties cus- alizer [4]. QEMU runs without any administrative rights tomization in order to configure bandwidth, delay, jitter and emulates a lot of various hardware architectures [25]. and bit error rate. Several other solutions exist in order to Therefore, instantiating a QEMU virtual machine as a vir- make this job. For instance NetEm [40], which relies on the tual node allows the user to configure freely its hardware Linux kernel tools Traffic Control [11], enables to manip- and software layers which fills perfectly with the flexibility ulate same properties. However, it requires root privileges and heterogeneity properties of a NVE. on the real infrastructure. Others systems such as VMware [26] or Xen [27, 28] Dummynet [41] enables to create some customizable have better I/O performances but can only emulate x86 virtual links between two entities. Therefore, it cannot and x64 architectures [29, 30] which compromises the legacy play the role of a multi-point device like a hub or a switch. support defined in Section 2. Moreover, Xen is too close Moreover, this solution operates directly in the kernel which is not compliant with the accessibility target. 10 A similar project called netpath [42] uses the Click li- The topology is static during runtime and the intercon- brary in order to create customize virtual links. nections between virtual machines are made in the host Another project called Trickle [43] can be used without kernel which requires special rights on the physical sys- any administrative rights in order to fix the maximum date tem. rates for a process. However, it can only fix this property VNUML [55] is a static virtual system relying on UML for the entire process. Thus our virtual switch is much and which requires administrative rights. more configurable. VNX [56] is the successor of VNUML is a compatible with other virtualization systems. However, as VNUML, 6.3. Network Virtualization Environment it requires administrative rights and does not include any Dynagen [44] is for Dynamips, the equivalent of NEmu link property mechanism. for QEMU. Dynagen manages fleet of Dynamips machines Cloonix [57] is a dynamic virtual network environment. and their inter-connections. However, the dynamicity of Nevertheless, it does not include link property mechanism. the topology is strongly limited, adding network services Mininet [5] is a Container-Based Emulator which al- is quite impossible and there is not any community aspect. lows to create a custom and dynamic topology on a unique GNS [45] is an open source software which allows to physical host. The second version of this system is highly build a virtualized network topology with Dynamips, Vir- demanding about fairness of experimentation conditions tualBox and QEMU virtual machines. However, it does [21]. not provide the possibility to build a community network CORE [58] is a graphic tool which enables to emulate and adding network services is as complicated as Dynagen. virtual mobile networks. This system relies on a frame- Finally GNS is hardly usable without any graphical inter- work called IMUNES [59] which runs over the FreeBSD face making difficult the creation of a complex network. [60] operating system. Nodes and links are managed inside Velnet [46] is a virtual environment dedicated to teach- the operating system kernel which implies strong limita- ing which uses VMware virtual machines. The complete tions in terms of flexibility. topology can only run on a single host which implies strong MobiEmu simulates mobile networks through ns3 for limitations on the size of the virtual network. the mobility and LXC containers for nodes. This tool does ModelNet [47] emulates a distributed virtual network not enable NVE possibilities. but this one remains static at runtime. Thus, the dynam- OpenNebula [61] provides a powerful solution to man- icity is not ensured with ModelNet. Further, the manage a virtual cloud on a wide physical infrastructure with agement of this system is fully centralized on an unique an important number of nodes. OpenNebula manages the physical machine which disables the community aspect. whole cloud domain from a single access point. Storage Vagrant [48] uses VirtualBox virtual machines in order medias are centralized and are accessible through the net- to emulate virtual network. The topology is hosted on a work which implies the use of a NAS or NFS. single physical machine and remains static at runtime. Fi- NET [62] is a powerful hardware-based infrastructure nally, the inter-connections are built inside the host kernel which allows to perform realistic experimentation on mo- making a flat network, i.e. a network which not relies on bile networks. However, this solution uses real network standards ways of addressing and routing. inter-connection devices (i.e. switches, etc.) in order to VINI [49] is a distributed virtual network which over- build the virtual network. hangs the PlanetLab testbed [50] which is an international PdP [63] is partial NVE implementation which focuses distributed cluster system. VINI uses UML virtual ma- itself on flexibility, isolation, high-speed data rates and low chines which strongly limits operating systems for nodes. cost. It uses OS level virtualization nodes such as OpenVZ. Moreover, connections between nodes are made with vir- Onelab2 [64] is a well known emulation tool over Plan- tual networks interfaces inside the kernel of the physical etLab. It also relies on DummyNet which strongly limits machine which makes the configuration impossible with- the flexibility of this solution. out administrative rights. IP-TNE [65] is an original solution which enables hosts Violin [51] is similar to VINI but provides some virtual and real network to interact with a virtual mobile network. routers which hosted different services like NEmu. How- It is not really an emulation solution since it provides only ever the use of UML and the need of an existing overlay the mobility properties of nodes and not the real node limits the use scope of this solution. virtualization. NetKit [52] relies also on UML and VDE switches which Research platforms such as PlanetLab [50], GENI [66] do not require any administrative rights. Such a system and FEDERICA [67] supply virtual infrastructure built on cannot be distributed. the top of slices of third parties owned hardware. This Marionnet [53] is a virtual environment dedicated to approach leads to the concept of Testbed as a Service teaching. It provides several network services and the com- (TaaS), exemplified by FanTaaStic [68]. Federations of munity aspect but relies on UML. testbeds have also been recently emerging, such as the Virconel [54] uses OpenVZ virtual machines which also EU-driven OpenLab federation [69] as well as the OneLab strongly limits operating systems for nodes. Moreover, federation [70], managed by UPMC. In all these cases, the user needs a specific account, must comply to specific us- 11 age policies and has to use the tools, services and APIs of nearly identical thus demonstrating that NEmu can be these testbeds. used for similar purposes. The advantage being that it Table 2 exhibits several properties of those previous so- can be distributed over several physical machines unlike lutions compared to NEmu. We see that NEmu can cover Mininet. This is important as NEmu uses full system all usages (test and proof, performance evaluation as well emulation and not container based emulation. We have as learn and teach). It can achieve a high realism as a compared NEmu to JBotSim with regard to the AMi- research tool by managing dynamic virtual networks and RALE experiment and the obtained results are similar, by being able to be distributed over several physical ma- thus validating its use for virtual mobile devices’ exper- chines. Furthermore, it can be easily used and deployed imentation. NEmu can therefore overcome the cost of a as no special rights are required on the physical machines. physical testbed infrastructure while enabling the evalua- The creation of a complex network is facilitate by the im- tion of real applications thanks to its low level emulation portant number of available network services and the easy of network and system components. NEmu can also emu- way to add another ones. Finally, it offers a new fea- late mobile ad hoc networks which, as far as we know, is ture called community aspect that enables several users to a unique feature among network emulators. merge their virtual networks together in order to build a As envisioned by Conti et al. [71], the Internet of the single larger network. future will be polymorphic, i.e., it will allow various specific networking environments to coexist, thanks to virtualization and federation. The extended concepts of NVE Table 2: Comparison of network virtualization tools. presented in this paper clearly embrace those two properties. Our software is a first step towards building such virtual networks. Its current use is mainly for research experimentation and teaching but its long term use could include production. The variety of supported backends could indeed lead our software to provide new network

Test and proof Evaluation Teaching Dynamic Distributed Mobility Community aspect Network services Regular privileges Configurable links NVE+ compliant services. Dynagen • ◦ • ◦ • ◦ ◦ • • ◦ ◦ GNS3 • ◦ • ◦ • ◦ ◦ • • ◦ ◦ Several next steps are already planned for our future Velnet ◦ ◦ • ◦ ◦ ◦ ◦ • • ◦ ◦ work on NEmu. They consist in the following tasks by ModelNet • • ◦ ◦ • ◦ ◦ ◦ • • ◦ order of priority: Vagrant • ◦ • ◦ ◦ ◦ ◦ • • ◦ ◦ VINI • • ◦ • • ◦ ◦ ◦ ◦ • ◦ • the integration of migration capabilities for virtual • • ◦ • • ◦ ◦ • • • ◦ Violin machines in order to enable load balancing; NetKit • ◦ • • ◦ ◦ ◦ • • ◦ ◦ Marionnet • ◦ • • ◦ ◦ • • • ◦ ◦ • Virconel • • • ◦ • ◦ ◦ • ◦ ◦ ◦ the integration of new services inside the virtual router; • ◦ • ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ VNUML • VNX • • • ◦ • ◦ ◦ ◦ ◦ ◦ ◦ the implementation of more sophisticated map gen- Cloonix • • • • ◦ ◦ ◦ ◦ • ◦ ◦ eration algorithms; Mininet • • • • ◦ ◦ ◦ ◦ • • ◦ CORE • • ◦ • ◦ • ◦ • ◦ • ◦ • the improvement of the accuracy of the vnd. MobiEmu • • • • ◦ • ◦ ◦ ◦ • ◦ NET • • ◦ • • • ◦ ◦ ◦ • ◦ PdP • • ◦ ◦ • ◦ ◦ ◦ ◦ • ◦ 8. References Onlab2 • • ◦ ◦ • ◦ ◦ ◦ ◦ • ◦ NEmu • • • • • • • • • • • [1] T. Henderson, M. Lacage, G. Riley, C. Dowell, J. Kopena, Net- work simulations with the ns-3 simulator, ACM SIGCOMM demonstration. • Yes ◦ No [2] A. Varga, et al., The omnet++ discrete event simulation system, in: Proceedings of ESM, Vol. 9, 2001. [3] B. Yamini, D. Selvi, Cloud virtualization: A potential way to reduce global warming, in: Proceedings of IEEE RSTSCC, 2010, 7. Conclusion pp. 55–57. [4] F. Bellard, QEMU, a fast and portable dynamic translator, NEmu and its associated programs vnd and nemo en- in: Proceedings of USENIX Annual Technical Conference, able the creation and management of dynamic, heteroge- FREENIX Track, 2005, pp. 41–46. [5] B. Lantz, B. Heller, N. McKeown, A network in a laptop: Rapid neous and mobile virtual networks. They provide a good prototyping for software-defined networks, in: 9th ACM SIG- compromise between ease of use, low cost and realism. COMM Workshop on Hot Topics in Networks, 2010, pp. 19:1– Such virtual networks can be distributed over several phys- 19:6. ical hosts and be controlled without any administrative [6] V. Autefage, D. Magoni, Network emulator: a network virtualization testbed for overlay experimentations, in: 17th IEEE rights. They can also evolve in real time with nemo by International Workshop on Computer-Aided Modeling Analysis following a pre-calculated connectivity scenario. and Design of Communication Links and Networks, 2012, pp. We have compared NEmu to Mininet with regard to 38–42. the Mosh experiment and have shown that the results are 12 [7] N. Chowdhury, R. Boutaba, Network virtualization: state of the [33] J. Dike, A user-mode port of the Linux kernel, in: Proceedings art and research challenges, IEEE Communications Magazine of Linux Showcase and Conference, Vol. 2, 2000. 47 (7) (2009) 20–26. [34] Parallels, OpenVZ Linux Containers, http://wiki.openvz.org [8] KVM, Virtio, http://www.linux-kvm.org/page/Virtio. (2006). [9] NBD, Network Block Device, http://nbd.sourceforge.net. [35] D. Lezcano, LXC, http://lxc.sourceforge.net (2008). [10] Graphviz, Graph Visualization Software, http://www. [36] C. Fillot, Dynamips, https://github.com/GNS3/dynamips graphviz.org (1988). (2007). [11] B. Hubert, G. Maxwell, R. Van Mook, M. Van Oosterhout, [37] B. Pfaff, J. Pettit, T. Koponen, K. Amidon, M. Casado, P. Schroeder, J. Spaans, Linux advanced routing & traffic con- S. Shenker, Extending networking into the virtualization layer, trol, in: Ottawa Linux Symposium, 2003, pp. 213–222. Proceedings of ACM HotNets workshop. [12] R. Shingledecker, TinyCore Linux, http://tinycorelinux.net [38] A. I. Sundararaj, A. Gupta, P. A. Dinda, Dynamic topology (2008). adaptation of virtual networks of virtual machines, in: Pro- [13] R. Davoli, Vde: Virtual distributed ethernet, in: Proc. of the ceedings of the 7th LCR Workshop, 2004, pp. 1–8. First International Conference on Testbeds and Research Infras- [39] E. Kohler, R. Morris, B. Chen, J. Jannotti, M. F. Kaashoek, tructures for the DEvelopment of NeTworks and COMmunities, The click modular router, ACM Transactions on Computer Sys- 2005, pp. 213–220, http://vde.sourceforge.net. tems 18 (3) (2000) 263–297. [14] D. Magoni, Network Mobilizer, http://www.labri.fr/perso/ [40] S. Hemminger, et al., Network emulation with netem, in: Linux magoni/nemo (2012). Conf Au, 2005, pp. 18–23. [15] D. Magoni, Virtual Network Device, http://www.labri.fr/ [41] M. Carbone, L. Rizzo, Dummynet revisited, SIGCOMM Com- perso/magoni/vnd (2012). puter Communications Review 40 (2) (2010) 12–20. [16] B. Dawes, al., Boost C++ Libraries, http://www.boost.org. [42] S. Agarwal, J. Sommers, P. Barford, Scalable network path em- [17] N. Aschenbruck, E. Gerhards-Padilla, M. Gerharz, M. Frank, ulation, in: Proceedings of the 13th IEEE International Sympo- P. Martini, Modelling mobility in disaster area scenarios, in: sium on Modeling, Analysis, and Simulation of Computer and 10th ACM/IEEE International Symposium on Modeling, Anal- Telecommunication Systems, MASCOTS ’05, IEEE, 2005, pp. ysis and Simulation of Wireless and Mobile Systems, 2007, pp. 219–228. 4–12. [43] M. Eriksen, Trickle: A userland bandwidth shaper for unix-like [18] B. Liang, Z. J. Haas, Predictive distance-based mobility man- systems, in: Proceedings of USENIX Annual Technical Confer- agement for pcs networks, in: Proceedings of IEEE INFOCOM, ence, FREENIX Track, 2005, p. 43. 1999, pp. 1377–1384. [44] G. Anuzelli, Dynagen, http://dynagen.org (2006). [19] X. Hong, M. Gerla, G. Pei, C.-C. Chiang, A group mobility [45] GNS3, Graphical Network Simulator, http://www.gns3.net model for ad hoc wireless networks, in: Proceedings of the 2nd (2007). ACM MSWiM, 1999, pp. 53–60. [46] B. Kneale, A. Y. De Horta, I. Box, Velnet: virtual environment [20] K. Winstein, H. Balakrishnan, Mosh: An interactive remote for learning networking, in: Proceedings of the 6th Australasian shell for mobile clients, in: USENIX Annual Technical Confer- Conference on Computing Education, Vol. 30, 2004, pp. 161– ence, 2012, pp. 177–182. 168. [21] N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, N. McKeown, [47] A. Vahdat, K. Yocum, K. Walsh, P. Mahadevan, D. Kosti, Reproducible network experiments using container-based emu- J. Chase, D. Becker, Scalability and accuracy in a large-scale lation, in: 8th International Conference on Emerging Network- network emulator, in: The 5th ACM SIGOPS Operating Sys- ing Experiments and Technologies, 2012, pp. 253–264. tems Review, 2002, pp. 271–284. [22] A. Aljunied, Evaluation of Mosh performance results, http:// [48] M. Hashimoto, J. Bender, Vagrant, http://vagrantup.com reproducingnetworkresearch.wordpress.com/2013/03/13/ (2010). cs244-2013-evaluation-of-mosh-mobile-shell-performance-results[49]. A. Bavier, N. Feamster, M. Huang, L. Peterson, J. Rexford, In [23] V. Autefage, S. Chaumette, D. Magoni, A mission-oriented vini veritas: realistic and controlled network experimentation, service discovery mechanism for highly dynamic autonomous in: Proceedings of ACM SIGCOMM, 2006, pp. 3–14. swarms of unmanned systems, in: Autonomic Computing [50] A. Bavier, M. Bowman, B. Chun, D. Culler, S. Karlin, S. Muir, (ICAC), 2015 IEEE International Conference on, 2015, pp. 31– L. Peterson, T. Roscoe, T. Spalink, M. Wawrzoniak, Operating 40. system support for planetary-scale network services, in: Pro- [24] A. Casteigts, Jbotsim: a tool for fast prototyping of distributed ceedings of the 1st USENIX NSDI, 2004, pp. 253–266. algorithms in dynamic networks, in: Proceedings of the 8th [51] X. Jiang, D. Xu, Violin: Virtual internetworking on overlay International Conference on Simulation Tools and Techniques, infrastructure, in: Proceedings of the 2nd Springer ISPA, 2003, 2015. pp. 937–946. [25] A. Ribiere, Emulation of obsolete hardware in open source vir- [52] M. Pizzonia, M. Rimondini, Netkit: easy emulation of complex tualization software, in: Proceedings of the 8th IEEE INDIN, networks on inexpensive hardware, in: Proceedings of the 4th 2010, pp. 354–360. IEEE TridentCom, 2008, pp. 1–10. [26] EMC, VMware, http://www.vmware.com (2004). [53] J.-V. Lodo, L. Saiu, Marionnet, http://www.marionnet.org [27] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, (2007). R. Neugebauer, I. Pratt, A. Warfield, Xen and the art of vir- [54] Y. Benchaib, A. Hecker, Virconel: A network virtualizer, in: tualization, in: Proceedings of the 19th ACM SOPS, 2003, pp. Proceedings of the 19th IEEE MASCOTS, 2011, pp. 429–432. 164–177. [55] DIT, VNUML, http://dit.upm.es/vnumlwiki (2003). [28] M. Bourguiba, K. Haddadou, G. Pujolle, Packet aggregation [56] DIT, VNX, http://dit.upm.es/vnxwiki (2008). based network i/o virtualization for cloud computing, Computer [57] V. Perrier, Cloonix, http://clownix.net (2007). Communications 35 (3) (2012) 309–319. [58] J. Ahrenholz, C. Danilov, T. Henderson, J. Kim, Core: A real- [29] J. Che, Y. Yu, C. Shi, W. Lin, A synthetical performance evalu- time network emulator, in: Proceedings of IEEE MILCOM, ation of openvz, xen and kvm, in: Proceedings of IEEE APSCC, 2008, pp. 1–7. 2010, pp. 587–594. [59] Z. Puljiz, M. Mikuc, IMUNES, http://imunes.tel.fer.hr [30] P. Domingues, F. Araujo, L. Silva, Evaluating the performance (2003). and intrusiveness of virtual machines for desktop grid comput- [60] FreeBSD, FreeBSD, http://www.freebsd.org (1993). ing, in: Proceedings of IEEE IPDPS, 2009, pp. 1–8. [61] C. Labs, OpenNebula, http://www.opennebula.org. [31] Oracle, VirtualBox, https://www.virtualbox.org (2007). [62] S. Maier, D. Herrscher, K. Rothermel, Experiences with node [32] Microsoft, Hyper-V, www.microsoft.com/hyper-v-server virtualization for scalable network emulation, Computer Com- (2008). munications 30 (5) (2007) 943–956.

13 [63] Y. Liao, D. Yin, L. Gao, Network virtualization substrate with parallelized data plane, Computer Communications 34 (13) (2011) 1549–1558. [64] M. Carbone, L. Rizzo, An emulation tool for planetlab, Com- puter Communications 34 (16) (2011) 1980–1990. [65] R. Simmonds, B. W. Unger, Towards scalable network emulation, Computer Communications 26 (3) (2003) 264–277. [66] N. Van Vorst, M. Erazo, J. Liu, Primogeni: Integrating real- time network simulation and emulation in geni, in: Proceedings of IEEE PADS, 2011, pp. 1–9. [67] C. GARR, FEDERICA, http://www.fp7-federica.eu (2008). [68] A. Willner, S. Albrecht, S. Covaci, F. Schreiner, T. Magedanz, S. Avessta, C. Scognamiglio, S. Fdida, U. Bub, Fantaastic: Sustainable management of future internet testbed federations, in: Network Operations and Management Symposium (NOMS), 2014 IEEE, 2014, pp. 1–4. doi:10.1109/NOMS.2014.6838379. [69] S. Fdida, T. Korakis, H. Niavis, S. Salsano, G. Siracusano, The express sdn experiment in the openlab large scale shared experimental facility, in: Science and Technology Conference (Mod- ern Networking Technologies) (MoNeTeC), 2014 International, 2014, pp. 1–7. doi:10.1109/MoNeTeC.2014.6995584. [70] L. Baron, C. Scognamiglio, M. Rahman, R. Klacza, D. Ci- calese, N. Kurose, T. Friedman, S. Fdida, Onelab: Major computer networking testbeds open to the ieee infocom community, in: Computer Communications Workshops (INFO- COM WKSHPS), 2015 IEEE Conference on, 2015, pp. 3–4. doi:10.1109/INFCOMW.2015.7179314. [71] M. Conti, S. Chong, S. Fdida, W. Jia, H. Karl, Y.-D. Lin, P. Mhnen, M. Maier, R. Molva, S. Uhlig, M. Zuk- erman, Research challenges towards the future internet, Computer Communications 34 (18) (2011) 2115 – 2134. doi:http://dx.doi.org/10.1016/j.comcom.2011.09.001.