Linköping University | Department of and Information Science Bachelor thesis, 16 ECTS | Computer and Information Science 2018 | LIU-IDA/LITH-EX-G--18/044--SE

Inter- Communication in a Virtualized Environment

Inter Process Kommunikation inom en virtualiserad Miljö

Filip Johansson Christoffer Lindstrom

Supervisor : Erik Berglund Examiner : Anders Fröberg

External supervisor : Krister Gindahl

Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt

Detta dokument hålls tillgängligt på – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin- istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam- manhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.

Copyright

The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circum- stances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the con- sent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping Uni- versity Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.

c Filip Johansson Christoffer Lindstrom Thesis: Inter-Process Communication in a Virtualized Environment Filip Johansson Christoffer Lindstrom Linkoping University Linkoping University Linkoping, Sweden Linkoping, Sweden fi[email protected] [email protected]

ABSTRACT components will suffer more of the load than others, which Selecting the correct inter-process communication method is will lead to certain components reaching their limit before an important aspect of ensuring effective inter-vm and inter- the remaining components. Currently, the solution is to add container process communication. We will conduct a study of a new node containing all the same components even though IPC methods which might be useful and fits the Qemu/KVM most of them are unnecessary at the time. and container environments, and se- To remedy this, Ericsson has decided to move certain compo- lect those that fit our criteria. After implementing our chosen nents to virtual environments and at the same time redesign methods we will benchmark them in a test suite to find the the components so they can be individually and dynamically ones with highest performance in terms of speed. Our results started or stopped, depending on the current load. While this show that, at the most common message sizes, Domain will allow for easy scaling, it may also cause issues with the Sockets work best for containers and Transparent Inter Pro- communication between the components, mainly decreased cess Communication has the best performance between vir- data transfer rate and higher instability. tual machines out of the chosen methods. In order to maintain effective communication between the vir- INTRODUCTION tual components, we utilize a technique called Inter-Process A decade ago the most you did on your mobile-phone was to Communication. make calls and send text messages to your friends. Today, the 4G network is used for everything from making calls and tex- Purpose ting to streaming videos and playing multiplayer games. As The primary goal of this study is to provide a good choice a result, the amount of data a single user needs has increased of Inter-Process Communication for Ericsson, that can be uti- by a huge amount[1]. lized for effective inter-vm and inter-container communica- tion on a host. Smart-phones, tablets and laptops all contribute to an in- creased pressure on the 4G network to handle all the data[1]. Research Approach As figure1 shows, video streaming is the largest consumer of The aim of this thesis is to evaluate different Inter-Process bandwidth by a large margin. All the while, more and more communication (hereafter called IPC) methods. Our research devices are developed with internet connectivity which places question is: a higher burden on a network that is not always designed for all of these devices. Which among our chosen IPC methods is the most effective, in regards to message rate and latency, when compared to Figure 1. Mobile traffic forecast from Ericsson Mobility Report 2017 TCP across: • Docker containers • Qemu/KVM virtual machines We will accomplish this by first conducting a study of what kind of IPC methods could be useful. Ericsson has provided a messaging framework called XCM with certain specifications that our methods must follow. The selected methods will be reduced down to the three best, where at least one method will be usable by a container and another by a virtual machine. We will then implement the methods in XCM and benchmark them in a test suite.

Limitations Ericsson requested that we only look at IPC methods that Motivation suits Docker containers and virtual machines virtualized by A 4G node consists of a number of different components to Qemu/KVM. handle the connections. Depending on the traffic type some

1 THEORY Figure 2. Virtual Machine vs. Container Structure In this section we will present all the necessary information that you will need in order to understand the work we have done.

Virtualization technology has been around for a long time and it means to simulate a virtual environment with a certain hard- ware and on your machine[2]. It could be that you want to use a specific application that does not run on your and therefore you need a virtual machine or container to run it.

Docker A popular method for isolating processes or setting up envi- ronments are containers, which is quite commonly used as an alternative to virtual machines. There are different versions of container tools but Dockers is the one that is most well known. whereas the Docker Engine is the process which builds and runs Docker images. Docker1 is a popular, open-source tool for automating the de- ployment of, mainly Linux, applications inside a container. There can be many advantages to using a virtual machine By utilizing namespaces, Docker is able to hide the rest of compared to running everything on a host machine, like bet- the operating system from the running application, effectively ter security and more efficient use of hardware resources[6]. isolating the container from the host and other containers. However, processes on different virtual machines that do not With the help of the control groups API provided by Docker exist on the same host will have a harder time communicating we can also limit the number of resources used by a con- with each other since they need to go over the network which tainer[3]. will incur overhead. Communication between processes on the same host on the other hand can also use the methods that Qemu/KVM induce overhead but it is inefficient[7]. This is why choos- To virtualize our environment we use Qemu2, a machine em- ing the correct inter-process communication method for your ulator and KVM3, a virtualization infrastructure. Qemu’s pri- system is important. mary usage is to run a different operating system on top of another operating system such as Linux on Windows or to Inter-Process Communication debug the system since it is easy to stop and inspect the vir- IPC is the collective name for methods which processes use tual machine as well as save and restore its state[4]. KVM to manage their shared data. Data is either communicated by itself does not provide a virtual machine but instead exposes using memory sharing, where processes have a shared mem- a KVM device node which can be used to create a virtual ory space they can utilize, or . In message machine, allocate memory to it and run virtual CPUs[5]. passing there are two important concepts, blocking and non- Together they can be used to emulate virtual machines on a blocking message. A blocking message is where the process Linux operating system with a speed close to a physical ma- that sends the message will be blocked until it receives a re- chine which of course is the ideal. We want the separation and sponse and non-blocking is where the sender doesn’t need to efficient hardware utilization of a virtual machine but with the wait for a response to be able to continue to work[8]. same operating speeds as a normal host machine. The IPC methods4 vary greatly depending on the system re- Virtual Machines and Containers quirements. For example, cannot be used Like figure2 shows, while a virtual machine effectively to communicate between processes across two different hosts makes slices of the hardware to run a new operating system but is usually considered the fastest for local communication. on, a container instead virtualize the operating system itself. Sockets on the other hand can be used for both local and re- 5 This results in much more efficient resource utilization as the mote communication depending on what socket domain is container is essentially just using a protected portion of an chosen. already running host operating system. Virtual Machines has The Unix Domain AF UNIX is used for fast local communi- on the other hand been around for a lot longer and is of course cation between processes while the internet domain AF INET safer to use and better supported[3]. allows for easy process communication across but The is essentially the software that allocates the induces extra overhead which reduces their communication physical hosts resources to the virtual machines as required speed[8]. There are many socket domains but the UNIX and internet domains are the only ones normally used. 1https://docs.docker.com/ 2https://www.qemu.org/ 4https://en.wikipedia.org/wiki/Inter-process communication 3https://www.linux-kvm.org/page/Main Page 5https://users.cs.cf.ac.uk/Dave.Marshall/C/node28.html

2 XCM Framework Related Works We are to implement our IPC:s in Ericsson’s chosen IPC Already early on it was noticed that optimizing IPC was a bit framework XCM, Extensible Connection-Oriented Messag- of a challenge. One of the problems was that the kernels were ing. The framework has specifications that all implemented not handling IPC very well, causing extra latency[9]. Liedtke IPC methods must follow, these are: managed to show that there were still several ways to opti- mize kernels for IPC. This study continued and years later • Connection-Oriented , a connection is established before it was shown that even while optimizing the algorithms for any useful data can be sent. IPC methods the major bottleneck was still the kernel[10]. • Message Service , messages are passed between different However, it still did not mean that optimizing the methods processes. was a futile effort and that choosing the right IPC method for • Client-Server model , during communication there is a the right situation could still have a big impact on the perfor- server and a client. mance[8]. • Reliable, lost data will be re-transmitted. • In-order Delivery, the data is received in the same order it Three popular IPC methods are Shared Memory, Pipes and was sent. Sockets. All three can be used for communication between • Flow Control, the sender will not transmit more data than processes with varying results, but Internet Sockets, a varia- the receiver can handle. tion of Socket, can also be used for communication between processes on different nodes like a virtual machine. The XCM transport types works for the most part by setting up a server socket that a client can then connect to. Once a A study by Zhang Xiurong in 2011[11] evaluated the commu- connection has been established both the server and the client nication performance between the three previously mentioned can send messages to each other. methods. He came to the conclusion that while Shared Mem- ory was superior with low filesizes, Pipes and Sockets began Ericsson has already implemented a small number of IPC to have better latency as the filesize increased. methods which we will use to compare with our own imple- mentations. Once a method is implemented into the XCM As Shared Memory tend to be the focus of studies on local framework it is called a transport type. The ones we are in- IPC[12], there have been a lot of proposed methods that used terested in are: Shared Memory in one way or another. Macdonell proposed a method called Nahanni which use a shared memory buffer UX Transport which guests can access through a device developed for the The Unix Transport utilizes the , and method[13] which is supported by KVM. provides a connection-oriented service, with message bound- ary preservation and in-order delivery. Since all operations Another method named MemPipe was proposed by Zhang et happen within the kernel it avoids a lot of the overhead of the al which is similar to Nahanni with the difference that the Internet Socket and therefore it is used for communication size of the shared memory buffer is dynamic, which means within the same operating system. The transport uses the ab- the size increased the more memory is needed[14]. stract option which means the socket is not mapped to the file Socket outsourcing was a method which at first glance system and the socket will cease to exist once all connections seemed to be about sockets but it actually intercepts socket to it are closed. traffic to pass the message with a shared memory buffer in- TCP Transport stead[15]. The Transmission Control Protocol Transport utilizes the In- These were only a few methods built around Qemu/KVM vir- ternet Socket and provides a reliable byte stream with in- tual machines. There are a lot of other methods designed for order delivery, as well as good flow control. XCM adds a other like [12] which isn’t usable for us with header frame in order to keep track of the length of the mes- our environment but can give insight and inspiration for our sage sizes to be able to work with the data as messages rather work. than a stream of data. The most common method used for inter-vm process commu- Benchmark nication on the same host is TCP. The reasoning being that if The XCM framework has a benchmark test implemented, two computers connect to each others via TCP so should vir- called xcmpong, that measures the message rate and latency tual machines even if there exists more efficient IPC methods between different processes with options for how many mes- that could be used. TCP in this scenario adds a large amount sages are sent, the size of the messages and how many mes- of overhead and message cloning as it travels through the net- sages are sent at the same time. work and kernel stacks leading to a loss of performance[14].

Message Rate Our Chosen IPC Methods The message rate is the total amount of message that can be We have chosen three IPC methods to compare with the TCP delivered per second. Transport. Latency The latency is how much time it takes in microseconds to send one message.

3 VirtioVsock types. We will look at articles that examine IPC methods, VirtioVsock6 is a host-guest IPC for virtual machines. We check how well the IPC method fits into the XCM framework chose VirtioVsock due to it being similar to TCP with almost and find articles that compare the speed of different methods the same semantics which makes it easy to adapt to XCM. to determine which IPCs could be valuable to us. VirtioVsock avoids the TCP overhead which will hopefully give the method an advantage over TCP when communicating Implementation between virtual machines. In order to use the IPC the virtual In order to evaluate our IPC methods we first need to im- machines needs to have a vhost-vsock device attached with a plement them. XCM provides a framework for us to code a context id to identify the virtual machine. functional version of our three chosen methods that we can benchmark. Since XCM is written in the programming lan- VirtioVsock is only guest-host communication but if you link guage C, most of our methods will also be implemented in guest to guest via the host you avoid the network stack po- C although some of them will require some additional work tentially speeding up the process[16]. If its performance is outside of XCM. within an acceptable range of the other methods the simplic- ity might make it the best method. VVS Transport VirtioVsock uses the POSIX socket API and supports both Unix Transport between Docker containers the STREAM and DGRAM socket options which makes it The Unix Domain Socket is already implemented in XCM very trivial to map it to the XCM framework. We will choose but it has not been tested between Docker containers. While the stream option as it follows the XCM semantics the most. containers exist on the same host, they still create their own Like the TCP Transport that we use as a base for our imple- network namespace. Effectively, this separates their network mentation, we need to implement a frame header to keep track from the rest of the host and as a consequence the unix socket of the message boundary. will not work for communicating between two containers on the same host since they cannot detect each others sockets. To use the VVS Transport we need to start Qemu/KVM vir- However, it is possible to configure containers to be able to tual machine with a vhost-vsock device with a context id that share unix sockets. is higher than three since the numbers below that are reserved. Finally, since it is a host-guest communication method, we By mounting a shared volume on both containers they have will need to implement a message forwarding mechanism in a common place to create sockets and communicate between the host to pass along all the messages to their respective des- each other. Another way, which uses abstract unix sockets, is tinations. to make sure that the two containers share the same network namespace. This ensures that their unix sockets can still find Unix Transport between Docker containers each other and communicate. As previously mentioned the Unix Domain Socket is already implemented. However, communication between two con- Transparent Inter Process Communication 7 tainers via the Unix Transport will not work without the TIPC is an IPC that is meant to be utilized for communica- proper setup. We will configure the Docker containers to al- tion between cluster nodes[17] which is perfect for us since low the usage of abstract unix domain sockets by making sure we have virtual machines or containers trying to communi- that the created containers share the same network names- cate with each other. By setting up a cluster of TIPC nodes, pace. we can potentially achieve message rates that are close in per- formance to Unix Domain Sockets. TIPC Transport TIPC can provide, just like a unix domain socket, a Transparent Inter Process Communication utilizes the POSIX socket API and supports both STREAM, DGRAM and connection-oriented service with message boundary and in- 8 order delivery through the SEQ PACKET socket option [17]. SEQ PACKET socket options . The DGRAM option is unre- Thanks to this, it should be trivial to map the TIPC Transport liable for sending messages and the STREAM option needs a to the existing XCM framework. TIPC can be run on both frame header which would require extra effort to handle. We virtual machines and containers and since it avoids the TCP chose the SEQ PACKET option due to the fact that it fits per- network stack by transmitting directly across the Ethernet in- fectly into the XCM framework. By basing our new transport terface, we should see an increased performance over TCP in on the TCP Transport, we can easily implement TIPC by re- both systems. moving the unnecessary frame header and make sure that the address can be parsed in the XCM framework. METHOD To make sure that we can use the TIPC Transport we need In this section we present how we will accomplish the work to set our virtual machines or containers as TIPC nodes and we have set out to do and why we chose to do it this way. expose a network interface as a TIPC bearer. This ensures that they can find each other on the cluster network when they try IPC Study to communicate. Choosing a correct IPC to implement in XCM is critical to gain an improvement over the already existing transport 6https://wiki.qemu.org/Features/VirtioVsock 7http://tipc.sourceforge.net/ 8http://man7.org/linux/man-pages/man2/socket.2.html

4 Testing VVS Transport very easy. While it is only able to commu- Once our chosen methods are implemented, we will move on nicate from virtual machine to host, a forwarding mechanism to testing them using our benchmark provided by Ericsson. on the host side should make this method work VM to VM Since TCP is the current inter-vm transport type we will in- and potentially still be faster than the TCP Transport. clude it in our testing as the XCM transport type reference. The Unix Transport was chosen for Docker container com- Our tests will be used to record the latency and message rate munication since it was already implemented and proven to of our chosen IPCs which we will then use as data to deter- be significantly faster than the TCP Transport. The only prob- mine what the most effective IPC is in terms of speed. We lem would be to ensure that the two containers can reach each will also see how our IPC methods can handle larger mes- others Unix Domain Sockets, but this turned out to be a minor sage sizes and if there is a noticeable performance variation issue that could be easily solved. for certain methods at different sizes. Transparent Inter Process Communication is a cluster node We will set up and configure virtual environments on our own IPC that claims to have comparable speeds to unix domain computer to test our different IPC methods. To ensure that sockets between its nodes. This, along with the fact it can use our testing is as fair as it can be, we will make sure that our the SEQ PACKET socket option and is semantically similar virtual environments utilize the same host CPU core.The host to the TCP Transport and therefore easy to implement is the CPU core will also be isolated so it wont have any processes primary reason for our choice. running on it other than the ones we choose. Discarded methods The Qemu/KVM virtual machines we used has the following During the IPC study we found several IPC methods that had specifications: some potential. We want to list why we did not use certain • Qemu/KVM emulator version 2.11.1 methods for clarification. • Intel Core i7-4610m, 1 core 3GHz. All the Xen methods were directly off limits due to the fact • 4096 MB of RAM. that we use the KVM hypervisor to simulate our virtual ma- • 16.04.4. chines and the Xen IPC methods relies on the Xen hypervisor. • Linux kernel 4.13 These methods include: The Docker containers we used has the following specifica- • XenSocket tions: • XenLoop • Docker version 1.13.1 • XenVMC • Intel Core i7-4610m, 1 core 3GHz. • XWay • 4096 MB of RAM. Socket-outsourcing seemed like it fulfilled our requirements • Ubuntu 16.04.4. for the methods as it was sockets semantics but the modifying • Linux kernel 4.13 of the Linux code was deemed too complex for the time we For all our methods we are going to test them using a vari- had. able message size ranging from 100 bytes to 60KB. While the Nahanni and MemPipe did not match our requirements ex- most common messages vary from 100 to 1000 bytes, higher actly but we believe we could work around it. We believed messages sizes could be interesting for future studies. The MemPipe would be the better option of the two but without message rate and latency will be tested using 100000 mes- the code base for MemPipe it would take too long to imple- sages that are sent 1 at a time. The methods to be tested are: ment. We tried to implement Nahanni but with the consul- 1. VVS Transport between a Qemu/kvm virtual machine pro- tation of the XCM manager we believed our solution would cess and host process. have too many problems to be worth evaluating. 2. UX Transport between two Docker container processes on Linx was a method we were trying to implement as a fourth the same host. and final method but failed. The method was developed by 3. TIPC Transport between two Docker container processes a team that was part of developing TIPC and according to on the same host. their statement Linx has a higher performance than TIPC. The 4. TIPC Transport between two Qemu/kvm virtual machine problem was that Linx was made open source in 2006 and has processes on the same host. not seen much progress since. We tried to modernize the code to be able to run it but in the end we ran out of time and had RESULT to drop it. In this section we will present all the factual results of our completed work. Implementation In this section we will present the result of implementing our IPC Study chosen IPC methods. VirtioVsock is a host-guest communication IPC that was cho- sen because it eliminates the need to traverse the TCP layer VirtioVsock which reduces a lot of overhead. It also has very similar se- The implementation of VirtioVsock into XCM went smoothly mantics to the TCP Transport which makes implementing the for the most part. Since it is very similar to the TCP Transport

5 we could look at that as a guide for how our VVS Transport Figure 3. Docker Message Rate should function and after a few trial and errors we had a func- tioning implementation of VirtioVsock along with a frame header to preserve the message boundary. The real problem came when we set up our Qemu/KVM vir- tual machines and found that we needed to attach a device called ”vhost-vsock-pci” in order to use the VVS Transport between the virtual machine and the host. It took a while to find the correct way to start the virtual machine with the cor- rect device and a context id that is bound to it. Finally, we did not succeed in forwarding messages from one guest to another over the host and after a discussion with the XCM manager we decided to drop it entirely and do a host to guest comparison between VirtioVsock and TCP instead. Unix Transport for Docker containers At the higher message sizes, both the Unix and TIPC Trans- To implement the Unix Transport for the Docker containers ports keeps their advantage over TCP while slowly decreasing we had to make sure that the containers were on the same the amount of messages sent until TIPC overtakes the Unix network namespace. Docker allows the host to choose which Transport after the message size has risen to 16KB. network namespace the container should be in through the use of the command option ”–net=container:id”, where id is Figure 4. Qemu/KVM to host Message Rate the name of a previously created container. We were also able to set which CPU core the containers would use through the command option ”–cpuset cpus=’core’” to ensure a fair test environment. Transparent Inter Process Communication The implementation of TIPC into XCM, as with VirtioVsock, went well except for one problem that we have not been able to find the answer to. While similar to TCP, the TIPC socket will sometimes not finish connecting to a TIPC server socket. After spending quite some time researching and having dis- cussions with the manager of XCM we decided to drop the search for a perfect solution. We have instead solved this by increasing the time the con- nection socket has to make the connection to a server socket which works for our tests but would make performance suf- Figure4 displays a message rate comparison between a fer in a real environment since a socket would need to handle Qemu/KVM virtual machine process and a host process using more than one connection at once. XCM transport types VVS and TCP to communicate 100000 messages at varying sizes. To ensure that our virtual machines and containers can use TIPC we had to configure them to be TIPC nodes. This was As can be seen in the graph, the VVS transport has a message done by giving each node (a vm or container) a TIPC address rate of about 80000 messages per second compared to the ”zone.cluster.node” and making sure that TIPC traffic can tra- TCP transport’s 28000 messages per second. This continues verse over the node’s network interface. until the VVS transport type reaches a message size of 2KB where it begins a sharp decline of messages sent until it only TIPC works for inter-container and inter-vm IPC as well as barely surpasses TCP at 60KB. communication between virtual machines and containers. Figure5 displays a message rate comparison between two Result: Transport Message Rate Qemu/KVM virtual machine processes using XCM transport In this section we present the result of all our transport types types TIPC and TCP to communicate 100000 messages at message rate, which in our case is the total amount of mes- varying sizes. sages the chosen transport type can send per second. The TIPC transport type has a 1500 messages per second lead Figure3 displays a message rate comparison between two over the TCP transport for the message sizes of 100 to 1000 Docker container processes using XCM transport types TCP, bytes. After this, the TIPC transport begins a sharp decline UX and TIPC to communicate 100000 messages at varying of messages sent per second until the TCP transport type sur- sizes. As can be seen in the graph, the Unix and TIPC Trans- passes it at 4KB. ports has a much higher message rate than the TCP Transport at all tested message sizes.

6 Figure 5. Qemu/KVM Message Rate Figure 7. Qemu/KVM to host Latency

Figure 8. Qemu/KVM Latency Result: Transport Latency In this section we present the result of all our transports la- tency which is in our case the total time in microseconds it takes to send one message.

Figure 6. Docker Latency

As can be seen in the graph, the TIPC transport has an advan- tage of about 9 µs over TCP until its latency begins to sharply increase at 4KB. TCP on the other hand has a more gradual increase the higher the message size goes.

DISCUSSION Figure6 displays an average latency comparison between two In this section we will discuss what our results mean and how Docker container processes using XCM transport types UX, the method we used to achieve our results could be improved. TCP and TIPC to communicate 100000 messages at varying sizes. Result Docker As can be seen in the graph, the Unix and TIPC transport When it comes to Docker containers, the Unix transport type types always has around half as low latency as the TCP trans- has a huge advantage against the TCP transport type as can port. The Unix and TIPC transports remains on an even la- be seen in figure9. It has too much overhead to be able to tency until the message size reaches 16KB, where TIPC be- come close to the Unix transports message rate and even with gins to have a slightly lower latency than Unix. smaller message sizes it has only half the amount of messages Figure7 displays an average latency comparison between a sent per second that the Unix transport type has. Qemu/KVM virtual machine process and a host process using The TIPC transport type has a comparable message rate and XCM transport types VVS and TCP to communicate 100000 latency to the Unix transport and even better at messages sizes messages at varying sizes. that are greater than 16KB but if we are able to use the Unix As can be seen in the graph, up to a message size of 60KB Transport, we should use it since it requires no setup at all the VVS Transport always has a lower latency than TCP. except for configuring the XCM framework with the Unix Domain Socket. Figure8 displays an average latency comparison between two Qemu/KVM virtual machine processes using XCM transport Qemu/KVM types TIPC and TCP to communicate 100000 messages at Since we did not achieve a satisfactory way of forwarding varying sizes. messages over the host from one guest to another we feel that

7 Figure 9. Docker Message Rate about the VVS transports speed. A similar argument can be made for latency. Perhaps someday in the future when VirtioVsock supports true guest to guest communication, the VVS transport type might be an eligible option to achieve high speeds for inter- vm process communication.

Method This section will discuss how our three stage plan of IPC study, implementation and testing has functioned and what we could have done to improve our end results. IPC Study We searched online and found many articles and forum posts regarding different IPC methods and we feel this went well for the most part. However, since we failed to establish the limitation of IPC on the same host at first, we spent a fair the VVS transport type may not be immediately useful. How- amount of time researching IPC methods that in the end were ever, as a guest to host communication device, VirtioVsock of no use to us. has speeds that approach being almost three times better than its TCP counterpart. By establishing and making sure of our project limitations from the start we could have avoided a lot of unnecessary The TIPC transport on the other hand has a superior message work which could have been better spent working on some- rate compared to TCP up to about 4KB. Afterwards, the TCP thing else. transport type has a more gradual increase compared to the sharp dive that the TIPC transport takes which results in the Implementation TCP transport surpassing it. The latency of TIPC is only a At the start, it was hard to figure out how everything in XCM small marginal increase over TCP up till 4KB where TIPC interacted but we eventually learned enough to be able to starts to increase exponentially whereas TCP has a more sta- implement our methods based on how the existing transport ble curve. types functioned. The virtual machines and containers needed to run our IPC Figure 10. Qemu/KVM Message Rate methods took some time to set up properly. This was due to the fact that we had never used Docker or Qemu/KVM before and we had special options that were needed. At first we tried to use virt-manager, a graphical user interface for configuring virtual machines, but we needed special devices that were not supported by it so we eventually settled on using the Qemu command line interface. We should have set aside some time in the beginning be- fore hand to get accustomed to Docker containers and Qemu/KVM virtual machines so that when it was time for implementing them we would not make time consuming mis- takes, such as not setting up the network correctly for the VMs or sit a whole day trying to get abstract unix sockets to work for Docker when it was a simple command line option. Testing If we want to compare TIPC to VVS, we have to make an There were different options on how to start a Qemu/KVM assumption of how much of an decrease in message rate for virtual machine. We could start the virtual machines either VVS it would take to send from guest to guest. As can be through the which you could not seen in figure 10, TCP has about a halved message rate, from only control through the virsh commands on the terminal but 28000 msg/sec to 14000 msg/sec, most of which comes from also through a GUI which assisted in learning the ropes. The having to travel through an extra network stack. In compari- other way was through the qemu commands on the terminal. son, VVS saves time by avoiding the network stack so theo- retically it should allow for a slower message rate decrease. In the end we started the Qemu/KVM virtual machines through the terminal as the virtual machine manager made Therefore, we can speculate that using a halved message rate it impossible to add the device required for VirtioVsock. At decrease should be within the realm of possibility. According the end of the study when we were confirming that we were to our data, even if the message rate of the VVS transport is actually running the VMs on the correct CPU cores we ended halved it will still be faster than TIPC which speaks volumes up doing some of the tests through a VM set up by virtual

8 machine manager. We noticed some peculiar results, espe- Thesis in a future context cially for the TIPC transport, where the latency for a message Our work to find better and faster IPC methods will ensure size of 60K bytes shrunk to 114 µs per message compared that a communicating node will send its data faster, thus uti- to the 603 µs per message seen in figure8. Comparatively, lize the CPU less and therefore lead to less power consump- the amount of cycles required per message rose to roughly tion and a better environment for society. 300 µs compared to the 102 µs we have on our usual testing environment. As for any ethical aspects, we have not been able to find any ethical considerations around which IPC method to use. We believe this result stems from that the virtual machine manager uses devices to try and speed the communication CONCLUSIONS between the host and the virtual machine and it ultimately This thesis has been a selective examination of the various points to that not only the IPC methods affects the result but IPC methods that are available for inter-vm or inter-container also how the environment is set up, as was discussed in our communication between processes on the same host. As such, related works. we tried to find the best IPC methods that could fit well into the XCM framework. As we mentioned earlier we had to ensure the VMs and con- tainers were running on the same CPU core. This was difficult We implemented VirtioVsock and Transparent Inter Process because some tools needed to isolate and choose CPU cores Communication for Qemu/KVM virtual machines and Unix to run on used different indexing where some started from 0 Domain Sockets and Transparent Inter Process Communica- and some started from 1. After resolving this and confirming tion for Docker containers. They all have a more effective it through watching the CPU workload through the htop tool speed compared to TCP, at least on the more common mes- we were able to use XCM’s included test program to obtain sage sizes of 100 to 1000 bytes. test results for our chosen IPC methods. For the Docker containers, the Unix transport and TIPC trans- Also, our implementation of TIPC had difficulties with cre- port are comparable to each other which means that TIPC ating a connection without utilizing a busy-wait. While this more than likely uses a Unix Domain Socket as base for local busy-wait was minimal it might have affected the results neg- communication. An important thing to point out is the fact atively, making TIPC look worse than it is. A perfect solution that IPC via containers is always faster than virtual machines, for TIPC is more than likely possible and if found, could po- which goes to show that if you have parts of a system that tentially give us a better result. does not need the isolation or security of a virtual machines you could reap a huge performance increase by moving those Validity, Reliability & Replicability parts over to a container instead. By making sure everything is executed on the same CPU core that has been isolated from the rest of the host machine, we In the Qemu/KVM virtual machines on the other hand, the are able to get a measurement that we can say is precise. Any- TIPC transport performed the best but if VirtioVsock should one else that has access to the same tools we had and is able ever begin to support guest to guest or a workaround without to set up their hardware and software the same way should a huge performance hit is found, the VVS transport could be find a similar result. a viable option. As for replicability, anyone that truly would want to repeat If you are only running Docker containers on the same host our implementation and testing phases would need to have we recommend using the Unix socket for IPC as it has the access to the XCM framework. Without it, they would not best performance and is very easy to implement out of our have access to our tests which is practically what our entire chosen methods. If it is a Qemu/KVM virtual machine only study is based on. environment we recommend TIPC as long as the message size stays below 16KB. Source Criticism If both containers and virtual machines are being used we While IPC is not a new problem by any definition and there suggest that only TIPC should be used rather than a combina- has been done a lot of studies about it, finding sources that ac- tion of Unix, TIPC and TCP as the performance for TIPC for tually study and compare the IPC methods that we are inter- IPC between containers is negligible compared to Unix and ested in is hard. Add in that we want a study between virtual less methods would have to be maintained since TIPC works machines or containers and the articles become pretty much for virtual machines as well. non-existent. Our thesis can be potentially useful for any future researcher Since no articles has been exactly what we need for our the- that wants to find data about IPC comparisons between Unix sis we have chosen to find articles that simply compare IPC Domain Sockets, TCP Sockets, TIPC Sockets and VirtioV- methods on a host that we can apply to our work as a basis Sockets in an inter-virtual environment. They could use our for how IPC methods compare. data as a base to determine which IPC method would fit their systems like we did with certain articles[12].

9 Future Work , ser. Middleware ’07. New York, NY, For future studies of IPC, we have a few areas we would like USA: Springer-Verlag New York, Inc., 2007, pp. to research more: 184–203. [Online]. Available: http://dl.acm.org/citation.cfm?id=1516124.1516138 VirtioVsock Explore if it is possible to share guests context id with each other to enable guest to guest communication. 7. Z. Shan, X. Wang, T.-c. Chiueh, and X. Meng, “Facilitating inter-application interactions for os-level Shared Memory Since shared memory usually has the high- virtualization,” in Proceedings of the 8th ACM est performance, it would be interesting to see if a version SIGPLAN/SIGOPS Conference on Virtual Execution of it that follows the XCM requirements can be imple- Environments, ser. VEE ’12. New York, NY, USA: mented. Based on our initial study we consider MemPipe ACM, 2012, pp. 75–86. [Online]. Available: a good starting point. http://doi.acm.org/10.1145/2151024.2151036 VM Hypervisor Explore different virtual machine hypervi- 8. A. Venkataraman and K. K. Jagadeesha, “Evaluation of sors to see we if we can find methods tied to those hyper- inter-process communication mechanisms,” visors that performs better than TIPC. Xen seems to be a Architecture, vol. 86, p. 64, 2015. good starting point. 9. J. Liedtke, “Improving ipc by kernel design,” ACM ACKNOWLEDGEMENTS SIGOPS operating systems review, vol. 27, no. 5, pp. We want to give thanks to the many great persons at Ericsson 175–188, 1993. that has helped us throughout this thesis and to the company 10. I. Gelado, J. Cabezas, L. Vilanova, and N. Navarro, “The itself for allowing us to undertake this work as our thesis sub- cost of ipc: an architectural analysis,” WIOSCA, 2007. ject. We also want to thank all the teachers of the IP program at Linkoping University for all their help. 11. Z. Xiurong, “The Analysis and Comparison of Inter-Process Communication Performance Between REFERENCES Computer Nodes,” Management Science and 1. Ericsson, ’Ericsson Mobility Report November 2017’. Engineering, vol. 5, no. 3, pp. 162–164, 2011. Ericsson, 2017. 12. Y. Ren, L. Liu, Q. Zhang, Q. Wu, J. Guan, J. Kong, 2. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, H. Dai, and L. Shao, “Shared-memory optimizations for A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, “Xen inter-virtual-machine communication,” ACM Computing and the art of virtualization,” in Proceedings of the Surveys (CSUR), vol. 48, no. 4, p. 49, 2016. Nineteenth ACM Symposium on Operating Systems Principles, ser. SOSP ’03. New York, NY, USA: ACM, 13. A. C. Macdonell et al., Shared-memory optimizations 2003, pp. 164–177. [Online]. Available: for virtual machines. University of Alberta, 2011. http://doi.acm.org/10.1145/945445.945462 14. Q. Zhang and L. Liu, “Shared memory optimization in 3. D. Merkel, “Docker: Lightweight linux containers for virtualized ,” in (CLOUD), 2015 consistent development and deployment,” Linux J., vol. IEEE 8th International Conference on. IEEE, 2015, 2014, no. 239, Mar. 2014. [Online]. Available: pp. 261–268. http://dl.acm.org/citation.cfm?id=2600239.2600241 15. H. Eiraku, Y. Shinjo, C. Pu, Y. Koh, and K. Kato, “Fast 4. F. Bellard, “Qemu, a fast and portable dynamic networking with socket-outsourcing in hosted virtual translator.” in USENIX Annual Technical Conference, machine environments,” in Proceedings of the 2009 FREENIX Track, vol. 41, 2005, p. 46. ACM symposium on Applied Computing. ACM, 2009, pp. 310–317. 5. A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori, “kvm: the linux virtual machine monitor,” in 16. P. Ivanovic and H. Richter, “Performance analysis of Proceedings of the Linux symposium, vol. 1, 2007, pp. ivshmem for high-performance computing in virtual 225–230. machines,” in Journal of Physics: Conference Series, 6. X. Zhang, S. McIntosh, P. Rohatgi, and J. L. Griffin, vol. 960, no. 1. IOP Publishing, 2018, p. 012015. “Xensocket: A high-throughput interdomain transport 17. J. Maloy, “Tipc: Providing communication for linux for virtual machines,” in Proceedings of the clusters,” in Linux Symposium, vol. 2, 2004, pp. ACM/IFIP/USENIX 2007 International Conference on 347–356.

10