WHITE PAPER Evaluating Linux IPC Performance Ola Dahl, Principal Engineer, CTO Office Multicore System-on-Chip solutions, offering parallelization and partitioning, are increasingly used in communication systems. As the number of cores increase, often in combination with increased on-chip heterogeneity in the form of hardware accelerated functionality and digital signal processors, we see increased demands on effective communication, inside a multicore node but also on inter-node system-level. In addition, we see a steady increase in the usage of Linux in embedded systems. As a consequence, new challenges arise in balancing cost and performance when selecting, integrating, and adapting mechanisms for multicore inter-process communication (IPC) in Linux-based communication systems. Linux comes with a multitude of IPC mechanisms. This paper gives a brief overview of IPC in Linux, and metrics for evaluation of IPC performance are described. Measurements of throughput and latency are presented, using a selection of IPC methods. Profiling, for the purpose of finding bottlenecks and enhancing performance, is discussed and exemplified. Requirements and classification of requirements for IPC is discussed, and a systematic approach for IPC selection is outlined. The paper illustrates some of the complexity involved when choosing an IPC mechanism for Linux, and provides, through examples and discussions, a base for further development and research. Introduction interconnect complexity, and with onboard L3 caches. [4] Enea has worked with communication systems for The SOC Networking Driver candidate more than 30 years. The OSE Real-time operating architecture is used as a base for quantitative system, conceived in 1985, has been and is used in predictions regarding performance and the number a large part of today’s communication systems. of cores. Estimates of the number of cores are Enea has a product portfolio also including Linux, given, based on assumptions of a constant die area, complementing OSE and used together with other a per-year increase of number of cores by 1.4x, Enea products in Middleware and in Services. core frequency and accelerator engine frequency by The size of systems running OSE as of today can 1.05x, and assuming that logic, memory, cache be illustrated by the number of bits used for hierarchy, switching-fabric and system interconnect representing the process identity. This quantity was will scale consistently with the number of cores. The recently extended from 16 bits to 20 bits, indicating resulting prediction indicates that in a five year that the limit of 64K peers to keep track of was too perspective from now, the number of cores will be small. This gives a size estimate, in terms of the somewhere between 60 (assuming 1 core in 2007) number of processes that participate in the and well above 100 (assuming instead 4 cores in communication, of the communication systems we 2007). It can also be seen that in this time frame, a consider. performance increase of a factor of 10 can be The increase of multicore can be seen e.g. in expected. [4] reports from the International Technology Roadmap The increase of Linux in embedded systems is for Semiconductors (ITRS). The ITRS 2012 indicated by e.g. EE Times, in a market study [1] UPDATE [3] classifies hardware into different where it can be seen that among all users of categories, referred to as drivers [4] . The SOC embedded operating systems, half of them use Networking Driver captures future hardware designs Linux. It can also be seen, by consulting the for communication infrastructure and networking corresponding study from 2012 [2] , that this figure systems, and is therefore of interest for the has been increasing over the latest four years. questions considered in this paper. IPC in Linux is illustrated schematically in Figure ITRS gives predictions for the different drivers. The 1. There are cooperating processes, labelled P1 to predictions for the SOC Networking Driver, using a P4, communicating using an agreed IPC proposed architecture template as a base, indicate mechanism, and there are multiple cores, labelled a continued move towards multicore architectures C0 to C3, on which the processes are scheduled. with heterogeneity, with an increased on-chip The underlying hardware, with caches, controllers, Enea is a global vendor of Linux and Real-time operating system solutions including middleware, tools, protocols and services. The company is a world leader in developing software platforms for communication-driven products in multiple verticals, with extreme demands on high-availability and performance. Enea’s expertise in operating systems and high availability middleware shortens development cycles, brings down product costs and increases system reliability. The company’s vertical solutions cover telecom handsets and infrastructure, medtech, automotive and mil/aero. Enea has offices in Europe, North America and Asia, and is listed on NASDAQ OMX Nordic Exchange Stockholm AB. For more information please visit enea.com or contact us at [email protected]. WHITE PAPER and on-chip interconnect mechanisms, is also work remains. However, recent references from illustrated. LWN, such as [14] and [15] indicate the opposite. Another observation is that a new IPC mechanism known as kdbus, and referred to as kernel "dbus- like" code for the Linux kernel by its developers [5] has recently been developed. An recent overview of IPC in Linux is given by Michael Kerrisk in a presentation [6] , with accompanying slides [7] . A selection of IPC mechanisms in Linux, inspired from the presentation [6] , laid out on a timeline, is shown in Figure 2. Figure 1 A schematic view, illustrating Linux processes communicating using IPC, in a multicore system. The communication between two Linux nodes is illustrated in Figure 1, using the label Inter-node for the outgoing and incoming communication for process P4. We remark here that the remainder of Figure 2 A timeline, showing a selection of IPC this paper focuses on the intra-node mechanisms that are part of Linux. communication, i.e. communication between Linux processes inside one multicore system. The timeline in Figure 2 starts with pipes, in 1964. The communication between processes is This is when pipes were proposed, by Malcom typically performed in stages. At first, there may be Douglas Mcllroy [7] , who is also credited with the an initial phase where contact between the creation of UNIX commands such as spell, diff, sort, processes is established, and a communication and join. This was five years before UNIX, which channel is set up. Communication is then was born in 1969. performed, using an agreed protocol. During The release of UNIX System V in 1983 is shown communication it may be desirable to monitor the in the timeline in Figure 2. We see the release of communication and to react on events. As an Linux 1.0 in March 1994. The timeline also shows example, it may be required that failure to the POSIX real-time standard from 1995, the communicate - perhaps due to a process being introduction in the Linux kernel of POSIX shared terminated due to an exceptional condition - shall be memory in 2001, POSIX named semaphores in detected, and as a reaction to such an event, a 2003, and POSIX message queues in May 2004. communication channel shall be shut down and/or a There are also more recent changes shown, such process shall be restarted. As a final step, when the as cross-memory attach (CMA), which was communication session is ready, the communication introduced in Linux kernel 3.2, in January 2012. channel is closed. Cross-memory attach enables one process to The four functions of initialization, communication, write directly into another process' memory space. It monitoring and termination, are performed in also enables a process to read directly from another different ways, or for some of them perhaps not at process' memory space. CMA was motivated by a all, depending on the IPC mechanism chosen. desire to make MPI programs faster. Additional information about IPC mechanisms IPC and Linux built-in to Linux can be found e.g. in the book The Linux Programming Interface [9] . Useful information Considering the UNIX heritage, with stable and well may also be found by consulting additional tried-out mechanisms such as pipes and sockets, presentations and man pages, linked from [10] . one might assume that IPC in Linux is a solved and There are additional IPC mechanisms that can be done problem - and as a consequence, not much used together with Linux. A selection of such, not so Enea is a global vendor of Linux and Real-time operating system solutions including middleware, tools, protocols and services. The company is a world leader in developing software platforms for communication-driven products in multiple verticals, with extreme demands on high-availability and performance. Enea’s expertise in operating systems and high availability middleware shortens development cycles, brings down product costs and increases system reliability. The company’s vertical solutions cover telecom handsets and infrastructure, medtech, automotive and mil/aero. Enea has offices in Europe, North America and Asia, and is listed on NASDAQ OMX Nordic Exchange Stockholm AB. For more information please visit enea.com or contact us at [email protected]. WHITE PAPER tightly built-in, IPC mechanisms, is shown in Figure something is lacking in Linux. There is, in other 3. words, a real need for fast IPC that Linux doesn't address”. Performance Metrics When selecting an IPC mechanism, for a specific project or product, trade-offs between cost and performance are often considered. As measures of performance, one might use throughput, measuring the transferred number of bits per second, and/or latency, measuring the communication delay, from Figure 3 A timeline, showing a selection of IPC initiation to completion of transfer of a specific data mechanisms that can be used with Linux.