Bash on Ubuntu on macOS ABSTRACT of essential utilities; using a cross compiler is not sufficient Linux is a popular operating system (OS) as a production en- but a large collection of Linux utilities must be installed on vironment, while many developers prefer to use macOS for the system, some of which are not (yet) available on macOS. their daily development. One way to deal with it is running Apart from Linux on macOS, there have been many at- Linux in a virtual machine, and the other is porting develop- tempts to provide OS compatibility layers that allow pro- ment environments from Linux to macOS. However, using a grams implemented for one OS to run on another. In prin- virtual machine has a resource sharing problem, and porting ciple, these attempts are categorized into two groups. The environments is costly and often incomplete. A promising first group is those with API compatibility. They construct a approach to low-cost and seamless resource sharing is to de- software emulation stack that provides the same API as the velop a Linux compatibility layer for macOS. Unfortunately, original platform. Cygwin [2], MinGW [15], and POSIX [11] existing methods of implementing OS compatibility layers are counted in this group. The compatibility layer can be lack robustness or flexibility. In this paper, we propose anew purely implemented in user space, although programs must architecture of OS-compatibility layers. It allows user-space be recompiled. The major downside of this approach is inflex- implementation of the core emulation layer in the host OS ibility. Taking Cygwin for example, its fork implementation to improve robustness while maintaining the flexible and cannot use the copy-on-write technique due to the incom- powerful emulation ability without heavily depending on the patibility of process models between Linux and Windows. host OS kernel by exploiting virtualization technology. We This also leads to performance degradation. MinGW gives implemented our approach and confirmed that Ubuntu runs up full compliance to the Linux kernel to gain performance. on macOS. Our experimental results show that our approach The second group is those with ABI compatibility. They has reasonable performance for real world applications. aim at directly running existing binaries built for different platforms. WSL [7] and Linuxulator [8] are important ex- amples of this group. They are both in-kernel subsystems 1 INTRODUCTION that handle Linux system calls at the privileged level. Since Linux is one of the most popular operating systems (OSs). It privileged software has full control of software interrupts is widely used as not only a desktop environment but also a and page tables, they have enough flexibility to increase com- production environment. For example, 37% of the top 10 mil- patibility as much as the original kernel. Unfortunately, they lion websites were hosted by Linux [20], and more than 90% lack robustness since in-kernel subsystems are often unstable of 371,132 Amazon EC2 instances were Linux (56.4% were and they are not isolated from the host kernel. In fact, WSL Ubuntu) [14]. Therefore, many applications are developed sometimes causes the blue screen of death of Windows [6]. for Linux and there exist a large number of Linux binaries This observation indicates that robustness and compatibility and distributions. On the other hand, many software devel- are in a trade-off relationship in traditional approaches. opers prefer to use macOS instead of Linux as a development In this paper, we present a new architectural design of OS environment [16]. Therefore, there is a huge gap between compatibility layers. This design realizes both robustness production and development environments. and compatibility by utilizing virtualization technology. In To fill the gap, two different approaches are taken. One our execution model, an individual virtual machine (VM) is to install Linux in a virtual machine. However, resource is launched per guest process, and a guest binary runs in sharing between the guest and host OSs has difficulties. For a VM without the OS kernel. System calls issued by the example, the guest and host file system trees are different, guest process are trapped and emulated in a host process, and inter-process communications (IPCs) between the guest called a monitor process, created for each guest process. The and host processes is not supported. The other is to port monitor process issues host system calls to emulate guest applications and development environments from Linux to system calls, and leverages virtualization technology to trap macOS. Although various kinds of tools initially developed software interrupts and manipulate page tables for the guest for Linux were ported to macOS, porting all necessary ap- process. This design allows most of the emulation layer to plications is very costly and often incomplete. For example, be implemented in a user-space host process, while having the Linux kernel cannot be built on macOS due to the lack the flexible and powerful emulation ability without heavily depending on the host OS kernel. It also achieves seamless APSys ’17, September 2017, Mumbai, India communication between guest and host processes and high 2017. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn portability of the emulation layer. APSys ’17, September 2017, Mumbai, India We implemented a Linux compatibility layer for macOS, the Embassies ABI, which is narrower than normal system called Noah, based on our proposed design. Noah can run calls. Embassies inherits the same benefits and drawbacks of unmodified ELF binaries for x86-64 Linux 4.6 on macOS 10.11 the picoprocess’s work. El Capitan. We confirmed that Ubuntu 16.04 and Arch Linux Foreign LINUX (flinux) [19] is an emulation software to run on our implementation. We implemented emulation for run unmodified Linux binaries on Windows. It performs many Linux subsystems such as process management, mem- binary translation against Linux binaries to allow its user- ory management, virtual file systems, networks and signals. space implementation without losing flexibility. In flinux, Noah currently supports 157 out of 328 Linux system calls. system calls are intercepted via translated trampoline code. Although the implementation is still in progress, Noah can However, memory layout configuration is not so flexible build Linux kernels on it and run several X11 applications. because a guest process shares the memory space with the Noah uses Hypervisor.framework [1] for its virtualization corresponding host process. Additionally, it is significantly component, so we do not need to modify the macOS kernel. slower than ours due to the online scan-and-patch process. Our experimental results showed that the overhead of Dune [3] resembles our work in that they both run guest Linux kernel build time on Noah was around 7.2% and the programs in virtual machines with higher-level interfaces exec system call was 2.4 times faster than that of macOS. than the machine architecture. However, their goals are dif- This paper is organized as follows. Section 2 shows re- ferent; Dune aims at providing user programs direct access lated work. Section 3 explains the architectural design of to hardware features, whereas we emulate the kernel inter- our approach and Section 4 describes the implementation of face of a different OS. We only use ring 3 in VMX non-root our Linux compatibility layer for macOS. Section 5 presents mode for running its guest process, whereas Dune consists experimental results. Section 6 summarizes this paper. of processes running in different rings and VMX modes. Nova [17] is a redesign of virtual machine monitors (VMMs) 2 RELATED WORK from the viewpoint of microkernels. Both Nova and our work Xax [5] abstracts an execution environment of native code put complicated components, such as page table manage- as a lightweight process, called piroprocess. A picoprocess is ment, in user space in order to improve robustness. However, created and mediated by an OS-specific monitor program, they differ in that Nova only isolates complex parts ofVMMs and communicates with it via highly restricted system calls. from the host kernel, whereas we isolate all kernel compo- To set up a restricted execution environment, a picoprocess nents from the guest kernel and put them in a host process. has a boot loader and trampoline code inside it to communi- OSv [12] and our work are similar in that they both are cate with the monitor. A picoprocess is similar to our guest aiming at constructing from scratch a lightweight kernel in- process in that system calls are mediated by a host process. terface of Linux. However, their goals and implementations However, the boot loader and trampoline code inside the are distinct. On one hand, OSv is a mere operating system for picoprocess incur ahead-of-time or just-in-time patching virtual machines. OSv focuses on performance improvement procedure to restrict system calls. The implementation of rather than compatibility; it even exposes non-POSIX inter- picoprocesses is also different from ours. Their Linux imple- face to user programs and is optimized to run faster with mentation uses ptrace to restrict system calls, suffering from executables specially modified for OSv. On the other hand, performance hit and complication on memory management ours is not an operating system but an OS compatibility layer. because ptrace does not allow direct manipulation of memory Its main aim is to accomplish full compatibility with Linux, map of the target process. Their Windows implementation without giving up as much performance as possible. uses a kernel driver to mediate system calls, suffering from kernel dependency and robustness decline. Our monitor pro- 3 DESIGN cesses can directly trap system calls and other privileged events without depending on the host kernel by exploiting a Figure 1 shows the design of our OS compatibility layer.