Bash on on macOS

ABSTRACT of essential utilities; using a cross compiler is not sufficient is a popular operating system (OS) as a production en- but a large collection of Linux utilities must be installed on vironment, while many developers prefer to use macOS for the system, some of which are not (yet) available on macOS. their daily development. One way to deal with it is running Apart from Linux on macOS, there have been many at- Linux in a virtual machine, and the other is porting develop- tempts to provide OS compatibility layers that allow pro- ment environments from Linux to macOS. However, using a grams implemented for one OS to run on another. In prin- virtual machine has a resource sharing problem, and porting ciple, these attempts are categorized into two groups. The environments is costly and often incomplete. A promising first group is those with API compatibility. They construct a approach to low-cost and seamless resource sharing is to de- software emulation stack that provides the same API as the velop a Linux compatibility layer for macOS. Unfortunately, original platform. [2], MinGW [15], and POSIX [11] existing methods of implementing OS compatibility layers are counted in this group. The compatibility layer can be lack robustness or flexibility. In this paper, we propose anew purely implemented in user space, although programs must architecture of OS-compatibility layers. It allows user-space be recompiled. The major downside of this approach is inflex- implementation of the core emulation layer in the host OS ibility. Taking Cygwin for example, its fork implementation to improve robustness while maintaining the flexible and cannot use the copy-on-write technique due to the incom- powerful emulation ability without heavily depending on the patibility of process models between Linux and Windows. host OS kernel by exploiting virtualization technology. We This also leads to performance degradation. MinGW gives implemented our approach and confirmed that Ubuntu runs up full compliance to the Linux kernel to gain performance. on macOS. Our experimental results show that our approach The second group is those with ABI compatibility. They has reasonable performance for real world applications. aim at directly running existing binaries built for different platforms. WSL [7] and Linuxulator [8] are important ex- amples of this group. They are both in-kernel subsystems 1 INTRODUCTION that handle Linux system calls at the privileged level. Since Linux is one of the most popular operating systems (OSs). It privileged software has full control of software interrupts is widely used as not only a desktop environment but also a and page tables, they have enough flexibility to increase com- production environment. For example, 37% of the top 10 mil- patibility as much as the original kernel. Unfortunately, they lion websites were hosted by Linux [20], and more than 90% lack robustness since in-kernel subsystems are often unstable of 371,132 Amazon EC2 instances were Linux (56.4% were and they are not isolated from the host kernel. In fact, WSL Ubuntu) [14]. Therefore, many applications are developed sometimes causes the blue screen of death of Windows [6]. for Linux and there exist a large number of Linux binaries This observation indicates that robustness and compatibility and distributions. On the other hand, many software devel- are in a trade-off relationship in traditional approaches. opers prefer to use macOS instead of Linux as a development In this paper, we present a new architectural design of OS environment [16]. Therefore, there is a huge gap between compatibility layers. This design realizes both robustness production and development environments. and compatibility by utilizing virtualization technology. In To fill the gap, two different approaches are taken. One our execution model, an individual virtual machine (VM) is to install Linux in a virtual machine. However, resource is launched per guest process, and a guest binary runs in sharing between the guest and host OSs has difficulties. For a VM without the OS kernel. System calls issued by the example, the guest and host file system trees are different, guest process are trapped and emulated in a host process, and inter-process communications (IPCs) between the guest called a monitor process, created for each guest process. The and host processes is not supported. The other is to port monitor process issues host system calls to emulate guest applications and development environments from Linux to system calls, and leverages virtualization technology to trap macOS. Although various kinds of tools initially developed software interrupts and manipulate page tables for the guest for Linux were ported to macOS, porting all necessary ap- process. This design allows most of the emulation layer to plications is very costly and often incomplete. For example, be implemented in a user-space host process, while having the Linux kernel cannot be built on macOS due to the lack the flexible and powerful emulation ability without heavily depending on the host OS kernel. It also achieves seamless APSys ’17, September 2017, Mumbai, India communication between guest and host processes and high 2017. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn portability of the emulation layer. APSys ’17, September 2017, Mumbai, India

We implemented a Linux compatibility layer for macOS, the Embassies ABI, which is narrower than normal system called Noah, based on our proposed design. Noah can run calls. Embassies inherits the same benefits and drawbacks of unmodified ELF binaries for -64 Linux 4.6 on macOS 10.11 the picoprocess’s work. El Capitan. We confirmed that Ubuntu 16.04 and Arch Linux Foreign LINUX (flinux)19 [ ] is an emulation software to run on our implementation. We implemented emulation for run unmodified Linux binaries on Windows. It performs many Linux subsystems such as process management, mem- binary translation against Linux binaries to allow its user- ory management, virtual file systems, networks and signals. space implementation without losing flexibility. In flinux, Noah currently supports 157 out of 328 Linux system calls. system calls are intercepted via translated trampoline code. Although the implementation is still in progress, Noah can However, memory layout configuration is not so flexible build Linux kernels on it and run several X11 applications. because a guest process shares the memory space with the Noah uses .framework [1] for its virtualization corresponding host process. Additionally, it is significantly component, so we do not need to modify the macOS kernel. slower than ours due to the online scan-and-patch process. Our experimental results showed that the overhead of Dune [3] resembles our work in that they both run guest Linux kernel build time on Noah was around 7.2% and the programs in virtual machines with higher-level interfaces exec was 2.4 times faster than that of macOS. than the machine architecture. However, their goals are dif- This paper is organized as follows. Section 2 shows re- ferent; Dune aims at providing user programs direct access lated work. Section 3 explains the architectural design of to hardware features, whereas we emulate the kernel inter- our approach and Section 4 describes the implementation of face of a different OS. We only use ring 3 in VMX non-root our Linux compatibility layer for macOS. Section 5 presents mode for running its guest process, whereas Dune consists experimental results. Section 6 summarizes this paper. of processes running in different rings and VMX modes. Nova [17] is a redesign of virtual machine monitors (VMMs) 2 RELATED WORK from the viewpoint of microkernels. Both Nova and our work Xax [5] abstracts an execution environment of native code put complicated components, such as page table manage- as a lightweight process, called piroprocess. A picoprocess is ment, in user space in order to improve robustness. However, created and mediated by an OS-specific monitor program, they differ in that Nova only isolates complex parts ofVMMs and communicates with it via highly restricted system calls. from the host kernel, whereas we isolate all kernel compo- To set up a restricted execution environment, a picoprocess nents from the guest kernel and put them in a host process. has a boot loader and trampoline code inside it to communi- OSv [12] and our work are similar in that they both are cate with the monitor. A picoprocess is similar to our guest aiming at constructing from scratch a lightweight kernel in- process in that system calls are mediated by a host process. terface of Linux. However, their goals and implementations However, the boot loader and trampoline code inside the are distinct. On one hand, OSv is a mere operating system for picoprocess incur ahead-of-time or just-in-time patching virtual machines. OSv focuses on performance improvement procedure to restrict system calls. The implementation of rather than compatibility; it even exposes non-POSIX inter- picoprocesses is also different from ours. Their Linux imple- face to user programs and is optimized to run faster with mentation uses ptrace to restrict system calls, suffering from executables specially modified for OSv. On the other hand, performance hit and complication on memory management ours is not an operating system but an OS compatibility layer. because ptrace does not allow direct manipulation of memory Its main aim is to accomplish full compatibility with Linux, map of the target process. Their Windows implementation without giving up as much performance as possible. uses a kernel driver to mediate system calls, suffering from kernel dependency and robustness decline. Our monitor pro- 3 DESIGN cesses can directly trap system calls and other privileged events without depending on the host kernel by exploiting a Figure 1 shows the design of our OS compatibility layer. It hardware-based virtualization technology. consists of three components: a VMM module, guest VMs, Embassies [9] extends picoprocesses by adding rich func- and monitor processes. The VMM module is a component tions such as the IP protocol and user interface APIs. Based on in the host OS kernel that provides a VM management in- Embassies, a successor work [10] proposed an architecture to terface to user-space applications. We can exploit several run POSIX applications in picoprocesses. In this architecture, OS-standard VMM modules; for example, Linux has KVM, POSIX ABI is provided by a POSIX running inside a FreeBSD has vmmapi, and macOS has Hypervisor.framework. picoprocess. The POSIX emulator consists of several subsys- The guest VMs work as containers of guest binaries and are tems such as the virtual file system and IP multiplexer, andis used to trap access from guest processes. The monitor pro- implemented based on the Embassies ABI. This approach re- cesses are regular processes of the host OS that emulate quires a large emulation layer to convert the POSIX ABI into system calls and manage VMs through the VMM module. Bash on Ubuntu on macOS APSys ’17, September 2017, Mumbai, India

The host OS Guest VMs processes can be communicated with each other as if they communicates on the same OS, not on different OSs. An important characteristic of our design is that OS com- monitor process load & guest process manage patibility layers can be implemented in user space. This char- user acteristic leads to two advantages. The first is robustness. space emulate trap system callls system callls Existing OS compatibility layers with ABI compatibility are kernel upcall usually implemented in kernel space, therefore a bug in this monitor space layer might cause a kernel crash. In contrast, bugs in the kernel (no kernel) monitor process in our design do not lead to kernel crashes. VMM module Although the VMM module runs in the kernel, it is relatively robust because it is small and well-maintained as a part of Figure 1: The design of our OS compatiblity layer the standard kernel. The second is portability. The monitor process is implemented in a user-space process and loosely coupled with the internal APIs of the host OS kernel. User- A guest application is executed in the following way. First, space processes are relatively easy to be ported to another a monitor process corresponding to the guest application OS, and it can also use any cross-platform utility libraries or is launched. It creates a VM through the VMM module and even high-level languages such as Rust, Go, and Ruby. loads the guest binary image into the memory space of the While our design has the robustness by implementing OS VM. Then, the monitor process asks the VMM module to start compatibility layers in user space, it also has the flexible and execution of the VM from the entry point of the guest binary, powerful emulation ability thanks to virtualization technol- thereby the control is passed to the guest process. While the ogy. By running a guest process inside a VM, system calls guest process is running, it will issue system calls. These and other privileged events such as page faults are trapped system calls are trapped by the VMM module and the VMM by the hardware and the control is passed to the monitor module upcalls the monitor process. The monitor process process via the VMM module. The monitor process has the emulates the trapped system calls by using the system calls total control of the memory layout with the ability to ma- of the host OS. The monitor process then returns the result nipulate page tables. Therefore, it is possible to implement of emulated system calls to the VM by way of the VMM complex memory management of the target OS such as copy- module, and the control is returned to the guest process. on-write page mappings between processes. Therefore, this In our design, a guest process runs in a VM and a monitor approach can achieve ABI compatibility without requiring process is created for each VM. Therefore, when a guest modifications to either guest processes or host kernels. process tries to create another process, the corresponding monitor process first creates another monitor process, and 4 IMPLEMENTATION the new monitor process creates a new VM. In the case of We have implemented a Linux compatibility layer for macOS processes having a parent-child relationship like in , based on our design. The implementation, Noah, targets x86- the new monitor process clones the VM states of the original 64 Linux 4.6 and macOS 10.11 El Capitan or later. For the guest process and set the states to the newly created VM. virtualization foundation, we utilized Hypervisor.framework, Then, the original monitor process returns the result of the a built-in of macOS that provides a set of user space process creation to the original guest process, and the new APIs to create and manage VMs. By relying on the built-in monitor process passes the control to the new guest process. library of the OS, we could avoid writing kernel modules, One advantage of this design is that process scheduling can improving robustness and reducing implementation costs. be left to the host OS. A guest process can get control when Note that macOS only runs on Intel CPUs, not AMD’s ones. the corresponding monitor process is scheduled by the host OS, therefore, in effect, the guest process is schedule by the 4.1 Boot Process host OS as an ordinary process. This allows fair scheduling A guest Linux process is created when the noah command among multiple guest and host processes. Another advantage is executed or a guest process issues a fork(2) system call. is that resource sharing between guest and host processes When a guest process is being created, the monitor process becomes seamless. For example, they can share a file system first launches a new VM using Hypervisor.framework. Then, tree because both processes access the file system of the host to avoid injecting custom boot code into the VM, the monitor OS. They can also use a pipe to communicate with each other. process manipulates the VM registers so that the VM directly A pipe access from the guest process is converted to that of enters x86-64 long mode. Additionally, to prevent any code the host OS by the monitor process, and the communications from running in privileged mode in the VM, the monitor are handled by the host OS. Therefore, the guest and host process initializes some control registers (such as CR0 and APSys ’17, September 2017, Mumbai, India

CR4) and model-specific registers (including IA32_EFER) we designed a VM and its monitor process to share the same with empty settings. In particular, to trap system calls issued mapping from virtual addresses to physical addresses for a by the guest Linux process, the IA32_EFER.SCE bit is cleared, particular region. To do so in the VM’s page table, we need thereby the SYSCALL instruction is disabled. to obtain the physical address corresponding to the virtual In x86-64, some system registers need to hold physical address. Unfortunately, macOS does not support such opera- memory addresses of memory data structures (e.g., page ta- tion. On the other hand, Hypervisor.framework supports an bles and segment descriptor tables). To allocate such data API to set the physical address of an EPT entry by specifying structures, we reserved a region of 1 GB in the physical ad- a virtual address of the monitor process. dress space in the VM, which is not mapped in user space. In the VM’s page table, we use straight (identity) mapping The monitor process allocates data structures from this re- where the virtual address is identical to the physical address. gion and initializes them with empty settings, except for This mapping is simple but has a limitation due to the hard- page tables (see Section 4.3 for detail memory management). ware bus width. The physical address width of the current Intel CPU series is no more than 39 bits, whereas the virtual 4.2 ELF Loader address width is 48 bits. Therefore, we cannot map the upper We implemented our own ELF loader to load Linux ELF 9-bit virtual address space in a VM with this mapping. Fortu- executable files into VMs. The loader is invoked just after nately, current Linux does not use this part. In addition, we the noah command is executed or a guest Linux process have never observed any application exhausting 256 GB (i.e., issues the execve(2) system call. When the loader is given maximum addressable size with 39 bit) of virtual address a path to a Linux ELF executable file, it first opens the Linux space. The only exception to this mapping is the top 1-GB ELF loader fileld.so ( ) through the virtual file system in the region, where the physical page is not mapped in the virtual monitor process (see Section 4.5 in detail). It then uses the address space to hide it from user space. It is used to locate internal version of mmap() to map the content of the loader system data structures such as page tables and segment de- file into the guest Linux address space in the VM.After scriptor tables (described above in Section 4.2). Consequently, setting up the execution environment of ld.so, it passes the VM’s page table has 511-GB straight mapping and 1-GB the control to ld.so with the Linux ELF executable file as empty mapping. an argument. Finally, ld.so loads the Linux ELF executable, The implementation of the other part of memory manage- constructs memory segments, and resolves dynamic linked ment subsystem is surprisingly simple. We manage memory libraries in the emulated environment as usual. regions with the vm_area_struct structures of Linux. When a The ELF loader also supports the setuid bit. If the monitor guest Linux process issues mmap(2) or other memory-related process is executed as root and the setuid bit of the target system calls, the monitor process modifies these structures ELF executable file is set, the monitor process changes the and manipulates the EPT by way of Hypervisor.framework. effective user ID of the guest Linux process to that of thefile Since the API of Hypervisor.framework to manage EPTs ac- owner. For example, a setuid-root Linux command can write cepts a virtual address, physical page management can be to files owned by root. Note that guest Linux processes share left to macOS. the same user name and ID with the host macOS. To support multiple guest processes, monitor processes need to communicate with each other through an IPC mech- 4.3 Memory Management anism. We chose to use our own shared memory allocator, Since we use a VM to run a Linux guest process, we need to mainly for performance improvement. When the noah com- manage page tables by ourselves. In general, a VM involves mand starts, we pre-allocate a few gigabytes of memory two page tables: the VM’s page table and the Extended Page region using mmap(2) with the MAP_SHARED flag in the mon- Table (EPT). To avoid the cost of handling two page tables, itor process. Data structures to be shared among monitor we should fix one page table and manipulate only the other. processes are allocated from it. Note that the pre-allocated Which one should be fixed is a design choice. buffer consumes little memory owing to lazy page allocation. We chose to fix the VM’s page table and manipulate the EPT. One reason of this is performance. When a VM is 4.4 Process Management switched to another, the TLB of the VM’s page table is flushed We implemented a subset of the clone(2) system call; only because the page table is changed. On the other hand, the simple forking and thread creation are supported. Unfortu- TLB of EPT is not flushed on VM switches, if the tagged TLB nately, Hypervisor.framework does not support a fork of a for EPT, called Virtual Processor ID (VPID), is supported. process holding a VM, so we need a work around. The actual Therefore, we can reduce the number of page walks by using handling of a fork is as follows. First, the monitor process huge pages in the VM’s page table. Another reason is to saves the current VM state and destroys the VM. Second, make debugging easier. At the early stage of development, the monitor process forks. Third, each of the two processes Bash on Ubuntu on macOS APSys ’17, September 2017, Mumbai, India launches a new VM. Finally, they restore the saved state and Table 1: Macro benchmark results on macOS and Noah start the execution. We synchronize these VM restarts using condition variables to avoid race condition. Thread creation Benchmark name macOS Noah % is straightforward because Hypervisor.framework provides Dhrystone (LPS) 34438960.2 38265208.1 -10% such APIs. Process-specific data, such as memory region map- compress-7zip (MIPS) 9724 9277 4.80% pings, futex structures, signal handlers, and vfs structures, sqlite (sec) 3.55 4.11 15.8% are stored in the shared memory area (see Section 4.3). postmark (TPS) 2308 929 148% kernel build (sec) 106.2 113.8 7.2% 4.5 Virtual File System File access from guest Linux processes is basically forwarded to the host macOS file system. However, to support Linux- Before evaluations, we need to refer to an important bug of style file system trees, we emulate virtual file system (VFS) the current Hypervisor.framework, which is a performance of Linux in the monitor process. The VFS consists of a path degradation bug about VM creation speed. This bug causes translator and object-oriented programming (OOP) compo- the slowdown of VM creation as more VMs are created. In nent. The path translator converts a virtual path to the host addition to the creation speed, kernel_task, a core kernel path by resolving symbolic links and virtual mount points. thread of macOS, consumes more memory and never releases The OOP component provides an interface to install custom it, and eventually macOS freezes. Therefore, we suspect that file systems. The VFS allows us to expose macOS’s rootfile is a memory leak bug. Due to this bug, performance of fork system to Linux programs without breaking the Linux user on Noah degrades as processes are forked. This will be fixed space. Also, virtual file systems like sysfs and procfs canbe in the future version of Hypervisor.framework. Therefore, implemented upon this facility. The implementation of the the performance results will be worse than it actually is. VFS is designed to be independent from the host OS archi- tecture for the most part. This design significantly reduces 5.1 Macro Benchmarks the cost of porting Noah to other platforms than macOS. We measured the performance of Dhrystone from UnixBench [13] 4.6 Other Subsystems and compress-7zip, sqlite, postmark from Phoronix Test Suite [18]. We also measured the Linux kernel build time. Most of the other Linux system calls are passed-through to We ran all benchmarks from UnixBench with single core macOS with flag conversion and structure adjustment. For because some benchmarks got unstable with multiple cores getpid(2) example, the monitor process handles by simply on Noah. Other benchmarks ran on 4 cores. compress-7zip calling the equivalent system call of macOS and returning the and Dhrystone are CPU-bound, and sqlite and postmark value to the guest process. Signals sent to a monitor process are I/O bound. kernel build builds vmlinux of a Linux are routed to the corresponding VM with proper conversion. kernel with single core. The Linux kernel is based on ver- Signal handlers of the monitor process just record the ar- sion 3.4.113 but is modified so that it can be compiled on rivals of the signals. Before the next entrance to the VM, the macOS with cross toolchains. The kernel configuration is monitor process checks the records, and if signal arrivals are allnoconfig. Table 1 shows the result. The units are Loop recorded, it creates a new signal stack frame in the VM and Per Second (LPS) for Dhrystone, Million Instructions Per set the IP to the registered signal handler. Second (MIPS) for compress-7zip, Transaction Per Second (TPS) for postmark, and seconds for sqlite and kernel build. 5 EVALUATION For LPS, MIPS, and TPS, higher is better, and for seconds, This section shows evaluations of Noah performance and lower is better. compatibility. For performance, we ran a couple of bench- The results show that Noah incurred low overheads on marks on Noah and on macOS. The benchmarks are divided CPU-bound applications. This is reasonable because the into two groups: macro benchmarks and micro benchmarks. Linux process directly run on processors in hardware-assisted The former shows the overall performance of Noah in real- VMs. On the other hand, I/O-bound applications had rela- world applications, and the latter shows the overhead on tively large overheads. This would be caused by system call system calls. For compatibility, we describe the current com- emulation (see next section for detail). Especially, postmark patibility status of Noah with Linux kernel. incurred high overhead of 148%, but this benchmark uses We carried out all evaluations on MacBook Pro Early 2015 server workload for production environments and we believe model with 2-core / 4-thread 3.1GHz Intel Core i7, 16GB this is an acceptable overhead in development environments. DDR3 memory, and 512 GB SSD. It runs Ubuntu 16.04 on ma- The overhead of kernel build is within a reasonable range cOS Sierra 10.12.5. Noah’s version is “v.benchmark.2017.06”. (7.2%). This suggests that reasonable overhead. APSys ’17, September 2017, Mumbai, India

Table 2: Micro benchmark results on macOS and Noah

Benchmark name macOS Noah Unit Overhead System call 8279169.8 738597.3 LPS 1121% fork + exec 408.7 393.9 LPS 3.77% UnixBench/execl 546.2 1325.1 LPS -58.8% UnixBench/file read 1024 bufsize * 2000 1698850.0 455603.0 KBps 273% UnixBench/file read 4096 bufsize * 8000 4062016.5 1468246.2 KBps 177% UnixBench/file write 1024 bufsize * 2000 1502291.3 445372.9 KBps 137% UnixBench/file write 4096 bufsize * 8000 4271771.7 1530728.9 KBps 179% UnixBench/pipe 1209016.7 246907.7 LPS 390% UnixBench/pipe based context switching 175200.2 75834.3 LPS 131%

5.2 Micro Benchmarks 5.3 Compatibility with Linux Kernel We ran most benchmarks from UnixBench that could run on How compatible Noah is with the Linux kernel is an impor- Noah. In addition, we wrote micro benchmarks for fork + tant question. To measure the quality of Linux compatibility exec and System call by ourselves. fork + exec measures layers, Linux Test Project [4] can be used. It runs many test the number of fork + exec executed in a second. System call cases for all system calls to make sure that the system imple- measures the number of getpid(2) system calls, without ments expected behaviors of system calls. However, we did caching, executed in a second. Table 2 shows the results. not carried out such experiments because the current imple- The results show that System call incurred the highest mentation of Noah is still in early stage. The implementation overhead (1121%). This is the expected result because Noah lacks a large part of the full Linux kernel interface; there incurs six context switches for each system call: (1) the guest are 157 unimplemented system calls among 328 currently. VM to the VMM, the VMM to the monitor process, the mon- Hence, we would not have insightful result yet. Thorough itor process to the host kernel, and three reverse directions. quality evaluation will be executed in our future work. This benchmark measured only the context switch overhead Though there are still many unimplemented features, Noah and was the worst case scenario. On the other hand, in execl, already can run many real-world applications. It can run Noah outperforms macOS (2.4 times faster) since execve in package managers such as apt and pacman, development Noah just replaces the contents of VMs. The fork + exec tool chains such as gcc, vim, make, binutils, and Ruby. It of benchmark has almost the same performance on Noah and course runs bash and daily commands like ls, and even X macOS (3.77%). This indicates that the speed of a VM cre- applications such as xeyes, xfwrite, and DooM3. Network ation, snapshot and restore for fork emulation is fast enough. utilities such as nc also works. Much more applications will Note that this number will get even better if the performance work in the future as the development proceeds. degradation bug of Hypervisor.framework is fixed. The file and pipe showed modest overheads. Since we 6 SUMMARY AND FUTURE WORK used an SSD, both benchmarks were relatively CPU bound This paper described the design and implementation of a rather than I/O bound. Since Noah’s VFS is a thin emula- novel OS compatibility layer. Our design improves robust- tion layer, the overheads mostly came from those of system ness, compatibility, seamlessness, and portability by exploit- calls. In fact, the file benchmark measured file access per- ing virtualization technology. In our design, every virtual formance by calling read or write system calls many times. guest process runs inside a VM and the system calls are Therefore, the file access speed is limited by the buffer sizeof trapped by the corresponding monitor process. The imple- each system call multiplied by the number of System call mentation, Noah, demonstrated that unmodified Linux exe- execution per seconds. For example, in the file read experi- cutables, including gcc and X applications, run on macOS. ment with 1024 buffer size, the performance in KBps willbe The implementation still lacks a large part of the full Linux the same as the number of read(2) system calls issued per kernel interface. Some of them should require hard work or second. Since Noah can issue system calls at most 738597.3 new techniques to keep the implementation fast and concise. times per second, its upper limit is the same, although ac- For example, ptrace(2) is such a system call that is left tual value is slower than that. There are non-negligible cost unimplemented because of difficulties in implementing inter- in file access or other micro benchmarks, but we also note process synchronization with satisfactory performance. We that macro benchmarks indicated that such overhead for will complete the implementation in our future work. real-world applications was in an acceptable range. Bash on Ubuntu on macOS APSys ’17, September 2017, Mumbai, India

REFERENCES [1] Apple. 2017. Hypervisor | Apple Developer Documentation. https: //developer.apple.com/documentation/hypervisor. (2017). [accessed 2017-06-14]. [2] Cygwin authors. 2017. Cygwin. https://www.cygwin.com. (2017). [accessed 2017-06-14]. [3] Adam Belay, Andrea Bittau, Ali José Mashtizadeh, David Terei, David Mazières, and Christos Kozyrakis. 2012. Dune: Safe User-level Access to Privileged CPU Features. In Proceedings of the 10th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 2012). 335–348. [4] LTP developers. 2012. LTP - Linux Test Project. https:// linux-test-project.github.io/. (2012). [accessed 2017-06-17]. [5] John R. Douceur, Jeremy Elson, Jon Howell, and Jacob R. Lorch. 2008. Leveraging Legacy Code to Deploy Desktop Applications on the Web. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2008). 339–354. [6] gyf304. 2016. MLton crashes and BSODs. https://github.com/Microsoft/ BashOnWindows/issues/847. (2016). [accessed 2017-06-01]. [7] Jack Hammons. 2016. Windows Subsystem for Linux Overview. https://blogs.msdn.microsoft.com/wsl/2016/04/22/ windows-subsystem-for-linux-overview/. (2016). [accessed 2017-06-14]. [8] Brian N. Handy, Rich Murphey, and Jim Mock. 2017. Chapter 10. Linux Binary Compatibility. https://www.freebsd.org/doc/handbook/ linuxemu.html. (2017). [accessed 2017-06-14]. [9] Jon Howell, Bryan Parno, and John R. Douceur. 2013. Embassies: Radically Refactoring the Web. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2013). 529–545. [10] Jon Howell, Bryan Parno, and John R. Douceur. 2013. How to Run POSIX Apps in a Minimal Picoprocess. In Proceedings of the 2013 USENIX Annual Technical Conference. 321–332. [11] The IEEE and The Open Group. 2016. The Open Group Base Specifi- cations Issue 7, 2016 Edition. http://pubs.opengroup.org/onlinepubs/ 9699919799/. (2016). [accessed 2017-06-14]. [12] Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav Har’El, Don Marti, and Vlad Zolotarov. 2014. OSv - Optimizing the Operating System for Virtual Machines. In Proceedings of the 2014 USENIX Annual Technical Conference. 61–72. [13] Kelly Lucas and developers. 1989. Byte-UnixBench. https://github. com/kdlucas/byte-unixbench. (1989). [accessed 2017-06-17]. [14] The Cloud Market. 2017. EC2 Statistics. http://thecloudmarket.com/ stats. (2017). [accessed 2017-06-01]. [15] MinGW.org. 2017. MinGW | Minimalist GNU for Windows. https: //www.mingw.org. (2017). [accessed 2017-06-14]. [16] Stack Overflow. 2016. Developer Survey Results. https://insights. stackoverflow.com/survey/2016. (2016). [accessed 2017-06-01]. [17] Udo Steinberg and Bernhard Kauer. 2010. NOVA: A Microhypervisor- Based Secure Virtualization Architecture. In Proceedings of the 5th European Conference on Computer Systems (EuroSys 2010). 209–222. https://doi.org/10.1145/1755913.1755935 [18] Phoronix Test Suite. 2017. Phoronix Test Suite. https://www. phoronix-test-suite.com/. (2017). [accessed 2017-06-17]. [19] Xiangyan Sun. 2015. Foreign LINUX - Run unmodified Linux applica- tions inside Windows. https://github.com/wishstudio/flinux. (2015). [accessed 2017-06-14]. [20] W3Techs. 2017. World Wide Web Technology Surveys. https://w3techs. com/. (2017). [accessed 2017-06-01].