POSIX Abstractions in Modern Operating Systems: The Old, the New, and the Missing Vaggelis Atlidakis, Jeremy Andrus, Roxana Geambasu, Dimitris Mitropoulos, and Jason Nieh fvatlidak, jeremya, roxana, dimitro, [email protected] Columbia University code atop the same rough abstractions, and educators can Abstract teach widely applicable abstractions in their OS courses. Since its creation over 25 years ago, POSIX has evolved The POSIX standard, developed 25 years ago, comprises to some extent (e.g., the most recent update was published in a set of operating system (OS) abstractions that aid appli- 2013 [55]), but the changes have been small overall. Mean- cation portability across UNIX-based OSes. While OSes while, applications and the computing platforms they run on and applications have evolved tremendously over the last have changed dramatically: Modern applications for today’s 25 years, POSIX, and the basic set of abstractions it pro- smartphones, desktop PCs, and tablets, interact with multi- vides, has remained largely unchanged. Little has been done ple layers of software frameworks and libraries implemented to measure how and to what extent traditional POSIX ab- atop the OS. Although POSIX continues to serve as the sin- stractions are being used in modern OSes, and whether new gle standardized interface between these software frame- abstractions are taking form, dethroning traditional ones. We works and the OS, little has been done to measure whether explore these questions through a study of POSIX usage POSIX abstractions are effective in supporting modern ap- in modern desktop and mobile OSes: Android, OS X, and plication workloads, or whether new, non-standard abstrac- Ubuntu. Our results show that new abstractions are taking tions are taking form, dethroning traditional ones. Such form, replacing several prominent traditional abstractions in measurements can be valuable to developers, educators, re- POSIX. While the changes are driven by common needs and searchers, and standards bodies alike, who can adapt their are conceptually similar across the three OSes, they are not applications, teachings, and optimization and standardiza- converging on any new standard, increasing fragmentation. tion efforts toward the new or changed abstractions. 1. Introduction We present the first study of POSIX usage in modern The Portable Operating System Interface (POSIX) is the OSes focusing on three of today’s most widely used mobile IEEE standard operating system (OS) service interface for and desktop OSes – Android, OS X, and Ubuntu – and UNIX-based systems. It describes a set of fundamental ab- popular consumer applications characteristic to these OSes. stractions needed for efficient construction of applications. We built a utility, called libtrack, that supports both dynamic Born out of work in the early 1980s, when the fragmentation and static analyses of POSIX use in applications. Dynamic of UNIX was of concern, it was created to enable applica- analyses give us detailed and precise POSIX usage patterns, tion developers to easily write application source code that but can only be run at limited scale (e.g., tens of popular would be portable across multiple diverse OSes. While per- applications for each OS). Static analyses let us generalize fect portability was never a reality, the level of uniformity trends at large scale (e.g., 1.1M applications in our Android added by POSIX has been valuable both for application de- study), but conclusions are not as precise. Our study sheds velopers and for educators alike. Application developers can light into a number of important questions regarding the use of POSIX abstractions in modern OSes, including: which abstractions work well; which appear to be used in ways for Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee which they were never intended; which are being replaced by provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. new and non-standard abstractions; and whether the standard Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, contact the Owner/Author(s). Request permissions from [email protected] or Publications Dept., ACM, Inc., fax +1 (212) is missing any fundamental abstractions needed by modern 869-0481. workloads. Our findings can be summarized as follows: EuroSys ’16, April 18–21, 2016, London, United Kingdom Copyright c 2016 held by owner/author(s). Publication rights licensed to ACM. First, usage is driven by high-level frameworks, which ACM 978-1-4503-4240-7/16/04. $15.00 DOI: http://dx.doi.org/10.1145/2901318.2901350 impacts POSIX’s portability goals. The original goal of the POSIX standard was application source code portability. This paper is organized as follows. Section2 presents two However, modern applications are no longer being writ- motivating examples for our study and formulates our goals. ten to standardized POSIX interfaces. Instead, they rely Sections3 and4 give background and detail our study’s on platform-specific frameworks and libraries that lever- methodology. Section5 presents our measurement results. age high-level abstractions for inter-process communication Sections6 and7 discuss the broader implications of our (IPC), thread pool management, relational databases, and findings in the context of related work and in general. graphics support. These frameworks and libraries are of- ten implemented using underlying standard POSIX APIs, 2. Motivation but are also free to depart from POSIX and use OS-specific interfaces. Thus, instead of the POSIX API serving as the Motivating Examples. Our measurement study is motivated interface between applications and the OS environment, by our experience building two very different systems – modern OSes – such as Ubuntu, Android, and OS X– pro- an Android data protection system called Pebbles [52] and vide a more layered programming model with “taller” in- an Android/iOS binary compatibility system, called Cycada terfaces. Applications directly link against high-level frame- (formerly known as Cider)[7] – whose designs exposed, and works, which invoke other frameworks and libraries that were drastically impacted by, the changes in the ways appli- may eventually utilize POSIX. This new, layered program- cations on these platforms are using system abstractions. ming model imposes challenges with respect to applica- • Pebbles [52] provides data protection at the level of tion portability, and has given rise to many different cross- application-level objects, such as emails or bank accounts. platform SDKs [2,9, 22, 24, 39] that attempt to fill the gap This level of protection is made possible by the new ways in left by a standard which has not evolved with the rest of which Android applications use storage abstractions. Rather the ecosystem. However, these cross-platform SDKs are of- than using traditional unstructured POSIX file system ab- ten challenging to develop and maintain up-to-date with OS stractions, Android applications instead store data almost changes. exclusively in highly structured storage (SQLite). In Peb- Second, extension APIs, namely ioctl, dominate mod- bles, we leverage this structured information to transparently ern POSIX usage patterns as OS developers resort to them and accurately reconstruct application-level objects so the to build support for abstractions missing from the POSIX OS can provide protection at their level. Without applica- standard. Extension APIs have become the standard way tions’ almost exclusive reliance on structured storage, Peb- for developers to circumvent POSIX limitations and facili- bles would likely not be possible. tate hardware-supported, high-level functionality for graph- • Cycada [7] is a binary compatibility framework for An- ics, sound, and IPC. For example, the ioctl interface is now droid and iOS applications. Given that both Android and regularly used to mediate complex graphics commands be- iOS implement similar POSIX functionality, we initially tween the high-level OpenGL library and the graphics driver. thought that building Cycada would be relatively straight- Third, new abstractions are arising driven by the same forward compared to previous Windows-UNIX compatibil- POSIX limitations across the three OSes, but the new ab- ity efforts. However, achieving compatibility even between stractions are not converging. To deal with abstractions Android and iOS turned out to be a herculean task. A main missing from the aging POSIX standard, modern OSes are obstacle was the extensive use of POSIX’s ioctl exten- implementing new abstractions to support higher-level ap- sion API, which is highly platform-specific and loosely de- plication functionality. Although these interfaces and ab- fined. To address this challenge, we elevated the level of stractions are driven by similar POSIX limitations and are abstraction at which we constructed binary compatibility conceptually similar across OSes, they are not converging from POSIX to newer, high-level abstractions used by ap- on any new standard. Traditional POSIX threading models, plications, such as graphics and sound libraries. With this IPC interfaces, and file system access are being replaced by approach, and intuitively assuming that most applications platform and vendor-specific APIs and