Temporal System Call Specialization for Attack Surface Reduction Seyedhamed Ghavamnia, Tapti Palit, Shachee Mishra, and Michalis Polychronakis, Stony Brook University https://www.usenix.org/conference/usenixsecurity20/presentation/ghavamnia This paper is included in the Proceedings of the 29th USENIX Security Symposium. August 12–14, 2020 978-1-939133-17-5 Open access to the Proceedings of the 29th USENIX Security Symposium is sponsored by USENIX. Temporal System Call Specialization for Attack Surface Reduction Seyedhamed Ghavamnia, Tapti Palit, Shachee Mishra, Michalis Polychronakis Stony Brook University Abstract during the lifetime of the program. This is especially true for Attack surface reduction through the removal of unnecessary server applications, which once launched, remain running and application features and code is a promising technique for serving requests for a long period of time. This means that improving security without incurring any additional overhead. all kernel capabilities (i.e., system calls) remain available to Recent software debloating techniques consider an applica- a potentially vulnerable process, and can thus be used as part tion’s entire lifetime when extracting its code requirements, of exploitation attempts. and reduce the attack surface accordingly. Software debloating and specialization has recently gained In this paper, we present temporal specialization, a novel popularity as a technique for removing or constraining un- approach for limiting the set of system calls available to a used parts of applications, with the goal of reducing the process depending on its phase of execution. Our approach code and features available to attackers. While some ap- is tailored to server applications, which exhibit distinct ini- proaches use static analysis to identify unused parts of shared tialization and serving phases with different system call re- libraries [12,51], others rely on dynamic analysis and train- quirements. We present novel static analysis techniques for ing to identify unneeded parts of the application [13, 21,48]. improving the precision of extracting the application’s call Similar techniques have also been applied on containers to graph for each execution phase, which is then used to pinpoint constrain the set of system calls available to the hosted pro- the system calls used in each phase. We show that require- grams [22, 38, 59]. A key shared characteristic of the above ments change throughout the lifetime of servers, and many approaches is that they consider the entire lifetime of a pro- dangerous system calls (such as execve) can be disabled gram as part of the scope of their analysis. after the completion of the initialization phase. We have im- In this paper, we explore software specialization from a plemented a prototype of temporal specialization on top of the different perspective, and present temporal system call spe- LLVM compiler, and evaluated its effectiveness with six pop- cialization, a novel attack surface reduction approach for lim- ular server applications. Our results show that it disables 51% iting even further the set of system calls that are available more security-critical system calls compared to existing li- to a process, depending on its phase of execution. Instead of brary specialization approaches, while offering the additional treating each application as a single, monolithic entity with benefit of neutralizing 13 more Linux kernel vulnerabilities an unchanging set of requirements, temporal specialization that could lead to privilege escalation. takes into consideration the changes in an application’s re- quirements throughout its execution lifetime. In particular, 1 Introduction we focus on server applications, which typically exhibit two distinct initialization and serving phases. Modern software is complex. Applications typically support Our main motivation is that many dangerous system calls, a wide range of functionalities for different use cases [28,49], such as execve, which are frequently used as part of exploit as evidenced by the existence of multiple features, options, code, are often not removed by existing code debloating and and configuration settings. To support these different features, specialization techniques, because they are required by the programs typically require access to a vast range of privileged application for legitimate purposes. Crucially, however, oper- operations from the OS kernel (e.g., allocating memory, creat- ations such as spawning new processes or creating listening ing new processes, and accessing files or the network), which sockets are typically only performed during the very first are made available through the system call interface. moments of a server’s lifetime—the initialization phase. Tem- Some of these capabilities, however, are used by the ap- poral specialization automatically derives the set of system plication only once during startup, and are never used again calls required by each execution phase, and restricts the set of USENIX Association 29th USENIX Security Symposium 1749 available system calls once the server enters its stable serv- 2 Background and Motivation ing phase. This significantly reduces the set of system calls available to an attacker. User-space applications rely on the system call API to inter- A crucial requirement for pinpointing the system calls act with the OS. The Linux kernel v4.15 used in this work required in each phase is to construct a sound and precise provides 333 system calls, while its latest version 5.6 (as of call graph. As most server applications are developed using June 2020) provides 349. Applications, however, typically C/C++, which support indirect function invocations, we must rely only on a subset of these system calls for their opera- rely on static code analysis to resolve the possible targets tion. Moreover, their requirements change according to the of indirect call sites. Unfortunately, the state-of-the-art im- phase of execution, e.g., whether the application is being ini- plementations of points-to analysis algorithms suffer from tialized or serving requests. From a security perspective, this severe imprecision and overapproximation, which eventually overabundance of system calls allows an attacker to i) use results in the inclusion of many spurious system calls that the additional system calls to carry out malicious operations are not actually used. To address this challenge, we propose as part of exploiting a vulnerability, and ii) exploit underly- two pruning mechanisms that remove spurious edges from ing kernel vulnerabilities triggered through system calls and the derived call graph, significantly improving its precision achieve privilege escalation [22, 31, 32]. while retaining its soundness. After identifying the system calls needed in each phase, we use Seccomp BPF to block any system calls that are not needed anymore after the completion 2.1 Static vs. Temporal API Specialization of the initialization phase, thereby removing them from the Previous works in attack surface reduction [21, 26, 34, 48, 50] process’ attack surface. consider the entire application lifetime, and remove function- We implemented a prototype of temporal specialization ality that will never be used at any point. When considering for Linux on top of LLVM, and evaluated it with six popular the execution phases of typical server applications, however, applications (Nginx, Apache Httpd, Lighttpd, Bind, Mem- we observe that further specialization can be achieved. cached, and Redis). We show that many dangerous system In particular, servers typically start handling client requests calls, such as execve, can be disabled after the application after performing a series of one-time operations for setting up enters its serving phase, i.e., when the server application starts the process. This initialization phase mainly consists of op- handling client requests and becomes susceptible to attacks. erations such as parsing configuration files, binding network Our results show that temporal specialization disables 51% ports, and forking worker processes. After the completion more security-critical system calls compared to existing li- of these tasks, the server enters its main long-term serving brary specialization approaches [12, 51], while in many cases phase for handling client requests. In this stable state, the it does not leave room for evasion using alternative system server typically performs operations such as reading from and call combinations. As an added benefit, 53 Linux kernel vul- writing to sockets or files, managing memory, and allocating nerabilities are neutralized by removing system calls which tasks to the worker processes. Nginx [7] is an example of a serve as entry points for triggering them, 13 of which are not server which exhibits this behavior. Depending on whether it preventable by library specialization. is started in “single-process” or “multi-process” mode, Nginx Our work makes the following main contributions: either executes the function ngx_single_process_cycle, or forks the configured number of worker processes, each 1. We propose a novel temporal system call specialization of which invokes the function ngx_worker_process_cycle. approach that considers the different operational charac- Both functions mark the beginning of the serving phase by teristics of server applications throughout their different entering an infinite loop that processes client requests. execution phases. The operations performed in these two phases are distinc- 2. We present type-based