MMU-based Access Control for Libraries Marinos Tsantekidis1 a and Vassilis Prevelakis2 1Institute of Computer Science - FORTH, Greece 2AEGIS IT RESEARCH GmbH, Germany Keywords: Runtime Monitoring, Library Calls, Access Control Gates, Security Policies. Abstract: Code Reuse Attacks can trick the CPU into performing some actions not originally intended by the running program. This is due to the fact that the execution can move anywhere within a process’s executable memory area, as well as the absence of policy checks when a transfer is performed. In our effort to defend against this type of attacks, in an earlier paper we present a Proof-of-Concept mitigation technique based on a modified Linux kernel where each library - either dynamically or statically linked - constitutes a separate code region. The idea behind this technique is to compartmentalize memory in order to control access to the different mem- ory segments, through a gate. Taking our previous work one step further, in this paper we present an updated version of our kernel-side technique, where we implement security policies in order to identify suspicious behavior and take some action accordingly. 1 INTRODUCTION or replace strings), or return an error without mak- ing the call. A key concern is the level at which the As the advancement of technology offers capabilities call intercept should be made. Some papers propose which result in attackers on every level getting more to carry out the checks at the machine code level - competent and effective, attacks have become more via call graphs (e.g., CFI (Abadi et al., 2005) and elaborate. Therefore, we need to establish an ade- variants), while others at the system call level (e.g., quate level of security in software systems. As the systrace(8)). However, because these systems per- complexity of these systems is ever-increasing, we form the checks at fairly low level, they are quite labor have not seen a commensurate improvement in code intensive and error prone and exhibit high overhead. design and development. Hence, systems are becom- Coarse-grained CFI techniques mainly try to deal ing ever more vulnerable to attacks at multiple lev- with the increased performance issues of CFI, trad- els. Complete security of a program is unfeasible. ing off security and being left vulnerable to motivated Conceding that vulnerable code will be included in adversaries. Finer-grained solutions provide higher production systems, there is a need to detect when security, although even in this case studies (Evans one or more of these vulnerabilities is exploited by et al., 2015) have shown that some implementations an attacker. By monitoring the behavior of a pro- prove ineffective in defending against code-reuse at- gram, we can detect any deviations from nominal op- tacks (CRAs). eration. Once a program goes off-nominal we need to determine whether the cause is security-related and, if Contributions. In this paper, we present our hy- so, take appropriate action. Our idea is to implement pothesis that by carrying out behavioral monitoring such actions at an abstract level, between the Operat- at a higher level, we can create a flow diagram that ing System (OS) and a running application. is closer to the program logic and hence more under- Unlike most run-time security mechanisms, where standable. We have chosen the library function call violations in most cases lead to the termination of level as the ideal boundary to install our checks, be- the offending process, the call intercept technique cause libraries are used by the main program but are offers a variety of options in dealing with the se- not part of its execution logic. We present an updated curity breach. For example, systrace(8) (Provos, version of our previous Proof-of-Concept mitigation 2003) may rewrite function arguments (e.g. truncate technique, behind which the idea is to compartmen- a https://orcid.org/0000-0001-6710-5972 talize memory in order to provide access control to 686 Tsantekidis, M. and Prevelakis, V. MMU-based Access Control for Libraries. DOI: 10.5220/0010536706860691 In Proceedings of the 18th International Conference on Security and Cryptography (SECRYPT 2021), pages 686-691 ISBN: 978-989-758-524-1 Copyright c 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved MMU-based Access Control for Libraries memory segments based on security policies. We ex- gadgets to perform the write(2) system call and then tend our previous approach by installing a special cus- transfer the application’s binary from memory to the tom library, a gate, one for each separate segment and attacker’s socket. redirect the flow of execution through this gate where Over the years, important work has been carried we enforce the relevant security policies. Our mitiga- out with respect to defenses against CRAs. Backes tion technique requires kernel support, which is on of and Nurnberger¨ developed Oxymoron (Backes and its strengths since we can manipulate the memory se- Nurnberger,¨ 2014) to counter JIT-ROP attacks. Oxy- curely and efficiently. We chose to leverage the Mem- moron combines two methods: fine-grained memory ory Management Unit (MMU) and specifically the randomization with the ability to share the entire code NX-bit, because it is easier to us due to our previous among other processes. However, in order for this development experience. To the best of our knowl- defense to be effective, an executable along with all edge, it is the first time this is performed in this way. of its shared libraries need to be instrumented us- Furthermore, our mechanism can be implemented on ing Oxymoron (Mithra and Vipin P., 2015). In con- top of other defense mechanisms. trast, our approach is completely transparent, since The remainder of this paper is organized in the we require no access to the executable or its shared following manner: Section 2 offers a brief summary libraries. of the timeline of CRAs and related defenses. In Venkat, et al. (Venkat et al., 2016) propose a se- Section 3 we detail the design of our approach. In curity defense called HIPStR to thwart (JIT-)ROP. Section 4, we describe the implementation specifics. HIPStR performs dynamic randomization of run-time In Section 5, we evaluate our custom kernel both in program state. However, this measure requires ex- terms of effectiveness and performance. Discussion tensive run-time support to piece the randomized in- and conclusion are provided in Section 6. structions back together, imposing significant over- head (around 15% at best), counter to our mechanism which leaves only a minimal performance footprint. 2 BACKGROUND AND RELATED The systrace(8) (Provos, 2003) system which supports fine-grained policy generation, guards the WORK calls to the operating system at the lowest level, en- forcing policies that restrict the actions an attacker Code Reuse Attacks (CRA) work on the premise that can take to compromise a system. However, this an adversary can use code already present in a pro- fine-grain control is overly verbose. Additionally, it gram’s memory space in order to mislead the CPU may leave a library in an inconsistent state if a mis- to perform unintended by the program actions. So- configuration interrupts the sequence of these calls in phisticated approaches, such as Return Oriented Pro- the middle of execution (Kim and Prevelakis, 2006). gramming (ROP) (Shacham, 2007; Checkoway et al., Later research (Watson, 2007) showed that concur- 2010; Roemer et al., 2012) and Jump Oriented Pro- rency vulnerabilities were discovered, that gave an at- gramming (JOP) (Bletsch et al., 2011), form snippets tacker the ability to construct a race between the en- of code, “gadgets”, chaining together legitimate com- gine and a malicious process to bypass protections. mands already in memory, eventually forming a se- Our technique operates at a higher level, away from quence of gadgets that perform the unauthorized ac- low level calls, closer to the application logic, fa- tion desired by the attacker. Initially, CRAs were cilitating the generation of policies and being more based on the principle that gadgets are located at lightweight in this way. known addresses in memory. Address Space Lay- Access Controls For Libraries and Modules (Sec- out Randomization (ASLR) (PaX, 2001) was intro- Module) (Kim and Prevelakis, 2006) forces user-level duced as a defense measure, but was eventually by- code to perform library calls only via a library pol- passed (Shacham et al., 2004). icy enforcement engine providing mandatory policy Furthermore, Snow et al. introduced “Just-In- checks over not just system calls, as in the case of Time” ROP (JIT-ROP) (Snow et al., 2013) which systrace(8), but calls to user-level libraries as well. maps an application’s memory layout on-the-fly, thus Access to a specific function or procedure is con- bypassing ASLR. Next, it constructs and delivers a trolled by the kernel, however the overhead of two ROP payload based on collected gadgets. context switches per function invocation (caller to Later, Bittau et al. presented Blind Return Ori- kernel and kernel to caller), makes the technique quite ented Programming (BROP) (Bittau et al., 2014). expensive for more general use. One of the issues BROP works against modern 64-bit Linux with identified by the authors of the SecModule paper is ASLR, NX memory and stack canaries (Wagle and the difficulty in encapsulating library modules which Cowan, 2003) enabled. It can remotely find enough 687 SECRYPT 2021 - 18th International Conference on Security and Cryptography was error prone and extremely labor intensive. An- code of any application, making the mechanism suit- other issue is the inability to evaluate call arguments. able for legacy applications as well as binary pro- Although they are contained in a known structure grams.