secuBT : Hacking the Hackers with User-Space Virtualization Mathias Payer Department of Computer Science ETH Zurich Abstract Two interesting scenarios are (i) sandboxing of server processes, and (ii) execution of untrusted In the age of coordinated malware distribution and code. Server daemons that offer an interface to zero-day exploits security becomes ever more im- the Internet are under constant attack. If these portant. This paper presents secuBT, a safe ex- daemons are virtualized and restricted to only spe- ecution framework for the execution of untrusted cific system calls and parameters then an exploit binary code based on the fastBT dynamic binary is unable to escalate privileges and execute unin- translator. tended code. The second scenario targets plugins secuBT implements user-space virtualization us- and other downloaded untrusted code that a user ing dynamic binary translation and adds a system would like to run in a special sandbox. Whenever call interposition framework to limit and guard the the program issues a system call, the user can de- interoperability of binary code with the kernel. cide if it is allowed or not. The sandbox limits the Fast binary translation is a key component to possible interactions between the virtualized pro- user-space virtualization. secuBT uses and ex- gram and the operating system. System calls are tends fastBT, a generator for low-overhead, table- checked depending on name, parameters, and call based dynamic (just-in-time) binary translators. location. We discuss the most challenging sources of over- Such a sandbox could be implemented as a ker- head and propose optimizations to further reduce nel module. This would require additional code these penalties. We argue for hardening techniques that runs in the kernel itself which poses an addi- to ensure that the translated program cannot es- tional security risk. User-space virtualization uses cape out of the user-space virtualization. binary translation to sandbox processes. The virtu- An important feature of secuBT is that only alization layer controls all instructions that can be translated code is executed. This ensures code va- executed by the virtualized process. System calls lidity and makes it possible to rewrite individual are rewritten by the translation system so that an instructions. The system call interposition frame- authorization function is executed first. This au- work validates every system call and offers the thorization decides whether a system call (i) is al- choice to (i) allow it, (ii) abort the program, (iii) lowed, (ii) aborted, or (iii) redirected to a sandbox- redirect to an user-space emulation. internal function that emulates the system call in user-space. To the virtualized application it ap- pears as if the system call is executed directly. 1 Introduction We propose a sandbox that is built on a dy- namic binary translation (BT) system. Using BT In a world of increasing software complexity and ensures that all executed code is translated first, diversity it is important to sandbox and virtual- and therefore a program cannot escape the sand- ize running applications. Staying up-to-date on all box. If the translator encounters unsafe or invalid software applications and libraries is hard but nec- code it aborts the program. This builds a safe essary. On the other hand patches as well as mal- execution framework that validates executed code. ware discovery is reactive. secuBT is a step towards The translator can rewrite individual instructions, proactive security and fault detection. e.g. redirect system calls to handler functions. 1 Section 2 covers the design of the encapsulation original program and the code cache. This transla- framework and implementation of the binary trans- tion process ensures that the execution flow always lator fastBT [14]. Section 3 elaborates the security stays in the code cache. If the program is about to extension and the system call interposition frame- branch to an untranslated block of code then the work. The system is evaluated in Section 4, fol- translator is invoked to translate that block. The lowed by related work and our conclusion. translated block is then placed in the code cache and the execution of the program resumes at the newly translated block in the code cache. See Fig- 2 fastBT: Dynamic Binary ure 1 for an overview of such a generic translator. Translation Original program Translator Code cache Fast binary translation is the key component to im- Opcode table plement user-space virtualization. The dynamic 0 1' translation system checks and verifies every ma- 1 chine code instruction before it is executed. Static 3' Trampoline to binary translation does not suffice because it is un- translate 4 2 3 2' able to detect hidden code or malicious code that Mapping targets the static binary translator itself. Direct 3 3' control transfers are translated and redirected to 4 1 1' the code cache. Indirect control transfers are trans- 2 2' lated into an online lookup and dispatch to guaran- tee that only translated branch targets are reached. Figure 1: Runtime layout of the binary translator. The translation system can change, adapt, or re- move any invalid instruction. Using the translation system, system calls are rewritten and redirected to an interposition system. 2.1.1 Translation Tables To limit the overhead of binary translation the translation process must be fast and must produce The basic translation engine is a simple table-based competitive code. This section presents implemen- iterator. When the translator is invoked with a tation details of fastBT [14], a fast and flexible bi- pointer to a basic block, it first adds a reference to nary translator that is used to implement the exe- the mapping table from the original program loca- cutable space protection and system call interposi- tion to the location in the code cache. The iterator tion. An important feature of fastBT is that the then loops through the basic block one instruction return addresses on the stack remain unchanged. at a time. This adds additional complexity to handle return To decode the variable-length x86 instructions instructions as they are translated to a lookup and a large multidimensional translation table is con- an indirect control transfer. An unchanged stack structed that encodes all possible combinations of has the following advantages: (i) the program can machine code instructions and parameters. Start- validate and read its own call stack for debugging, ing with a base table each byte of the current in- (ii) exception handling uses return addresses to es- struction is checked. If the decoding of the instruc- cape to the correct frame, (iii) the program does tion is not finished in the current table, a pointer not know that it is translated, and (iv) the address redirects to the next table and the decoding pro- of the code cache is hidden from the program. cess continues with the next byte. This process is repeated until the instruction is completely de- 2.1 Basic Translator coded. As a next step the instruction and its arguments The translator processes basic blocks of the original are passed to a corresponding action function that program, places them in the code cache and adds handles the translation of this particular instruc- entries to the mapping table. The mapping table tion. Action functions can generate arbitrary code, is used to map between program locations in the alter, copy, or remove the instruction. Generated 2 code is emitted into the code cache. Therefore it makes sense to keep translated code in The translation process stops at recognizable ba- a cache for later reuse. As a result code is only sic block (BB) boundaries like branches or return translated once, reducing the overall translation instructions. Some BB boundaries like backward overhead. jumps are not recognizable, in such a case a part of fastBT uses a per thread cache strategy to in- the BB will be translated a second time. crease code locality. An additional advantage of At the end of a BB the translator checks if the thread local caches is that the action functions can outgoing edges are already translated, and adds emit hard-coded pointers to thread local data struc- jumps to the translated targets. If a target is not tures, otherwise the code would need to call expen- translated, the translator builds a trampoline that sive lookup functions. starts the translation engine for the corresponding The combination of the basic translator that BB, and adds a jump to the trampoline. stops at BB boundaries and the design of the code cache lead to a greedy trace extraction. These 2.1.2 Predefined Actions traces are formed as a side effect of the first ex- ecution and can possibly speed up the program. The translator needs different action functions to support the identity transformation. All safe in- structions that are executable in user-space are 2.1.4 Mapping Table copied to the code cache. Privileged instructions The mapping table maps between code locations in (e.g. all instructions that are not allowed to exe- the original program and code locations in the code cute in user-space) are intercepted and the program cache. The translator adds entries for all translated is aborted, otherwise the kernel would signal a seg- basic blocks. An entry consists of two pointers, the mentation fault or general protection fault. first pointing to the original location, the second Special care is needed for control instructions. pointing to the translated location. These are handled by specific action functions that The hash function used in any lookup function check the target of the branch instruction. These returns a relative offset into the mapping table. action functions emit extra code that gives the il- Adding the result from the hash function to the lusion as if the original code was executed.