Digital Investigation 14 (2015) S16eS24 Contents lists available at ScienceDirect Digital Investigation journal homepage: www.elsevier.com/locate/diin DFRWS 2015 US The impact of GPU-assisted malware on memory forensics: A case study * Davide Balzarotti a, Roberto Di Pietro b, 1, Antonio Villani c, a Eurecom, Sophia Antipolis, France b Bell Labs, Paris, France c Department of Mathematics and Physics, University of Roma Tre, Rome, Italy abstract Keywords: In this paper we assess the impact of GPU-assisted malware on memory forensics. In Graphic processing units particular, we first introduce four different techniques that malware can adopt to hide its Memory analysis presence. We then present a case study on a very popular family of Intel GPUs, and we Malware analyze in which cases the forensic analysis can be performed using only the host's Digital Forensics memory and in which cases it requires access to the GPU's memory. Our analysis shows Direct Rendering Manager that, by offloading some computation to the GPUs, it is possible to successfully hide some malicious behavior. Furthermore, we provide suggestions and insights about which arti- facts could be used to detect the presence of GPU-assisted malware. © 2015 The Authors. Published by Elsevier Ltd on behalf of DFRWS. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Introduction the best of our knowledge, no GPU-assisted malware has been discovered in the wild e with the sole exception of For more than twenty years computers have relied on BadMiner (ArsTechnica, 2015) which exploits the presence dedicated Graphics Processing Units (GPUs) to perform of a GPU to mine bitcoins. However, researchers have graphical computations and rendering. More recently, with already shown that several malicious activities can take the advent of general-purpose computing on graphics advantage of GPUs, for instance to steal sensitive infor- processing units (GPGPU), GPUs have been increasingly mation (Kotcher et al., 2013; Lee et al., 2014), unpack ma- adopted also to perform other generic tasks. In fact, thanks licious code (Vasiliadis et al., Oct 2010), and hide malicious to their many-core architecture, GPUs can provide a sig- activities from malware detection and analysis tools nificant speed-up for several applications e such as finan- (Triulzi, 2008; Ladakis et al., 2013). It is important to notice cial and scientific computations, regular expression that these threats are not confined to the graphic envi- matching, bitcoins mining, video transcoding, and pass- ronment (e.g., screen grabbing or unrestricted access to word-cracking. GPU memory) but that GPUs can also be abused to perform Despite their pervasiveness and their ability to perform attacks that are outside the graphic domain. In fact, oppo- generic computations, the role and impact of GPUs to site to others commonly available PCI devices, the fact that perform malicious activities is still largely understudied. To GPUs can be programmed by userspace applications makes them a very attractive attack vector. As a matter of fact, also other peripherals (such as hard disks, network devices, printers) have been abused to perform malicious activities * Corresponding author. E-mail addresses: [email protected] (D. Balzarotti), (Zaddach et al., 2013; Sparks et al., 2009; Cui et al., 2013), [email protected] (R. Di Pietro), villani@mat. but those scenarios required to introduce custom modifi- uniroma3.it (A. Villani). cations to the device firmware. GPUs are instead meant to 1 He is also with the Department of Maths, University of Padua, Padua, be programmed by the end-user and require no firmware Italy. http://dx.doi.org/10.1016/j.diin.2015.05.010 1742-2876/© 2015 The Authors. Published by Elsevier Ltd on behalf of DFRWS. This is an open access article under the CC BY-NC-ND license (http:// creativecommons.org/licenses/by-nc-nd/4.0/). D. Balzarotti et al. / Digital Investigation 14 (2015) S16eS24 S17 modification to execute arbitrary code. To make things worse, graphic cards vendors pay more attention to the performance of their products than to their security. Un- fortunately, this has often a negative impact on several security mechanisms, such as the memory isolation be- tween independent process running on the same GPU (Di Pietro et al., 2013; Maurice et al., 2014). From a defense perspective, the main problem is that neither antivirus softwares nor memory forensics tools are currently able to analyze the content of the GPU memory. However, it is not clear how severe this limitation is in practice, and what is the real level of stealthiness that can be achieved by GPU-assisted malware. In particular, the main goal of this paper is to understand what is the impact of GPU-assisted malware on memory forensics. Are traditional collection and analysis techniques sufficient to detect and fi fully understand the malicious behavior? This is a funda- Fig. 1. A simpli ed view of a GPU-based heterogeneous architecture. The GPUs have access both to a dedicated device memory and to the host mental question for the computer forensic domain. Nowa- memory. days, memory forensics tools have an incomplete view of the system, limited only to the content of the operating system memory. It is very important to understand under introduce four possible attack scenarios and discuss which conditions this piece of information is enough to their impact for a forensic analyst. For each scenario we detect the presence of a malicious activity (even if the implement a different proof-of-concept for Linux and malware relies to a certain extent on the use of the GPU). we report under what conditions the memory analysis Only recently GPUs gained the attention of the digital can be performed through the kernel data structures, forensic community and a common assumption in the cur- the video driver or the GPU memory rent research (Richard, 2015) is that GPU-assisted malware We present a detailed case study on Intel integrated needs at least a component running in the system memory. GPUs, which are part of all modern processors. Our re- If so, the analysis of this component could be sufficient to sults show that an adversary can take advantage of these detect that the malware is using the GPU and maybe even to integrated GPUs to obtain full access to any memory retrieve the purpose of the outsourced computation. page in the host. Even worse, we show how it is possible However, in this paper we show that in certain condi- to obtain a persistent malicious code running on the tions it is possible for an attacker to leave no trace in the OS GPU without any associated process running on the memory. In this case, malware coders have a substantial host. advantage that could only be limited by developing custom techniques to acquire and analyze the internal GPU mem- ory of each vendor (as well as graphic card model, and Background device driver). Unfortunately, this can be a very difficult e and time-demanding task especially for proprietary Modern GPUs are specialized massively parallel pro- products for which no documentation exists about their cessors with hundreds of computational units. In fact, internal data structures. while traditional CPUs dedicate most of their die area to Our goal is to answer a number of very practical ques- cache memory, GPUs dedicate it to logic circuits. The tions. For instance, is it possible for an analyst to identify combination of the two processing units, which takes which processes are using the GPU and which code is being advantage of the CPU for general applications tasks and of executed there? To answer the previous question, is it the GPU for highly parallel computation, is today the most enough to analyze an image of the system memory, or does common example of a heterogeneous architecture (see the analyst need also to inspect the regions memory- Fig. 1 for a simplified example). The host, namely the logical mapped to the video card (potentially containing pro- element containing the CPU, has its own memory and co- prietary data structures) or, worst case, does the internal ordinates the execution of one or more GPUs in a master- memory of the video card also need to be collected and slave configuration. The GPUs, which also have their own analyzed? Which of these questions can be answered by local memory, consist of several identical computational only looking at the system memory? Can the GPU be used units, each containing the same amount of processing el- to implement anti-forensic techniques? To address these ements. A task which is executed on a GPU is called kernel. questions, we designed and implemented a number of tests The GPU drivers manages kernels through the means of and experiments, based on a popular family of Intel GPUs. contexts, the equivalent of a process control block in the To summarize, this paper makes the following GPU.2 A key aspect of this architecture is that the GPU is contributions: also able to directly access arbitrary pages on the host We conduct the first study of the impact of GPU-assisted malware on memory forensics. In particular, we 2 To disambiguate between the operating system kernel and the GPU kernel, the second one will be typed in italic. S18 D. Balzarotti et al. / Digital Investigation 14 (2015) S16eS24 memory. One of the motivation for this is that, in order to scenario, and a more general approach (independent from coordinate the executions of tasks on the GPU, the CPU the device that has been compromised) is needed to handle typically writes commands into a command buffer that is these powerful cases.