Outsmarted - Why Malware Works in face of Antivirus Software Matthias Deeg, Sebastian Nerz, Daniel Sauder Outsmarted – Why Malware Works in face of Antivirus Software For many years, different types of malware rank among the biggest IT security threats both in the business and the private domain. In order to protect one- self from the dangers of malware, numerous software manufacturers offer IT security products like antivirus and endpoint protection software. But these products alone offer no sufficient protection from malware that knows some tricks, as the results of our recent research with the topic antivirus evasion show. n the recent past, there were several com- from antivirus software. Thereby, current results puter-based attacks against IT networks that of our recent research are presented and recom- became public and raised a lot of media at- mendations are given for dealing with threats and Itention. Especially the attacks against the New security risks caused by malware. York Times [1] and the Washington Post [2] at the beginning of 2013 had a world-wide media How Antivirus Software Works coverage and also heated the debate about such Current antivirus software, no matter if a stand- cyber threats with manufacturers of IT security alone software product or a component of a soft- products like antivirus and endpoint protection ware suite (host intrusion detection/prevention software. In both mentioned cases, attackers were software, endpoint protection software, etc.), uses able to install malware on computer systems of different methods to detect known and unknown employees in order to literally spy on the affect- threats by means of malware. ed companies – and this probably undetected for In general, these methods used for protecting sev eral months. computer systems from unwanted, malicious Once more, incidences like these have pointed software can be assigned to the following two out that in spite of the use of IT security prod- strategies: ucts like antivirus software or host intrusion de- tection/prevention software (HIDS/HIPS) such 1. Blacklisting attacks cannot be entirely prevented. This kind 2. Whitelisting of threat illustrates that enterprises and also gov- ernment agencies require a master plan with a In the context of antivirus software, the two working information security management and terms blacklisting and whitelisting simply mean security awareness of all employees. that the execution of a program is either explic- itly forbidden (being on a black list) or explicitly This paper discusses how developers of malware allowed (being on a white list). Thus, by following like trojan horses (in short trojans), viruses, and the blacklisting approach antivirus software worms proceed to hide their malicious intentions will prevent the execution of programs that are SySS GmbH | August 2014 Page 1|8 Outsmarted - Why Malware Works in face of Antivirus Software software and are added to their signature data- Exploiting base. Hence, with this method it is only possible Antivirus software does not offer any pro- to detect malware for which at least one corre- tection against attacks of vulnerable network sponding signature exists. services, for example, an outdated web server. The manner in which manufacturers of antivirus Because in such attacks, malicious code, so- software create signatures for malware and search called shellcode, is directly loaded into the for the specific patterns of course influences the main memory of the affected system and ex- error rate of signature-based malware detection ecuted there, for example by exploiting a buf- (error type 1 [false positive], error type 2 [false fer overflow vulnerability. Thus, there is no negative]). file containing the malicious code within the file system of the target system that could be Malware authors and users exploit this situa- found by the usual malware detection mecha- tion by developing and modifying malware in nisms of antivirus software. such a way that it does not contain any known signatures that are sufficient for classifying it as malware. Thus, this kind of malware cannot be not found on a black list. In contrast, using the detected by solely signature-based malware de- whitelisting approach antivirus software will only tection mechanisms. For bypassing some sig- allow the execution of programs found on a white nature-based malware detections it is sufficient list. These two strategies help to pursue the goal to compile the malware with different compiler allowing only desired, expected, and benign be- settings or with a different compiler (malware havior and preventing unwanted, unexpected, source code is available), or to compress and/or and malicious behavior. encrypt the executable file (malware source code The adminstration of black and white lists can is not available) using so-called EXE packers or either be the responsibility of the antivirus soft- EXE cryptors. In this way, the original function- ware, which is the default case for the blacklisting ality of known malware can be preserved but its strategy, or it can be the responsibility of the end form has changed. users or administrators, which is primarily the Thus, polymorphism is a big challenge for signa- case for the whitelisting strategy. ture-based detection mechanisms and is used by The majority of used antivirus software only fol- malware for a long time. lows the blacklisting strategy and thus tries to detect whether software is malicious and should Behavior-Based Malware Detection therefore not be executed. Behavior-based malware detection tries to deter- For malware detection there are generally the fol- mine the behavior of software and to classify it lowing two methods which are described in more according to defined criteria as benign or mali- detail in the following sections: cious. For this, usually rule-based techniques are used in combination with a scoring system and 1. Signature-based specified thresholds for calculated scores. This 2. Behavior-based detection method, also known as heuristic pro- cedure, is capable of detecting unknown malware Signature-Based Malware Detection due to its behavior. The used scoring system and By using signature-based malware detection, an- threshold values have a major influence on the tivirus software is searching for known patterns error rate of this generic method (error type I in the form of byte sequences (also called signa- [false positive], error type II [false negative]). The tures) in files that allow to identify specific mal- analysis of a software program can be performed ware. A disadvantage of this methodology is that only statically at which the code of the executable only known patterns can be searched for. These file is analyzed in a so-called static code analysis patterns are created by the manufacturers of an- and corresponding software behavior is deter- tivirus software based on analyses of malicious mined by means of the presence of specific fea- SySS GmbH | August 2014 Page 2|8 Outsmarted - Why Malware Works in face of Antivirus Software tures. However, a disadvantage of this approach Research Project: Antivirus Evasion is that only program code can be analyzed that is In the course of our research project with the top- directly accessible to the antivirus software. Thus, ic antivirus evasion, the authors have tested dif- program code that is only available during run- ferent antivirus evasion techniques with several time respectively detectable as code, for example well-known and widely used antivirus software due to compression, encryption or self-modi- products that follow the blacklisting strategy. fying code, is not considered for the static code The applied techniques are not new, they have analysis. been used by malware for years and exploit the abovementioned weaknesses in signature- and For this reason, most of the antivirus software behavior-based detection methods of antivirus uses a so-called sandbox environment besides a software. static code analysis. In the context of antivirus software, a sandbox means a secure and control- lable execution environment in which the behav- Characteristics of Antivirus Software: Up- ior of an executable file can be analyzed during dates and Scan Engines runtime and be classified as benign or malicious During the conducted antivirus evasion tests due to defined criteria. In this way, the effective- of this research project and during penetra- ness of some malware obfuscation methods, for tion tests, the SySS GmbH noticed on several instance based on compression or encryption, occasions that the behavior of scan engines can be reduced. Furthermore, another goal of of tested antivirus software products varied antivirus sandbox environments is to hide their in different environments. For example, if a presence and not to be detectable by malware. malicious file is not detected by the locally in- This prevents malware from adjusting its behav- stalled antivirus software, nevertheless, it can ior according to the discovered execution envi- be rated as malicious by the assumed identical ronment and thus malicious code is not executed scan engine when uploaded to VirusTotal. The within a sandbox. In general, the emulation of an reason for this different behavior one should execution environment is a very tough problem be aware of is that antivirus companies often and far from being perfectly implemented con- parametrize their scan engines specifically for sidering antivirus software sandboxes. There- VirusTotal (stronger heuristics, cloud interac- fore, there