The University of Manchester Anti-virus tool: Using anti-virus techniques for malware detection Author: Radoslav Ralinov Supervisor: Ning Zhang The School of Computer Science Compute Science BSc (Hons) with Industrial Experience Third Year Project Report May 2016 Abstract Anti-virus tool: Using anti-virus techniques for malware detection Malware detection in the 21st century has become one of the most if not the most important aspects of security software in the Computer Science area. With the growing numbers of personal computer users the need for good malware detection algorithms has been increasing with considerable rate. The aim of this project is to build an application that demonstrates some of the main virus detection techniques used by anti-virus software to detect and remove malicious files. The techniques considered in the report could be split in two – signature-based and heuristic-based malware detection. The report also investigates the possibility of having an application that could learn based on the heuristic approaches. The report begins with a review of one of the existing malware scanning programs and the approaches it used for malware detection. At the end of the report, performance of the techniques implemented is compared as well as the effectiveness of each one of them is evaluated. Author: Radoslav Ralinov Supervisor: Ning Zhang 1 Acknowledgements Firstly I would like to thank my supervisor Ning Zhang for her support and guidance throughout the project. Her questions as well as answers really helped me drive this project forward in the right direction. I would also like to thank my family for believing in me and continuously supporting me throughout the whole third year. 2 Contents 1 Introduction ........................................................................................................................ 5 1.1 Motivation ............................................................................................................................... 5 1.2 Project Aim and Objectives ..................................................................................................... 6 1.3 Report Structure ..................................................................................................................... 6 2 Background ......................................................................................................................... 7 2.1 Types of malware .................................................................................................................... 7 2.2 ClamAntiVirus anti-virus scanner ............................................................................................ 7 2.2.1 ClamAV malware signatures ........................................................................................... 7 2.2.2 ClamAV heuristic ............................................................................................................. 9 2.3 Anti-virus techniques nowadays ............................................................................................. 9 3 Design ............................................................................................................................... 10 3.1 Requirements ........................................................................................................................ 10 3.1.1 Functional ...................................................................................................................... 10 3.1.2 Non-Functional .............................................................................................................. 11 3.2 Technologies ......................................................................................................................... 12 3.2.1 Programming languages ............................................................................................... 12 3.2.2 Orchestrate ................................................................................................................... 12 3.2.3 Version control .............................................................................................................. 13 3.2.4 Java Native Interface ..................................................................................................... 13 3.3 The malware detection cycle ................................................................................................ 14 4 Implementation ................................................................................................................ 16 4.1 Key libraries ........................................................................................................................... 16 4.2 Signature-based detection .................................................................................................... 17 4.3 Heuristic-based detection ..................................................................................................... 18 4.4 Concurrency .......................................................................................................................... 18 4.5 Malware management .......................................................................................................... 19 4.6 User interface implementation ............................................................................................. 19 5 Testing and evaluation ..................................................................................................... 21 5.1 Automated testing ................................................................................................................ 21 5.2 Manual Testing ...................................................................................................................... 21 5.3 Evaluation ............................................................................................................................. 22 3 6 Conclusion ........................................................................................................................ 23 6.1 Challenges ............................................................................................................................. 23 6.2 Gained knowledge ................................................................................................................ 23 6.3 Future work ........................................................................................................................... 23 7 References ........................................................................................................................ 24 Table of Figures FIGURE 1 CLAMAV'S FULL HASH SIGNATURE FORMAT ............................................................. 8 FIGURE 2 CLAMAV'S PARTIAL HASH SIGNATURE FORMAT ....................................................... 8 FIGURE 3 CLAMAV'S BYTE SIGNATURE FORMAT ....................................................................... 9 FIGURE 4 A BYTE SIGNATURE IN JSON FORMAT IN ORCHESTRATE ......................................... 13 FIGURE 5 JNI IN AN APPLICATION [15] .................................................................................... 13 FIGURE 6 THE MALWARE DETECTION CYCLE ........................................................................... 14 FIGURE 7 AHO-CORASICK TRIE WITH KEYWORDS "AB" "BCA" AND "CAA" [17] ..................... 16 FIGURE 8 COMPARISON BETWEEN MD5 AND SHA256 ........................................................... 17 FIGURE 9 HASH SIGNATURE DETECTION PROCESS .................................................................. 18 FIGURE 10 MULTITHREADING EXAMPLE ................................................................................. 19 FIGURE 11 SCREENSHOT OF THE TOOL WHILE SCANNING ..................................................... 20 Table of Tables TABLE 1 FUNCTIONAL REQUIREMENTS ................................................................................... 10 TABLE 2 NON-FUNCTIONAL REQUIREMENTS .......................................................................... 11 TABLE 3 COMPARISON BETWEEN AVAST ANTIVIRUS AND THE PROJECT TOOL ..................... 22 4 Chapter 1 1 Introduction Nowadays anti-virus software applications have become essential to the everyday personal computer user. These modern tools for computer malware scanning use a mixture of algorithms to detect and prevent malicious software from causing damage. This project is about building an online anti-virus tool for scanning file systems and detecting malware. The project encompasses the two main areas of malware discovery techniques. One approach which has been around since the inception of antivirus tools is the signature-based detection which examines the file’s key aspects for a known static fingerprint. The signature itself could be a sequence of bytes that represent the malicious code in the file or the cryptographic hash of the whole infected file. [1] Once the program has access to the malware signature it goes through the suspected file and looks for a match. The current number of virus signatures is over 100,000 and it is growing constantly. This is why it is really important what string-matching algorithm is chosen. The other and more modern technique for virus detection is the heuristic-based. A heuristic technique means an approach to problem solving that is not guaranteed to be optimal or perfect but sufficient enough. The heuristics-based detection approach relies on inspecting files for suspicious characteristics without the help