Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Kjetil Tangen Gardåsen Master’s Thesis Master of Science in Information Security 30 ECTS Department of Computer Science and Media Technology Gjøvik University College, 2014 Avdeling for informatikk og medieteknikk Høgskolen i Gjøvik Postboks 191 2802 Gjøvik Department of Computer Science and Media Technology Gjøvik University College Box 191 N-2802 Gjøvik Norway Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Abstract In this master’s thesis we show how we can use the Application Programming Interface (API) calls a program makes to detect if it is a Remote Administration Trojan (RAT) or not. We have looked at which API calls that are performed when the webcam, microphone and keylogger are used by a program, and we use this information to determine if a program can be considered to be spying on its user by covertly capturing information from any of these sources. In addition, we have looked at the network traffic as seen through the API calls of the examined RAT, and we use this to better pinpoint which RAT that is present on the system. To distinguish a RAT from a program with similar behavior (e.g. video chat vs. webcam spy- ing) we have used user interaction with the program as a discriminator for whether or not it is malicious, with our theory being that a program such as a video chat program would interact with the user and have a user interface, while a RAT would not and would instead attempt to stay hidden. By observation we show that this is what is happening in real-life scenarios, and that the RATs do not interact with the user or provide a user interface. We have subsequently modeled the API calls that denote the various forms of behavior and user interaction as finite-state machines that can be used to detect whether or not a RAT is present on the system and what it is doing. ii Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Sammendrag I denne masteroppgaven viser vi hvordan vi kan bruke Application Programming Interface (API) kallene et program utfører for å oppdage om det er en Remote Administration Trojan (RAT) eller ikke. Vi har undersøkt hvilke API-kall som blir gjort når webkamera, mikrofon og tastaturet brukes av et program, og vi bruker denne informasjonen til å finne ut av om et program spionerer på brukeren via noen av disse grensesnittene eller ikke. I tillegg har vi sett på nettverkstrafikken via API-kallene som den undersøkte RATen utførerer, og vi bruker denne informasjon til å mer nøyaktig finne ut hvilken RAT som befinner seg på systemet. For å skille en RAT fra et program med lignende oppførsel (f.eks. video chat og wekamera- spionering) har vi brukt interaksjon med brukeren som et skille på hvorvidt programmet er skadelig eller ikke, hvor teorien vår har vært at et program som video chaten vil samhandle med brukeren og dermed ha et brukergrensesnitt, mens en RAT ikke vil det og heller forsøker å holde seg skjult. Ved observasjon har vi vist at det er dette som skjer, og at RATene ikke samhandler med brukeren eller har noe brukergrensesnitt. Vi har til slutt modellert API-kallene som brukes for de forskjellige typene oppførsel og bruker- samhandling som tilstandsmakiner som kan brukes til å oppdage om en RAT finnes på systemet og hva den driver med. iii Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Acknowledgments I would like to thank my supervisor Lasse Øverlier for all the insightful help and guidance he has given me throughout the thesis. My dear friends André, Eirik, Espen and Ola have my eternal gratitude for enduring all the relevant academic and non-relevant academic discussions we have had during this thesis. I would also like to thank my colleagues at Nets who have been very sup- portive and provided me with encouragement along the way. Lasse Halaas and FLO/IKT deserves a thank-you for steering me onto this topic and their helpful sparring on related topics. And last, but not least, I want to thank my parents and sister for their everlasting love and support. iv Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Glossary • RAT - Remote Administration Trojan/Tool • FSM - Finite-State Machine • IDS - Intrusion Detection System • SSDT - System Service Descriptor Table • IAT - Import Address Table • FUD - Fully Undetectable • DDoS - Distributed Denial of Service v Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines Contents Abstract .............................................. ii Sammendrag ........................................... iii Acknowledgments ........................................ iv Glossary ............................................. v Contents ............................................. vi List of Figures .......................................... ix List of Tables ........................................... xi 1 Introduction ......................................... 1 1.1 Topics Covered by the Project . .3 1.2 Keywords . .4 1.3 Problem description . .4 1.4 Justification, Motivation and Benefits . .5 1.5 Research Questions . .5 1.6 Contributions . .6 2 Background ......................................... 7 2.1 What is a RAT? . .7 2.1.1 Backdoors and Zombies . .7 2.1.2 Trojan Horses . .7 2.1.3 Spyware . .8 2.1.4 Summary . .9 2.2 A Brief History . .9 2.2.1 Back Orifice and Back Orifice 2000 . .9 2.2.2 Poison Ivy . 10 2.2.3 Today . 10 2.3 Detection Mechanisms . 10 2.3.1 Static Anti-Virus Detection . 10 2.3.2 Dynamic Anti-Virus Detection . 11 2.3.3 Properties . 12 2.3.4 Intrusion Detection Systems . 13 2.4 Technical Details . 13 2.4.1 User and Kernel Mode . 14 2.4.2 System Calls . 14 2.4.3 API calls . 14 2.4.4 Hooks . 15 2.4.5 Summary . 17 2.4.6 Obfuscation . 17 vi Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines 2.4.7 Summary . 19 2.5 Finite State Machines . 19 2.5.1 Finite-State Machine with Output . 20 2.5.2 Finite-State Machine without Output . 20 2.5.3 Using Finite-State Machines to Detect Malicious Behavior . 20 3 Related Work ......................................... 21 3.1 API Call Monitoring . 21 3.2 Signatures . 22 3.3 Policy . 23 3.4 Graphs . 23 3.5 Similarity Classification . 23 3.6 Identification of Dangerous API Calls . 24 3.7 Behavior Classification . 25 3.8 Finite-State Machines . 26 4 Methodology ......................................... 28 4.1 Rationale for Testing . 28 4.2 Test Setup . 28 4.2.1 Experiments . 31 5 Analysis ............................................ 33 5.1 DarkComet . 33 5.1.1 Webcam . 34 5.1.2 Audio Capture . 34 5.1.3 Keylogging . 35 5.1.4 Network Traffic . 35 5.2 njRAT . 38 5.2.1 Webcam . 39 5.2.2 Microphone . 39 5.2.3 Keylogger . 40 5.2.4 Network Traffic . 40 5.2.5 The Future . 43 5.3 jRAT . 44 5.3.1 Microphone . 44 5.3.2 Network Traffic . 44 5.4 jSpy . 46 5.4.1 Webcam . 46 5.4.2 Microphone . 46 5.4.3 Keylogger . 46 5.4.4 Network Traffic . 47 5.5 Cloud Net . 48 5.5.1 Webcam . 48 5.5.2 Microphone . 48 5.5.3 Keylogger . 48 vii Detecting Remote Administration Trojans through Dynamic Analysis using Finite-State Machines 5.5.4 Network Traffic . 48 5.6 LuxNET . 51 5.6.1 Webcam . 51 5.6.2 Microphone . 51 5.6.3 Keylogger . 51 5.6.4 Network Traffic . 51 5.7 Skype . ..