Eastern Washington University EWU Digital Commons EWU Masters Thesis Collection Student Research and Creative Works 2014 Bridging the detection gap: a study on a behavior- based approach using malware techniques Geancarlo Palavicini Eastern Washington University Follow this and additional works at: http://dc.ewu.edu/theses Part of the Computer Sciences Commons Recommended Citation Palavicini, Geancarlo, "Bridging the detection gap: a study on a behavior-based approach using malware techniques" (2014). EWU Masters Thesis Collection. 186. http://dc.ewu.edu/theses/186 This Thesis is brought to you for free and open access by the Student Research and Creative Works at EWU Digital Commons. It has been accepted for inclusion in EWU Masters Thesis Collection by an authorized administrator of EWU Digital Commons. For more information, please contact [email protected]. BRIDGING THE DETECTION GAP: A STUDY ON A BEHAVIOR-BASED APPROACH USING MALWARE TECHNIQUES ________________________________________________________________________ A Thesis Presented To Eastern Washington University Cheney, WA ________________________________________________________________________ In Partial Fulfillment of the Requirements for the Degree Master of Science, in Computer Science ______________________________________________________ By Geancarlo Palavicini Jr Winter 2014 ii THESIS OF GEANCARLO PALAVICINI JR APPROVED BY _________________________________________ DATE_________ CAROL TAYLOR, GRADUATE STUDY COMMITEE _________________________________________ DATE_________ BILL CLARK, GRADUATE STUDY COMMITEE iii MASTER'S THESIS In presenting this thesis in partial fulfillment of the requirements for a master's degree at Eastern Washington University, I agree that the JFK Library shall make copies freely available for inspection. I further agree that copying of this project in whole or in part is allowable only foe scholarly purposes. It is understood ; however, that any copying or publication of this thesis for commercial purposes, or for financial gain, shall not be allowed without my written permission. Signature______________________ Date__________________________ iv Abstract In recent years the intensity and complexity of cyber attacks have increased at a rapid rate. The cost of these attacks on U.S. based companies is in the billions of dollars, including the loss of intellectual property and reputation. Novel and diverse approaches are needed to mitigate the cost of a security breach, and bridge the gap between malware detection and a security breach. This thesis focuses on the short term need to mitigate the impact of undetected shellcodes that cause security breaches. The thesis's approach focuses on the agents driving the attacks, capturing their actions, in order to piece together the attacks for forensics purposes, as well as to better understand the opponent. The work presented in this thesis employs models of normal operating system behavior to detect access to the operating system's shell interface. It also utilizes malware techniques to avoid detection and subsequent termination of the monitoring system , as well as dynamic shellcode execution methodologies in the testing of the thesis' modules to implement a monitoring system. v Acknowledgements First and foremost I would like to thank God for leading me through this thesis. I would like to thank Dr. Taylor for her guidance, time, and diligence throughout this entire process. Thank you for helping me through the murky waters, and giving me the opportunity to pursue the darker side of security. I specially would like to thank Miriam, my wife, for attention to her detail in helping me polish up the little details, as well as supporting and encouraging me through this long process. Thank you for all your sacrifice, patience, and hard work, so I could focus on the work set before me. I would also like to thank my children for enduring the pain of playing quietly so daddy could get his work done. vi Contents Abstract iv Acknowledgements v List of Figures viii 1. Introduction.......................................................................................................................1 1.1 Detection Problem.....................................................................................................2 1.2 Thesis Goal...............................................................................................................8 2. Literature Review........................................................................................................11 3. Implementation................................................................................................................18 3.1 Implementing the Kernel Module...........................................................................20 3.2 Implementing the Management Service..................................................................31 3.3 Implementing the Monitoring Facility....................................................................42 4. Experiment Development............................................................................................47 4.1 Exploit Setup...........................................................................................................48 4.2 Verification Procedures...........................................................................................51 vii 4.2.1 Reverse_shell Verification Procedure............................................................51 4.2.2 Bind_shell Verification Procedure.................................................................53 4.2.3 Remote Shell Access Verification Procedure................................................55 5. Procedure......................................................................................................................57 5.1 Iterations..................................................................................................................57 5.2 Testing Verification Procedures..............................................................................58 5.3 Obtaining a Target Score.........................................................................................63 6. Results...........................................................................................................................64 6.1 Individual Results....................................................................................................65 7. Future Work 72 8. Conclusion 74 Bibliography 76 A. Experiment Scripts 80 Vita 84 viii List of Figures 3.1 Implementation Overview..........................................................................................19 3.2 System Call Unhooked & Hooked.............................................................................21 3.3 Pseudoterminals..........................................................................................................22 3.4 SSH Access to Shell....................................................................................................23 3.5 Linux Security Module Framework Unhooked..........................................................28 3.6 Linux Security Module Framework Hooked..............................................................29 3.7 Transfer Mechanism....................................................................................................38 3.8 The TTY Layer's Line Discipline................................................................................44 1. Introduction Summer of 2009 witnessed the largest cyber espionage campaign against U.S. based companies to date. It is now known as Operation Aurora Attacks. A small software development company called Solid Oak was among those U.S. based companies caught in the wave of attacks. Their flagship product, a web filtering application called CyberSitter, was at the epicenter of a copy right infringement battle. The company claimed the Chinese government had stolen the source code for CyberSitter to implement a national web censoring service. Curious of the claim made by Solid Oak, University of Michigan researchers examined the code, and confirmed the company’s claim.[32] They discovered an upgrade announcement comment for CyberSitter accidentally left in the censorship service’s code. Soon after the University of Michigan researcher's findings, Solid Oak began a civil lawsuit. Within less than 2 weeks of accusations, strange things began happening on Solid Oaks networks and services. For the next 3 years the company is under intense cyber attacks. Product orders begin to fail, servers reboot on their own, support websites become unavailable. In short, it brings the company to the brink of bankruptcy. Tired of fighting and short on cash, the company settles out of court, and two months later the lawsuit is dropped. Within those same two months, the cyber attacks on the company's networks stopped. 2 On Feb. 2012, a security firm based out of Washington DC by the name of Mandilant, released a report on the suspected perpetrators of Operation Aurora. It claimed the operation was undertaken by 50 to 100 hackers, trained on network breaches and information stealing. The report provides details on the level of organization, skill, and methodology used by this group of hackers.[22] It also alleges that it’s a state run military unit, based out of China. On Sept. 2013, Symantec Corp. released a report confirming Mandilant’s findings, short of pointing fingers to any nation.[9] In short, U.S. based companies are losing billions of dollars due to cyber attacks.