Malware Detection Advances in Information Security Sushil Jajodia Consulting Editor Center for Secure Information Systems George Mason University Fairfax, VA 22030-4444 email: ja jodia @ smu.edu The goals of the Springer International Series on ADVANCES IN INFORMATION SECURITY are, one, to establish the state of the art of, and set the course for future research in information security and, two, to serve as a central reference source for advanced and timely topics in information security research and development. The scope of this series includes all aspects of computer and network security and related areas such as fault tolerance and software assurance. ADVANCES IN INFORMATION SECURITY aims to publish thorough and cohesive overviews of specific topics in information security, as well as works that are larger in scope or that contain more detailed background information than can be accommodated in shorter survey articles. The series also serves as a forum for topics that may not have reached a level of maturity to warrant a comprehensive textbook treatment. Researchers, as well as developers, are encouraged to contact Professor Sushil Jajodia with ideas for books under this series. Additional titles in the series: ELECTRONIC POSTAGE SYSTEMS: Technology, Security, Economics by Gerrit Bleumer; ISBN: 978-0-387-29313-2 MULTIVARIATE PUBLIC KEY CRYPTOSYSTEMS by Jintai Ding, Jason E. Gower and Dieter Schmidt; ISBN-13: 978-0-378-32229-2 UNDERSTANDING INTRUSION DETECTION THROUGH VISUALIZATION by Stefan Axelsson; ISBN-10: 0-387-27634-3 QUALITY OF PROTECTION: Security Measurements and Metrics by Dieter Gollmann, Fabio Massacci and Artsiom Yautsiukhin; ISBN-10; 0-387-29016-8 COMPUTER VIRUSES AND MALWARE by John Aycock; ISBN-10: 0-387-30236-0 HOP INTEGRITY IN THE INTERNET by Chin-Tser Huang and Mohamed G. Gouda; ISBN-10: 0-387-22426-3 CRYPTOGRAPHICS: Exploiting Graphics Cards For Security by Debra Cook and Angelos Keromytis; ISBN: 0-387-34189-7 PRIVACY PRESERVING DATA MINING by Jaideep Vaidya, Chris Clifton and Michael Zhu; ISBN-10: 0-387- 25886-8 BIOMETRIC USER AUTHENTICATION FOR IT SECURITY: From Fundamentals to Handwriting by Claus Vielhauer; ISBN-10: 0-387-26194-X IMPACTS AND RISK ASSESSMENT OF TECHNOLOGY FOR INTERNET SECURITY:Enabled Information Small-Medium Enterprises (TEISMES) by Charles A. Shoniregun; ISBN-10: 0-387-24343-7 SECURITY IN E-LEARNING by Edgar R. Weippl; ISBN: 0-387-24341-0 IMAGE AND VIDEO ENCRYPTION: From Digital Rights Management to Secured Personal Communication by Andreas Uhl and Andreas Pommer; ISBN: 0-387-23402-0 Additional information about this series can be obtained from http://www.springer.com Malware Detection edited by Mihai Christodorescu Somesh Jha University of Wisconsin, USA Douglas Maughan Department of Homeland Security, USA Dawn Song Carnegie Mellon University, USA Cliff Wang Army Research Office, USA ^ Sprin ge r Mihai Christodorescu Somesh Jha Computer Sciences Department Computer Sciences Department University of Wisconsin University of Wisconsin 1210 W Dayton St 1210 W Dayton St Madison, WI 53706-1685 Madison, WI53706-1685 [email protected] [email protected] Douglas Maughan Dawn Song Dept. of Homeland Security CIC 2122 Washington, D.C. 20528 Carnegie Mellon University [email protected] 4720 Forbes Ave Pittsburgh, PA 15213 [email protected] CI iff Wang Computing and Information Science Div. U.S. Army Research Office P.O. Box 12211 Research Triangle Park, NC 27709-2211 [email protected] Library of Congress Control Number: 2006933728 Malware Detection edited by Mihai Christodorescu, Somesh Jha, Douglas Maughan, Dawn Song, and Cliff Wang ISBN-10: 0-387-32720-7 ISBN-13: 978-0-387-32720-4 e-ISBN-10: 0-387-44599-4 e-ISBN-13: 978-0-387-44599-1 Printed on acid-free paper. © 2007 Springer Science+Business Media, LLC. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now know or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks and similar terms, even if the are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed in the United States of America. 987654321 springer.com Preface Malicious programs present an increasing threat to the privacy of sensitive data and the availability of critical services. As Internet connectivity exploded and online ser­ vices have become omnipresent, malware has targeted all aspects of the cyberworld. Driven by profit, malware authors have sharpened their skills to attack all online services, from banking to social networking to instant messaging, with increased frequency and sophistication. This book captures recent advances in the defense against all types of threats, and the chapters reflect a diversity of defensive techniques. Chapter 1 presents a detailed view of the threat landscape and analyzes the malware trends. The remaining chapters are organized into themes corresponding to the various malware threats. Chapters 2-5 present techniques for analyzing existing programs to determine their trustworthiness, as well as techniques for armoring programs against remote at­ tacks. Chapter 2 introduces robust approaches to the disassembly and static analysis of obfuscated binaries, including obfuscated malware, while Chapter 3 describes a static analysis to recover high-level variables and data structures from binaries. Tech­ niques that characterize the behavioral and structural properties of binary code are used to generate semantically-rich descriptions of malware in Chapter 4. New ap­ proaches for the detection and prevention of SQL injection attacks against database- driven web applications are presented in Chapter 5. The second part of the book (chapters 6-9) tackles the problem of distributed threats and the challenge of distributed detection. Network containment of worms (Chapter 6) complements the host-based self-healing architecture of Sting (Chap­ ter 7) to provide end-to-end defenses against fast Internet-scale worm attacks. Chap­ ter 8 presents the inner workings of botnets, the large networks of infected hosts under the control of a remote attacker. Chapter 9 analyzes the benefits of cooperation between network-based and host-based intrusion detectors and provides practical guidelines for obtaining the maximum detection rate out of a cooperative setup. Targeted and stealthy threats meet their match in Chapters 10 and II. Shadow honeypots in Chapter 10 combine the power of anomaly detectors with the preci­ sion of honeypots to detect targeted attacks. Statistical methods for binary content analysis are then used in Chapter 11 to detect malware hiding in document files. VI Preface The last part of the book presents new techniques for constructing trustworthy services and applications from the ground up. Pioneer in Chapter 12 can verify the correct execution of a program on an untrusted remote host. Chapter 13 explains the principles of secure information flow analysis, with the goal of proving that a program does not leak sensitive information. We are grateful to the authors appearing in this edited volume for their contribu­ tions to the field of malware detection, in all of its aspects, and for striving to make the Internet a safer, more trustworthy place. Mihai Christodorescu Somesh Jha Douglas Maughan Dawn Song Cliff Wang Contents Part I Overview 1 Malware Evolution: A Snapshot of Threats and Countermeasures in 2005 Brian Witten, Carey Nachenberg 3 Part II Software Analysis and Assurance 2 Static Disassembly and Code Analysis Giovanni Vigna 19 3 A Next-Generation Platform for Analyzing Executables Thomas Reps, Gogul Balakrishnan, Junghee Lim, Tim Teitelbaum 43 4 Behavioral and Structural Properties of Malicious Code Christopher Kruegel 63 5 Detection and Prevention of SQL Injection Attacks William G.J. Halfond, Alessandro Orso 85 Part III Distributed Threat Detection and Defense 6 Very Fast Containment of Scanning Worms, Revisited Nicholas Weaver, Stuart Stamford, Vern Paxson 113 7 Sting: An End-to-End Self-Healing System for Defending against Internet Worms David Brumley, James Newsome, Dawn Song 147 8 An Inside Look at Botnets Paul Barford, Vinod Yegneswaran 171 VIII Contents 9 Can Cooperative Intrusion Detectors Challenge the Base-Rate Fallacy? Mihai Christodorescu, Shai Rubin 193 Part IV Stealthy and Targeted Threat Detection and Defense 10 Composite Hybrid Techniques For Defending Against Targeted Attacks Stelios Sidiroglou, Angelas D. Keromytis 213 11 Towards Stealthy Malware Detection Salvatore J. Stolfo, Ke Wang, Wei-Jen Li 231 Part V Novel Techniques for Constructing Trustworthy Services 12 Pioneer: Verifying Code Integrity and Enforcing Untampered Code Execution on Legacy Systems Arvind Seshadri, Mark Luk, Adrian Perrig, Leendert van Doom, Pradeep Khosla 253 13 Principles of Secure Information Flow Analysis Geoffrey Smith 291 Index 309 Introduction Shared resources, such as the Internet, have created a highly interconnected cyber- infrastructure. Critical infrastructures in domains such as medical, power, telecom­ munications, and finance are highly dependent on information systems. These two factors have exposed our