The Antivirus Hacker’s Handbook ffi rs.indd 08:14:22:AM 08/13/2015 Page i The Antivirus Hacker’s Handbook Joxean Koret Elias Bachaalany ffi rs.indd 08:14:22:AM 08/13/2015 Page iii The Antivirus Hacker’s Handbook Published by John Wiley & Sons, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2015 by John Wiley & Sons, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-1-119-02875-8 ISBN: 978-1-119-02876-5 (ebk) ISBN: 978-1-119-02878-9 (ebk) Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permis- sion of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley .com/go/permissions. Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or war- ranties with respect to the accuracy or completeness of the contents of this work and specifi cally disclaim all warranties, including without limitation warranties of fi tness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that Internet websites listed in this work may have changed or disappeared between when this work was written and when it is read. For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com. Library of Congress Control Number: 2015945503 Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affi liates, in the United States and other countries, and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book. ffi rs.indd 08:14:22:AM 08/13/2015 Page iv About the Authors Joxean Koret has been working for the past +15 years in many different com- puting areas. He started as a database software developer and DBA, working with a number of different RDBMSs. Afterward he got interested in reverse- engineering and applied this knowledge to the DBs he was working with. He has discovered dozens of vulnerabilities in products from the major database vendors, especially in Oracle software. He also worked in other security areas, such as developing IDA Pro at Hex-Rays or doing malware analysis and anti- malware software development for an antivirus company, knowledge that was applied afterward to reverse-engineer and break over 14 AV products in roughly one year. He is currently a security researcher in Coseinc. Elias Bachaalany has been a computer programmer, a reverse-engineer, an occa- sional reverse-engineering trainer, and a technical writer for the past 14 years. Elias has also co-authored the book Practical Reverse Engineering, published by Wiley (ISBN: 978-111-8-78731-1). He has worked with various technologies and programming languages including writing scripts, doing web development, working with database design and programming, writing Windows device drivers and low-level code such as boot loaders or minimal operating systems, writing managed code, assessing software protections, and writing reverse- engineering and desktop security tools. Elias has also presented twice at REcon Montreal (2012 and 2013). While working for Hex-Rays SA in Belgium, Elias helped improve and add new features to IDA Pro. During that period, he authored various technical blog posts, provided IDA Pro training, developed various debugger plug-ins, amped up IDA Pro’s scripting facilities, and contributed to the IDAPython project. Elias currently works at Microsoft. v ffi rs.indd 08:14:22:AM 08/13/2015 Page v Credits Project Editor Professional Technology & Sydney Argenta Strategy Director Barry Pruett Technical Editor Daniel Pistelli Business Manager Amy Knies Production Editor Saleem Hameed Sulthan Associate Publisher Jim Minatel Copy Editor Marylouise Wiack Project Coordinator, Cover Brent Savage Manager of Content Development & Assembly Proofreader Mary Beth Wakefi eld Nicole Hirschman Production Manager Indexer Kathleen Wisor Nancy Guenther Marketing Director Cover Designer David Mayhew Wiley Marketing Manager Cover Image Carrie Sherrill Wiley; Shield © iStock.com/DSGpro vii ffi rs.indd 08:14:22:AM 08/13/2015 Page vii Acknowledgments I would like to acknowledge Mario Ballano, Ruben Santamarta, and Victor Manual Alvarez, as well as all my friends who helped me write this book, shared their opinions and criticisms, and discussed ideas. I am most thankful to my girlfriend for her understanding and support during the time that I spent on this book. Many thanks to Elias Bachaalany; without his help, this book would not have been possible. Also, special thanks to everyone at Wiley; it has been a great pleasure to work with you on this book. I am grateful for the help and support of Daniel Pistelli, Carol Long, Sydney Argenta, Nicole Hirschman, and Marylouise Wiack. ix ffi rs.indd 08:14:22:AM 08/13/2015 Page ix Contents at a Glance Introduction xix Part I Antivirus Basics 1 Chapter 1 Introduction to Antivirus Software 3 Chapter 2 Reverse-Engineering the Core 15 Chapter 3 The Plug-ins System 57 Chapter 4 Understanding Antivirus Signatures 77 Chapter 5 The Update System 87 Part II Antivirus Software Evasion 103 Chapter 6 Antivirus Software Evasion 105 Chapter 7 Evading Signatures 117 Chapter 8 Evading Scanners 133 Chapter 9 Evading Heuristic Engines 165 Chapter 10 Identifying the Attack Surface 183 Chapter 11 Denial of Service 207 Part III Analysis and Exploitation 217 Chapter 12 Static Analysis 219 Chapter 13 Dynamic Analysis 235 Chapter 14 Local Exploitation 269 Chapter 15 Remote Exploitation 297 xi ffi rs.indd 08:14:22:AM 08/13/2015 Page xi xii Contents at a Glance Part IV Current Trends and Recommendations 321 Chapter 16 Current Trends in Antivirus Protection 323 Chapter 17 Recommendations and the Possible Future 331 Index 347 ffi rs.indd 08:14:22:AM 08/13/2015 Page xii Contents Introduction xix Part I Antivirus Basics 1 Chapter 1 Introduction to Antivirus Software 3 What Is Antivirus Software? 3 Antivirus Software: Past and Present 4 Antivirus Scanners, Kernels, and Products 5 Typical Misconceptions about Antivirus Software 6 Antivirus Features 7 Basic Features 7 Making Use of Native Languages 7 Scanners 8 Signatures 8 Compressors and Archives 9 Unpackers 10 Emulators 10 Miscellaneous File Formats 11 Advanced Features 11 Packet Filters and Firewalls 11 Self-Protection 12 Anti-Exploiting 12 Summary 13 Chapter 2 Reverse-Engineering the Core 15 Reverse-Engineering Tools 15 Command-Line Tools versus GUI Tools 16 Debugging Symbols 17 Tricks for Retrieving Debugging Symbols 17 Debugging Tricks 20 xiii ftoc.indd 05:49:16:PM 08/10/2015 Page xiii xiv Contents Backdoors and Confi guration Settings 21 Kernel Debugging 23 Debugging User-Mode Processes with a Kernel-Mode Debugger 25 Analyzing AV Software with Command-Line Tools 27 Porting the Core 28 A Practical Example: Writing Basic Python Bindings for Avast for Linux 29 A Brief Look at Avast for Linux 29 Writing Simple Python Bindings for Avast for Linux 32 The Final Version of the Python Bindings 37 A Practical Example: Writing Native C/C++ Tools for Comodo Antivirus for Linux 37 Other Components Loaded by the Kernel 55 Summary 56 Chapter 3 The Plug-ins System 57 Understanding How Plug-ins Are Loaded 58 A Full-Featured Linker in Antivirus Software 58 Understanding Dynamic Loading 59 Advantages and Disadvantages of the Approaches for Packaging Plug-ins 60 Types of Plug-ins 62 Scanners and Generic Routines 63 File Format and Protocol Support 64 Heuristics 65 Bayesian Networks 66 Bloom Filters 67 Weights-Based Heuristics 68 Some Advanced Plug-ins 69 Memory