DOCSLIB.ORG

The Art of Unpacking 1

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Art of Unpacking 1 The Art of Unpacking 1 The Art of Unpacking Mark Vincent Yason Malcode Analyst, X-Force Research & Development IBM Internet Security Systems Abstract: Unpacking is an art—it is a mental challenge and is one of the most exciting mind games in the reverse engineering field. In some cases, the reverser needs to know the internals of the operating system in order to identify or solve very difficult anti-reversing tricks employed by packers/protectors, patience and cleverness are also major factors in a successful unpack. This challenge involves researchers creating the packers and on the other side, the researchers that are determined to bypass these protections. The main purpose of this paper is to present anti-reversing techniques employed by executable packers/protectors and also discusses techniques and publicly available tools that can be used to bypass or disable this protections. This information will allow researchers, especially, malcode analysts to identify these techniques when utilized by packed malicious code, and then be able decide the next move when these anti-reversing techniques impede successful analysis. As a secondary purpose, the information presented can also be used by researchers that are planning to add some level of protection in their software by slowing down reversers from analyzing their protected code, but of course, nothing will stop a skilled, informed, and determined reverser. Keywords: reverse engineering, packers, protectors, anti-debugging, anti-reversing Revision 4.0 The Art of Unpacking 2 Table of Contents Page Table of Contents...........................................................................................................................2 1. INTRODUCTION.....................................................................................................................3 2. TECHNIQUES: DEBUGGER DETECTION .....................................................................................4 2.1. PEB.BeingDebugged Flag: IsDebuggerPresent() ................................................................4 2.2. PEB.NtGlobalFlag, Heap Flags .........................................................................................5 2.3. DebugPort: CheckRemoteDebuggerPresent() / NtQueryInformationProcess()........................6 2.4. Debugger Interrupts ......................................................................................................7 2.5. Timing Checks ..............................................................................................................8 2.6. SeDebugPrivilege ..........................................................................................................9 2.7. Parent Process ............................................................................................................ 10 2.8. DebugObject: NtQueryObject() ..................................................................................... 11 2.9. Debugger Window ....................................................................................................... 12 2.10. Debugger Process ................................................................................................... 12 2.11. Device Drivers ........................................................................................................ 12 2.12. OllyDbg: Guard Pages.............................................................................................. 13 3. TECHNIQUES: BREAKPOINT AND PATCHING DETECTION.......................................................... 14 3.1. Software Breakpoint Detection ...................................................................................... 14 3.2. Hardware Breakpoint Detection ..................................................................................... 15 3.3. Patching Detection via Code Checksum Calculation.......................................................... 16 4. TECHNIQUES: ANTI-ANALYSIS .............................................................................................. 17 4.1. Encryption and Compression......................................................................................... 17 4.2. Garbage Code and Code Permutation............................................................................. 18 4.3. Anti-Disassembly ........................................................................................................ 20 5. TECHNIQUES : DEBUGGER ATTACKS ..................................................................................... 22 5.1. Misdirection and Stopping Execution via Exceptions ......................................................... 22 5.2. Blocking Input ............................................................................................................ 23 5.3. ThreadHideFromDebugger ............................................................................................ 24 5.4. Disabling Breakpoints .................................................................................................. 25 5.5. Unhandled Exception Filter ........................................................................................... 26 5.6. OllyDbg: OutputDebugString() Format String Bug ........................................................... 26 6. TECHNIQUES : ADVANCED AND OTHER TECHNIQUES .............................................................. 27 6.1. Process Injection......................................................................................................... 27 6.2. Debugger Blocker........................................................................................................ 28 6.3. TLS Callbacks ............................................................................................................. 29 6.4. Stolen Bytes ...............................................................................................................30 6.5. API Redirection ........................................................................................................... 31 6.6. Multi-Threaded Packers................................................................................................ 32 6.7. Virtual Machines.......................................................................................................... 32 7. TOOLS ............................................................................................................................... 34 7.1. OllyDbg...................................................................................................................... 34 7.2. Ollyscript.................................................................................................................... 34 7.3. Olly Advanced............................................................................................................. 34 7.4. OllyDump ...................................................................................................................34 7.5. ImpRec ...................................................................................................................... 34 8. REFERENCES....................................................................................................................... 35 The Art of Unpacking 3 1. INTRODUCTION In the reverse engineering field, packers are one of the most interesting puzzles to solve. In the process of solving these puzzles, the reverser gains more knowledge about a lot of things such operating system internals, reversing tricks, tools and techniques. Packers (the term used in this paper for both compressors and protectors) are created to protect an executable from analysis. They are used legitimately by commercial applications to prevent information disclosure, tampering and piracy. Unfortunately, malcodes also use packers for the same reasons but for a malicious purpose. Due to a large number of packed malcode, researchers and malcode analysts started to develop the skills to unpack samples for analysis. However, as time goes by, new anti- reversing techniques are constantly added into packers to prevent reversers from analyzing the protected executable and preventing a successful unpack. And the cycle goes on - new anti-reversing techniques are developed while reversers on the other side of the fence develop the skills, techniques, and tools to defeat them. The main focus of this paper is to present anti-reversing techniques employed by packers, tools and techniques on how to bypass/disable these protections are also discussed. Conversely, some packers can easily be bypassed by process dumping and thus, dealing with anti-reversing techniques seems unnecessary. However, there are instances where the protector code needed to be traced and analyzed, such as: • Parts of the protector code needed to be bypassed in order for a process dumping and import table rebuilding tool to properly work • In-depth analysis of a protector code in order to integrate unpacking support into an AV product Additionally, understanding anti-reversing techniques is also valuable in cases where they are directly applied to a malcode in order prevent tracing and analysis of their malicious routines. This paper is by no means contain a complete list of anti-reversing techniques as it only covers the commonly used and interesting techniques found in packers. The reader is advised to refer to the last section which contains links and books information to learn more about other anti-reversing and reversing techniques. The author hopes that the reader found this material useful
Load more

Recommended publications
  • Lecture 1: Introduction to Reverse Engineering
    Boston, 2001 Table of Contents Table of Contents Table of Contents............................................................................................. 2 1. Introduction................................................................................................. 5 1.1 About the Course and Notes ............................................................................ 5 1.2 Definitions...................................................................................................... 5 1.3 Typical Examples ............................................................................................ 6 1.3.1 Hacking ................................................................................................... 7 1.3.2 Hiding Information from Public .................................................................. 7 1.3.3 Cell Phones ............................................................................................ 10 1.3.4 Computer Applications ............................................................................ 10 1.4 Requirements............................................................................................... 12 1.5 Scope .......................................................................................................... 13 1.6 Ethics .......................................................................................................... 13 1.7 Miscellaneous Information ............................................................................. 14 2. Programming Processors
    [Show full text]
  • Softice Patch
    Softice patch click here to download SoftICE is a kernel mode debugger for Microsoft Windows up to Windows XP. Crucially, it is Newer versions of SoftICE patch deep into Microsoft Windows. i have installed softice along with the windows xp patch. When i go to start softice however i get a message stating 16 bit MS-DOS Subsystem. THE SOFTiCE PRESERVATiON SOCiETY - NuMega SoftICE v NT with XP patch & French keyboard. Compuware DriverStudio v Compuware. WindowsXP-deadlyxdose crack Softice +Patch&Serial keygen >> Download Actual patch SoftICE Depositfiles. SoftICE from "SoftICE Driver Suite " is used for debugging my program. Even I try the patch, softice still doesn't break!! mesfet. 年11月18日 Softice patch serial. It is designed to run underneath windows such that the operating system isconvert softice v3. Softice patch serial. SoftICE/Win 95 Patch clavier QWERTY -> AZERTY 4 APR 97 SoftICE Command Reference 30 MAR 97 SoftICE User Guide 30 MAR Even if Softice is not running, Skype complains with this error message and then exits: "Skype is not To patch Skype, I've used Ollydbg, a free debugger. Loading SoftICE for Windows 9x. .. Using Windows NT//XP Symbol (DBG) Files with SoftICE. to prevent SoftICE from patching the keyboard driver. SoftICE can be used for local kernel debugging, which is a feature that is very Very useful for patching, disassembling, and debugging. www.doorway.ru accessories: ice iron on patches cute soft ice applique as iron-on badge made of denim to embellish your jeans jacket or repair your pants as knee patches. An uninstall of Softice is required for correct.
    [Show full text]
  • The Shellcoder's Handbook
    80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page iii The Shellcoder’s Handbook Discovering and Exploiting Security Holes Second Edition Chris Anley John Heasman Felix “FX” Linder Gerardo Richarte The Shellcoder’s Handbook: Discovering and Exploiting Security Holes (1st Edition) was written by Jack Koziol, David Litchfield, Dave Aitel, Chris Anley, Sinan Eren, Neel Mehta, and Riley Hassell. Wiley Publishing, Inc. 80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page ii 80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page i The Shellcoder’s Handbook Second Edition 80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page ii 80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page iii The Shellcoder’s Handbook Discovering and Exploiting Security Holes Second Edition Chris Anley John Heasman Felix “FX” Linder Gerardo Richarte The Shellcoder’s Handbook: Discovering and Exploiting Security Holes (1st Edition) was written by Jack Koziol, David Litchfield, Dave Aitel, Chris Anley, Sinan Eren, Neel Mehta, and Riley Hassell. Wiley Publishing, Inc. 80238ffirs.qxd:WileyRed 7/11/07 7:22 AM Page iv The Shellcoder’s Handbook, Second Edition: Discovering and Exploiting Security Holes Published by Wiley Publishing, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2007 by Chris Anley, John Heasman, Felix “FX” Linder, and Gerardo Richarte Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-0-470-08023-8 Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600.
    [Show full text]
  • Compuware.Com
    Using SoftICE® Version 3.2 Technical support is available from our Technical Support Hotline or via our FrontLine Support Web site. Technical Support Hotline: 1-800-538-7822 FrontLine Support Web Site: http://frontline.compuware.com This document and the product referenced in it are subject to the following legends: Access is limited to authorized users. Use of this product is subject to the terms and conditions of the user’s License Agreement with Compuware Corporation. © 2004 Compuware Corporation. All rights reserved. Unpublished - rights reserved under the Copyright Laws of the United States. U.S. GOVERNMENT RIGHTS Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in Compuware Corporation license agreement and as provided in DFARS 227.7202-1(a) and 227.7202-3(a) (1995), DFARS 252.227-7013(c)(1)(ii)(OCT 1988), FAR 12.212(a) (1995), FAR 52.227-19, or FAR 52.227-14 (ALT III), as applicable. Compuware Corporation. This product contains confidential information and trade secrets of Compuware Corporation. Use, disclosure, or reproduction is prohibited without the prior express written permission of Compuware Corporation. DriverStudio, SoftICE Driver Suite, DriverNetworks, DriverWorks, TrueCoverage, and DriverWorkbench are trademarks of Compuware Corporation. BoundsChecker, SoftICE, and TrueTime are registered trademarks of Compuware Corporation. Acrobat® Reader copyright © 1987-2003 Adobe Systems Incorporated. All rights reserved. Adobe, Acrobat, and Acrobat Reader are trademarks of Adobe Systems Incorporated. All other company or product names are trademarks of their respective owners. US Patent Nos.: Not Applicable. Doc. 11577a December 10, 2004 Table of Contents Preface Purpose of This Manual .
    [Show full text]
  • The Measured Performance of Personal Computer Operating Systems
    The Measured Performance of Personal Computer Operating Systems J. Bradley Chen, Yasuhiro Endo, Kee Chan, David Mazi&res Antonio Dias, Margo Seltzer, and Michael D. Smith Division of Applied Sciences Harvard University Abstract dews. The differences between the OS platforms used in research and mainstream OS products makes some systems research mrele- This paper presents a comparative study of the performance of vant with respect to the current needs of the software industry. At three operating systems that run on the personal computer archi- the same time, researchers ignore important problems in non- tecture derived from the IBM-PC, The operating systems, Win- UNIX systems. dows for Workgroups, Windows NT, and NetBSD (a freely The operating system most commonly used on PC platforms available variant of the UNIX operating system), cover a broad is Microsoft Windows. Windows lacks many features that the OS range ofs ystem functionalist y and user requirements, from a single research community takes for granted, most notably preemptive address space model to full protection with preemptive multi-task- multitasking and protected address spaces. New operating systems ing. Our measurements were enabled by hardware counters in for the PC market such as Windows NT and 0S/2 incorporate the Intel’s Pentium processor that permit measurement of a broad multitasking support found in modern UNIX operating systems, range of processor events including instruction counts and on-chip while also supporting Windows applications. Though the operat- cache miss counts. We used both microbenchmarks, which expose ing systems community is familiar with the benefits of features specific differences between the systems, and application work- such as separate protected address spaces, little attention has been loads, which provide an indication of expected end-to-end perfor- given to systems without them.
    [Show full text]
  • Attacking the Windows Kernel
    Attacking the Windows Kernel Jonathan Lindsay ([email protected]) An NGSSoftware Insight Security Research (NISR) Publication ©2007 Next Generation Security Software Ltd http://www.ngssoftware.com Abstract Most modern processors provide a supervisor mode that is intended to run privileged operating system services that provide resource management transparently or otherwise to non-privileged code. Although a lot of research has been conducted into exploiting bugs in user mode code for privilege escalation within the operating system defined boundaries as well as what can be done if one has arbitrary supervisor access (typically related to modern root kit work), not a great deal of research has been done on the interface between supervisor and non-supervisor, and potential routes from one to the other. The biggest problem arises when trying to protect the kernel from itself - for example, under the IA32 architecture implementation of Windows, the distinction between user mode and kernel mode from the user mode perspective is easily enforced through hardware based protection. However, as the kernel is running as supervisor, how does the kernel make distinctions between what it should be accessing? This would be irrelevant if the supervisor was not exposed to interaction with supervisee; but that would defeat the purpose of having a kernel. This paper is focused on Windows and the Intel Architecture, and will briefly outline the current supervisor boundaries provided. Different attack vectors, along with relevant examples, will be provided to demonstrate how to attack the supervisor from the perspective of the supervised, as well as an outline of what possible architectures could be used to mitigate such attacks, such as the research operating system Singularity.
    [Show full text]
  • Stealth Breakpoints
    Stealth Breakpoints Amit Vasudevan and Ramesh Yerraballi Department of Computer Science and Engineering University of Texas at Arlington {vasudeva,ramesh}@cse.uta.edu Abstract 1. Introduction Microscopic analysis of malicious code (malware) re- Microscopic malware analysis — a fine-grained analysis quires the aid of a variety of powerful tools. Chief among process that provides insight into malware structure and in- them is a debugger that enables runtime binary analysis at ner functioning — helps in gleaning important information an instruction level. One of the important services provided regarding a malware to facilitate the development of an anti- by a debugger is the ability to stop execution of code at dote. Fine-grained analysis requires the aid of various pow- an arbitrary point during runtime, using breakpoints. Soft- erful tools, chief among them being a debugger that enables ware breakpoints support an unlimited number of breakpoint runtime binary analysis at an instruction level. One of the im- locations by changing the code being debugged so that it portant services provided by a debugger is the ability to stop can be interrupted during runtime. Most, if not all, malware execution of code being debugged at an arbitrary point dur- are very sensitive to code modification with self-modifying ing runtime. This is achieved using breakpoints, which can and/or self-checking (SM-SC) capabilities, rendering the use be of two types: Hardware and Software. Hardware break- of software breakpoints limited in their scope. Hardware points, as the name suggests, are provided by the underly- breakpoints supported by the underlying processor, on the ing processor and support precise breakpoints on code, data other hand, use a subset of the processor register set and ex- and I/O.
    [Show full text]
  • The Measured Performance of Computer Operating Systems Personal
    The Measured Performance of Personal Computer Operating Systems J. BRADLEY CHEN, YASJHIRO ENDO, KEE CHAN, DAVID MAZIERES, ANTONlO DIAS, MARGO SELTZER, AND MICHAEL D. SMITH Harvard University This article presents a comparative study of the pefiormance of three operating systems that run on the personal computer architecture derived from the IBM-PC. The operating systems, Windows for Workgroups, Windows NT, and NetBSD (a freely available variant of the UNIX operating system), cover a broad range of system functionality and user requirements, from a single-address-space model to full protection with preemptive multitasking, Our measurements are enabled by hardware counters in Intel’s Pentium processor that permit measurement of a broad range of processor events including instruction counts and on-chip cache miss counts, We use both microbenchmarks, which expose spcific differences between the systems, and applica- tion workloads, which provide an indication of expected end-to-end performance, Our micro- benchmark results show that accessing system functionality is often more expensive in Windows for Workgroups than in the other two systems due to frequent changes in machine mode and the use of system call hooks. When running native applications, Windows NT is more efficient than Windows, but it incurs overhead similar tQthat of a microkernel, since its application interface (the Win32 API) is implemented as a user-level server. Overall, system functionality can be accessed most efficiently in NetBSD; we attribute this to its monolithic structure and to the absence of the complications created by hardware backward-compatibility requirements in the other systems, Measurements of application performance show that although the impact of these differences is significant in terms of instruction counts and other hardware eventa (often a factor of 2 to 7 difference between the systems), overall performance is sometimes determined by the functionality provided by specific subsystems, such as the graphics subsystem or the file system buffer cache.
    [Show full text]
  • Rootkits and Bootkits with a Record of Discovering Advanced Malware.” — Rodrigo Rubira Branco Rootkits
    “Follow in the footsteps of professionals Bootkits and Rootkits with a record of discovering advanced malware.” Rootkits — Rodrigo Rubira Branco Rootkits and Bootkits will teach you how How to better understand the delivery stage to understand and counter sophisticated, of threats against BIOS and UEFI firmware advanced threats buried deep in a machine’s in order to create detection capabilities boot process or UEFI firmware. How to use virtualization tools like VMware and Bootkits With the aid of numerous case studies and Workstation to reverse engineer bootkits professional research from three of the world’s and the Intel Chipsec tool to dig into forensic leading security experts, you’ll trace malware analysis development over time from rootkits like TDL3 Reversing Modern Malware and to present-day UEFI implants and examine Cybercrime syndicates and malicious actors how they infect a system, persist through will continue to write ever more persistent reboot, and evade security software. As you and covert attacks, but the game is not lost. Next Generation Threats inspect and dissect real malware, you’ll learn: Explore the cutting edge of malware analysis with Rootkits and Bootkits. How Windows boots—including 32-bit, 64-bit, and UEFI mode—and where to find About the Authors vulnerabilities and Malware Modern Reversing Alex Matrosov is an Offensive Security The details of boot process security Threats Generation Next mechanisms like Secure Boot, including Research Lead at NVIDIA with over 20 years of an overview of Virtual Secure Mode (VSM) experience in reverse engineering, advanced and Device Guard malware analysis, firmware security, and exploitation techniques.
    [Show full text]
  • Malware Dynamic Analysis Evasion Techniques: a Survey
    Malware Dynamic Analysis Evasion Techniques: A Survey Amir Afianian1, Salman Niksefat1, Babak Sadeghiyan1, and David Baptiste2 1APA Research Center, Amirkabir University of Technology 2ESIEA (C + V)O lab 1fa.afianian, niksefat, [email protected] [email protected] June 2018 Abstract The Cyber world is plagued with ever-evolving malware that readily infiltrates all defense mechanisms, operates viciously unbeknownst to the user and surreptitiously exfiltrate sensitive data. Understanding the in- ner workings of such malware provides a leverage to effectively combat them. This understanding, is pursued through dynamic analysis which is conducted manually or automatically. Malware authors accordingly, have devised and advanced evasion techniques to thwart or evade these analyses. In this paper, we present a comprehensive survey on malware dynamic analysis evasion techniques. In addition, we propose a detailed classification of these techniques and further demonstrate how their effi- cacy hold against different types of detection and analysis approach. Our observations attest that evasive behavior is mostly interested in detecting and evading sandboxes. The primary tactic of such malware we argue, is fingerprinting followed by new trends for reverse Turing test tac- tic which aims at detecting human interaction. Furthermore, we will posit that the current defensive strategies beginning with reactive methods to arXiv:1811.01190v1 [cs.CR] 3 Nov 2018 endeavors for more transparent analysis systems, are readily foiled by zero- day fingerprinting techniques or other evasion tactics such as stalling. Ac- cordingly, we would recommend pursuit of more generic defensive strate- gies with emphasis on path exploration techniques that has the potential to thwart all the evasive tactics.
    [Show full text]
  • Freebsd 802.11 Remote Integer Overflow
    FreeBSD 802.11 Remote Integer Overflow Vulnerability found and exploit developed by Karl Janmar <[email protected]> IEEE802.11 framework in FreeBSD The IEEE802.11 system in FreeBSD in its current shape is relatively new (around 2001). The framework unifies all the handling of wireless devices. Problems faced auditing the code Complex link-layer protocol IEEE802.11 has a complex link-layer protocol, as a rough metric we compare the size of some input functions. ● IEEE802.11 input function, ieee80211_input(), 437 lines ● Ethernet input function, ether_input(), 107 lines ● Internet Protocol input function, ip_input(), 469 lines Source-code hard to read The code itself is not written to be easily read. It contains huge recursive switch- statements, for example a 274-line recursive switch-statement in the input function. Other examples are macros that include return statements and so on. User-controlled data The link-layer management in IEEE802.11 is unencrypted and unauthenticated, and because the traffic is transmitted in the air it's very easy for an attacker to manipulate state. Issues found An issue was found in an IOCTL, this issue was the result of a logical error. The vulnerability could allow a local user-process to disclose kernel-memory. Another more interesting issue was also found, it is in a function called by the IOCTL which retrieves the list of access-points in a scan. This list is maintained by the kernel, and is built from beacon frames received. Here is a snippet of the code in question: static int ieee80211_ioctl_getscanresults(struct
    [Show full text]
  • KERNEL-EXPLOITATION DEMYSTIFIED Introduction to Kernel-Mode Vulnerabilities and Exploitation
    KERNEL WARS: KERNEL-EXPLOITATION DEMYSTIFIED Introduction to kernel-mode vulnerabilities and exploitation • Why exploit kernel level vulnerabilities? – It's fun! – Relatively few are doing it – Bypasses defense mechanisms and restrictions 2 Introduction to kernel-mode vulnerabilities and exploitation • Why exploit kernel level vulnerabilities? – Attacks at the lowest level • Does not rely on any particular application being installed • Does not rely on how applications are configured • Does not rely on file / registry permissions 3 Introduction to kernel-mode vulnerabilities and exploitation • Reasons not to exploit kernel level vulnerabilities – Usually one-shot, exploit needs to be very reliable – Kernel debugging can be tedious setting up – Need some knowledge about kernel internals 4 Introduction to kernel-mode vulnerabilities and exploitation • Common targets for attack in a kernel – Systemcalls – I/O and IOCTL-messages through devicefiles – Handling of files in pseudofilesystems (like procfs) – Handling of data from the network (wireless/wired) – Interaction with hardware (USB, Firewire, etc) – Executable file format loaders (ELF, PE, etc) 5 Introduction to kernel-mode vulnerabilities and exploitation • Payload strategy – Elevating privileges • Altering the UID-field (Unix) • Stealing access tokens (Windows) – Injecting backdoors • Stealth! Do everything in kernel-mode 6 Introduction to kernel-mode vulnerabilities and exploitation • Payload strategy – Breaking chroot / jail / other restrictions • Everything can be bypassed in kernel-mode
    [Show full text]