Malware Armoring: The case against incident related binary analysis Author Name: Steve Hendrikse Date: February 2011 Supervisor: John Austen Registration Number (SRN): 060352033 Submitted as part of the requirements for the award of the MSc in Information Security of the University of London Table of Contents List of Figures ................................................................................................................................................ 4 1. Executive Summary .............................................................................................................................. 8 2. Introduction ......................................................................................................................................... 9 3. Incident Response .............................................................................................................................. 10 3.1. Introduction ............................................................................................................................... 10 3.2. Preparation ................................................................................................................................ 11 3.3. Identification ............................................................................................................................. 16 3.4. Containment .............................................................................................................................. 22 3.5. Eradication ................................................................................................................................. 28 3.6. Recovery .................................................................................................................................... 29 3.7. Lessons Learned ........................................................................................................................ 29 3.8. Conclusion ................................................................................................................................. 30 4. Armoring Techniques ......................................................................................................................... 30 4.1. Introduction ............................................................................................................................... 30 4.2. Encoding .................................................................................................................................... 32 4.2.1. String Encoding .......................................................................................................................... 32 4.2.2. Payload Encoding (Encryption) .................................................................................................. 33 4.2.3. Executable packing .................................................................................................................... 35 4.3. Virtual Machine Environment (VME) Detection ........................................................................ 39 4.4. Run-time decryption .................................................................................................................. 42 4.5. Polymorphism and Metamorphism ........................................................................................... 45 4.6. Anti-Debugging .......................................................................................................................... 48 4.6.1. API Based Detection .................................................................................................................. 49 4.6.2. Process and Thread Block Detection ......................................................................................... 49 4.6.3. Hardware and Register Based Detection................................................................................... 50 4.6.4. Exception Based Detection ........................................................................................................ 52 4.6.5. Timing Based Detection ............................................................................................................. 53 2 4.6.6. Modified Code Based Detection ................................................................................................ 54 4.7. Multi-partite .............................................................................................................................. 54 4.8. Conclusions ................................................................................................................................ 56 5. Case Study .......................................................................................................................................... 56 5.1. Introduction ............................................................................................................................... 56 5.2. Preparation ................................................................................................................................ 57 5.3. Identification ............................................................................................................................. 57 5.4. Containment .............................................................................................................................. 57 5.5. Eradication ................................................................................................................................. 66 5.6. Recovery .................................................................................................................................... 66 5.7. Lessons Learned ........................................................................................................................ 67 5.8. Conclusions ................................................................................................................................ 67 6. Conclusions ........................................................................................................................................ 68 7. Bibliography ....................................................................................................................................... 70 Appendix A – Glossary ................................................................................................................................ 77 Appendix B – Case study findings ............................................................................................................... 78 From investigation of malware that is known not to be armored ......................................................... 78 From investigation of malware that is known not to be armored ......................................................... 95 Appendix C – Case study Rootkit source code .......................................................................................... 131 Appendix D – Creation of infection ShellCode .......................................................................................... 133 3 List of Figures Figure 1 - Function Hooking (Scambray & McClure, 2008) ......................................................................... 19 Figure 2 - Process Hiding (Scambray & McClure, 2008) .............................................................................. 20 Figure 3 - Registry (Scambray & McClure, 2008) ........................................................................................ 20 Figure 4 - NTFS Layout (SANS, 2006) ........................................................................................................... 22 Figure 5 - 3 Interconnected Phases of Analysis (Zeltser, 2010) .................................................................. 23 Figure 6 - Behavioural Analysis (Aquilina, Casey, & Malin, 2008) ............................................................... 24 Figure 7 - Binary and Code Analysis (Aquilina, Casey, & Malin, 2008) ....................................................... 25 Figure 8 - Memory Analysis (Aquilina, Casey, & Malin, 2008) .................................................................... 27 Figure 9 - String encoding/replacement ..................................................................................................... 33 Figure 10 - Payload Encoding ...................................................................................................................... 34 Figure 11 - PE File Layout (Pietrek, 1994) ................................................................................................... 35 Figure 12 - IMAGE_OPTIONAL_HEADER (Websense) ................................................................................. 36 Figure 13 - IMAGE_SECTION_HEADER (Websense) .................................................................................... 37 Figure 14 - Redpill VME detection routine (Rutkowska, 2004) ................................................................... 40 Figure 15 - LDTR, GDT based VME detection (Klein, 2010) ......................................................................... 40 Figure 16 - Packer and Cypter Comparison (Aquilina, Casey, & Malin, 2008) ............................................ 43 Figure 17 - Runtime Decryption - Executable View (Oreans Technologies, 2008) .................................... 43 Figure 18 - Crypter with armoring options (xinfiltrate) .............................................................................. 44 Figure