An Introduction to Reverse Engineering for Beginners
Total Page:16
File Type:pdf, Size:1020Kb
An Introduction To Reverse Engineering for Beginners Dennis Yurichev <[email protected]> cbnd ○c 2013, Dennis Yurichev. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Text version (November 23, 2013). There is probably a newer version of this text, and also Russian language version also accessible at http://yurichev.com/RE-book.html You may also subscribe to my twitter, to get information about updates of this text, etc: @yurichev, or to subscribe to mailing list. CONTENTS CONTENTS Contents Preface vi 0.1 Mini-FAQ1 ................................................. vi 0.2 About the author............................................. vi 0.3 Thanks.................................................. vi 0.4 Donate.................................................. vii 1 Compiler’s patterns 1 1.1 Hello, world!...............................................1 1.1.1 x86................................................1 1.1.2 ARM................................................4 1.2 Stack...................................................8 1.2.1 What is the stack used for?...................................9 1.3 printf() with several arguments.................................... 13 1.3.1 x86................................................ 13 1.3.2 ARM: 3 printf() arguments.................................. 14 1.3.3 ARM: 8 printf() arguments.................................. 15 1.3.4 By the way............................................ 18 1.4 scanf()................................................... 18 1.4.1 About pointers.......................................... 18 1.4.2 x86................................................ 18 1.4.3 ARM................................................ 20 1.4.4 Global variables......................................... 21 1.4.5 scanf() result checking...................................... 23 1.5 Passing arguments via stack....................................... 25 1.5.1 x86................................................ 25 1.5.2 ARM................................................ 27 1.6 One more word about results returning................................. 28 1.7 Pointers.................................................. 29 1.7.1 C++ references.......................................... 30 1.8 Conditional jumps............................................ 30 1.8.1 x86................................................ 31 1.8.2 ARM................................................ 32 1.9 switch()/case/default........................................... 34 1.9.1 Few number of cases...................................... 34 1.9.2 A lot of cases........................................... 38 1.10 Loops................................................... 43 1.10.1 x86................................................ 43 1.10.2 ARM................................................ 46 1.10.3 One more thing......................................... 47 1.11 strlen().................................................. 48 1.11.1 x86................................................ 48 1.11.2 ARM................................................ 50 1Frequently Asked Questions i CONTENTS 1.12 Division by 9............................................... 53 1.12.1 ARM................................................ 54 1.13 Work with FPU.............................................. 55 1.13.1 Simple example......................................... 55 1.13.2 Passing floating point number via arguments......................... 59 1.13.3 Comparison example...................................... 60 1.14 Arrays................................................... 68 1.14.1 Simple example......................................... 68 1.14.2 Buer overflow.......................................... 71 1.14.3 Buer overflow protection methods.............................. 75 1.14.4 One more word about arrays.................................. 78 1.14.5 Multidimensional arrays..................................... 78 1.15 Bit fields.................................................. 81 1.15.1 Specific bit checking....................................... 81 1.15.2 Specific bit setting/clearing................................... 85 1.15.3 Shis............................................... 87 1.15.4 CRC32 calculation example................................... 91 1.16 Structures................................................. 94 1.16.1 SYSTEMTIME example...................................... 94 1.16.2 Let’s allocate space for structure using malloc()........................ 96 1.16.3 struct tm............................................. 98 1.16.4 Fields packing in structure................................... 103 1.16.5 Nested structures........................................ 105 1.16.6 Bit fields in structure....................................... 106 1.17 C++ classes................................................ 112 1.17.1 Simple example......................................... 112 1.17.2 Class inheritance in C++..................................... 117 1.17.3 Encapsulation in C++...................................... 120 1.17.4 Multiple inheritance in C++................................... 122 1.17.5 C++ virtual methods....................................... 125 1.18 Unions.................................................. 127 1.18.1 Pseudo-random number generator example.......................... 127 1.19 Pointers to functions........................................... 129 1.19.1 GCC................................................ 132 1.20 SIMD.................................................... 133 1.20.1 Vectorization........................................... 133 1.20.2 SIMD strlen() implementation................................ 139 1.21 64 bits................................................... 142 1.21.1 x86-64.............................................. 142 1.21.2 ARM................................................ 149 1.22 C99 restrict................................................ 149 1.23 Inline functions.............................................. 152 2 Couple things to add 154 2.1 LEA instruction.............................................. 154 2.2 Function prologue and epilogue..................................... 154 2.3 npad.................................................... 155 2.4 Signed number representations..................................... 156 2.4.1 Integer overflow......................................... 156 2.5 Arguments passing methods (calling conventions)........................... 157 2.5.1 cdecl............................................... 157 2.5.2 stdcall.............................................. 157 2.5.3 fastcall.............................................. 157 2.5.4 thiscall.............................................. 158 ii CONTENTS 2.5.5 x86-64.............................................. 158 2.5.6 Returning values of float and double type........................... 158 2.6 Position-independent code....................................... 158 2.7 Thread Local Storage........................................... 161 2.8 LD_PRELOAD hack in Linux........................................ 161 3 Finding important/interesting stu in the code 164 3.1 Communication with the outer world.................................. 164 3.1.1 Oen used functions in Windows API.............................. 165 3.1.2 tracer: Intercepting all functions in specific module...................... 165 3.2 String................................................... 166 3.3 Calls to assert().............................................. 166 3.4 Constants................................................. 167 3.4.1 Magic numbers.......................................... 167 3.4.2 Constant searching....................................... 168 3.5 Finding the right instructions...................................... 168 3.6 Suspicious code patterns........................................ 170 3.6.1 Hand-written assembly code.................................. 170 3.7 Using magic numbers while tracing................................... 170 3.8 Other things............................................... 171 3.9 Old-school techniques, nevertheless, interesting to know....................... 171 3.9.1 Memory “snapshots” comparing................................ 171 4 OS-specific 172 4.1 File formats................................................ 172 4.1.1 Win32 PE............................................. 172 4.2 System calls (syscall-s).......................................... 177 4.2.1 Linux............................................... 178 4.2.2 Windows............................................. 178 5 Tools 179 5.0.3 Debugger............................................. 179 5.0.4 System calls tracing....................................... 179 6 Books/blogs worth reading 180 6.1 Books................................................... 180 6.1.1 Windows............................................. 180 6.1.2 C/C++............................................... 180 6.1.3 x86 / x86-64........................................... 180 6.1.4 ARM................................................ 180 6.2 Blogs................................................... 180 6.2.1 Windows............................................. 180 7 More examples 181 7.1 “QR9”: Rubik’s cube inspired amateur crypto-algorithm........................ 181 7.2 SAP.................................................... 209 7.2.1 About