Index

Symbols and Numbers address space layout randomization (ASLR) \ (backlash), 47, 220 bypassing with partial overwrites, / (forward slash), 81, 220 272–273 - (minus sign), 55 exploiting implementation flaws in, + (plus sign), 55 271–272 7-bit integer, 39–40 memory information disclosure 8-bit integer, 38–39 vulnerabilities, 270–271 32-bit system, 263 Adleman, Leonard, 160 32-bit value, 40–41 Advanced Encryption Standard (AES), 64-bit system, 263 133, 150, 152 64-bit value, 40–41 AJAX (Asynchronous JavaScript and 8086 CPU, 114 XML), 57 algorithms A complexity of, 224–225 cryptographic hashing, 164–165 A5/1 stream cipher, 159 Diffie–Helman Key Exchange, A5/2 stream cipher, 159 162–164 ABI (application binary interface), hash, 165 123–124, 259–260 key-scheduling, 151 Abstract Syntax Notation 1 (ASN.1), message digest (MD), 164 53–54 MD4, 165 accept system call, 123 MD5, 133, 165–167 acknowledgment (DHCP packet), 72 RSA, 149, 160–162, 165 acknowledgment flag (ACK), 41 secure hashing algorithm (SHA), active network capture, 20, 280–282. 164, 202 See also passive network SHA-1, 133, 165–166 capture SHA-2, 165 add() function, 124 SHA-3, 168 ADD instruction, 115 signature, 146 add_longs() method, 198 asymmetric, 165 add_numbers() method, 197 cryptographic hashing Address Resolution Protocol (ARP), algorithms, 164–165 6–7, 74–77 message authentication codes, addresses, 4 166–168 32-bit, 5 symmetric, 166 destination, 5 AMD, 114 MAC, 6–8, 74–77 American Fuzzy Lop, 285–286 source, 5 AND instruction, 115 address sanitizer, 243–244 antivirus, 23 application, 3 Big-O notation, 225 content parsers, 4 binary conversions, 90–92 network communication, 4 binary protocols. See also protocols passive network traffic capture, 11 binary endian, 41–42 user interface, 4 bit flags, 41 application binary interface (ABI), Booleans, 41 123–124, 259–260 formats, 53–54 application layer, 3 numeric data, 38–41 apt command line utility, 31 strings, 42–46 arbitrary writing of memory, 253–254 variable binary length data, 47–49 ARM architecture, 42, 118 bind system call, 15 ARP poisoning, 74–77 bit flags, 41 ASCII bit format, 38 character encoding, 42 block ciphers. See also stream ciphers code pages, 44 AES, 150, 152 control characters, 43 common, 152 printable characters, 43 DES, 150–151 text-encoding conversions, 229–230 initialization vector, 154 ASLR. See address space layout modes, 152–155 randomization (ASLR) cipher block chaining, 153–155 ASN.1 (Abstract Syntax Notation 1), Electronic Code Book, 152 53–54 Galois Counter, 155 assembler, 113, 258 padding, 155–156 assemblies, 138 padding oracle attack, 156–158 assembly language, 113 Triple DES, 151 assembly loading, 190–193 Blowfish, 152 asymmetric key cryptography, 159–164. Booleans, 41, 55 See also symmetric key BPF (Berkeley packet filter), 180 cryptography breakpoints, 135, 137 private key, 160 BSD (Berkeley Sockets Distribution), 15 public key, 160 bss data, 120 RSA algorithm, 160–162 Bubble Sort, 224 RSA padding, 162 bucket, 225 trapdoor functions, 160 buffer overflows asymmetric signature algorithms, 165 fixed-length, 211–213 Asynchronous JavaScript and XML heap, 248–249 (AJAX), 57 integer, 214–215 AT&T syntax, 116 stack, 246–248 attributes (XML), 58 variable-length, 211, 213–214 authentication bypass, 209 Burp Suite, 283–284 authorization bypass, 209–210 bytes, 38 automated code, identifying, 133–134 C B C# language, 112, 189, 210 backslash (\), 47, 220 C++ language, 112, 132 base class library, 141 ca.crt file, 203 Base64, 60–61 CALL instruction, 115 Berkeley packet filter (BPF), 180 Camellia, 152 Berkeley Sockets Distribution (BSD), 15 Canape Core, 21–22, 25, 103–105, Berkeley Sockets model, 15, 121 280–281 big endian, 42, 52, 122 Canape.Cli, xxiv, 202

294 Index canonicalization, 220–221 cipher feedback mode, 159 ca.pfx file, 203 cipher text, 146 capture.pcap file, 180 ciphers, 146 capturing network traffic block, 150–159 active method, 20 stream, 158–159 passive method, 12–20 substitution, 147 proxies CJK character sets, 44 HTTP, 29 –35 CLANG C compiler, 243–244 man-in-the-middle, 20 C language, 112, 123, 132, 210, 212 port-forwarding, 21–24 Class files, 141 SOCKS, 24–29 Class.forNa m e() method (Java), 194 resending captured traffic, 182–183 certificate (TLS), 175 system call tracing client random (TLS), 173 Dtrace, 17–18 C library, 268 Process Monitor tool, 18–19 CLR (common language runtime), 137 strace, 16 CMD command, 255 carriage return, 56 CMP instruction, 115, 119 carry flag, 117 code CBC (cipher block chaining), 153–155 error, 262 CDB (debugger), 236–241 executable. See executable codes cdecl, 199 message authentication. See cdll, 199 message authentication Cert Issuer, 200–202 codes (MACs) Cert Subject, 200–201 pages (ASCII), 44 certificate point, 44 authority, 170, 202 section, 120 chain verification, 170–172 collision attacks, 166–168 pinning, 177 collision resistance (hashing revocation list, 171 algorithm), 165 root, 170 command injection, 228 store, 204 common intermediate language (CIL), X.509, 53–54, 169–171, 173 137–138 certmgr.msc, 203 common language runtime (CLR), 137 CFLAGS environment variable, 243 Common Object Request Broker change cipher spec (TLS), 176 Architecture (CORBA), 22 char types, 212 compiled languages, 113 character encoding compilers, 113, 132, 243 ASCII, 43 compression, 20, 108, 217 Unicode, 44–45 conditional branches, 118–119 character mapping, 44–45 CONNECT HTTP method, 30 chat_server.csx script, 187 Connect() method, 185, 192–193 ChatClient.exe (SuperFunkyChat), CONNECT proxy, 32 80–81, 200 connect system call, 15 ChatProgram namespace (.NET), 190 content layer, 8–10 ChatServer.exe (SuperFunkyChat), 80 content parsers, 4 checksum, 93–94, 107 Content-Type values, 57 Chinese characters, 44 control characters (ASCII), 43 chosen plaintext attack, 162 control flow, 118 CIL (common intermediate language), control registers, 117 137–138 Conversations window (Wireshark), Cipher and Hash algorithm, 202 84–85 cipher block chaining (CBC), 153–155 cookies, 212, 273–276

Index 295 CORBA (Common Object Request data expansion attack, 217 Broker Architecture), 22 DataFrame, 108 counter mode, 159 datagram, 5 CPU, 39 datagram socket, 122 8086, 114 Datagram assembly language and, 113 (DTLS), 172 exhaustion attacks, 224–226 data section, 120 instruction set architecture, dates, 49–50, 55 114–116 .ddl extension, 137–138 registers, 116–118 debuggers, 111, 134–137, 236–240, signed integers, 39 243–245, 258–259 x86 architecture, 114–119, 125 debugging, 236–243 crashes analyzing crash in, 238–240 debugging, 238–240 applications, 236 example, 240–243 default or hardcoded finding root cause of, 243–245 credentials, 218 CreateInstance() method (.NET), 191 shell code, 258–259 cron jobs, 254 starting, 236–237 cross-site scripting (XSS), 58 debugging symbols package Crypt32.dll, 132 (dSYM), 131 CryptoAllPermissionCollection.class, 142 DEC instruction, 115 cryptanalysis, 146 decimal numbers, 55 cryptography decompilation, 113 asymmetric key, 159–164 decryption. See also encryption configurable, 226 asymmetric, 160 hashing algorithms, 164–165 block cipher, 150 libraries, 132 breakpoints, 137 symmetric key, 149–159 cipher block chaining, 155, 157–158 CS register, 116, 118 dealing with obfuscation, 143–144 ctypes library (Python), 195 padding, 155–157 curl command line utility, 31 RSA, 161, 165 TLS, 200–202 D Triple DES, 151 default credentials, 218 Dante, 27 default gateway, 8, 66 data defined memory pools, 252–253 controlling flow of, 2 delimited text, 56 encapsulation, 4–7 denial-of-service, 208 endianness of, 41 DEP (data execution prevention), formatting and encoding, 2 267–268 implicit-length, 48–49 DER (Distinguished Encoding inbound, 92 Rules), 53 integrity, 164 DES (Data Encryption Standard), numeric, 38–41 150–151 padded, 49 DES cracker, 151 terminated, 47–48 destination address, 5 transmission, 2, 6–7 destination network address translation variable-length, 56 (DNAT), 24, 68–71 Data Encryption Standard (DES), DHCP. See Dynamic Host 150–151 Configuration Protocol data execution prevention (DEP), (DHCP) 267–268 Diffie, Whitfield, 162

296 Index Diffie–Hellman Key Exchange (DH), EIP register, 116–117, 135 162–164 Electronic Frontier Foundation, 151 Digital Signature Algorithm (DSA), 165 elements (XML), 58 disassembly, 113 ELF (Executable Linking Format), 120, discover (DHCP packet), 71 131, 144 dissector() function, 99 Elliptic Curve Diffie–Hellman dissector.lua file, 98 (ECDH), 202 dissectors elliptic curves, 160 creating, 97 encoding Lua, 99 Base64, 60–61 message packet parsing, 100–103 binary data, 59–61 Wireshark, 95–103 hex, 59–60 Distinguished Encoding Rules percent, 60 (DER), 53 encoding layer, 8–10 DLL extension, 80, 120, 189 encryption, 20, 30. See also decryption DNAT (destination network address AES, 133, 150, 152 translation), 24, 68–71 asymmetric, 160 DNSMasq, 287 block cipher, 150 dnsspoof, 34 breakpoints, 137 (DNS) cipher block chaining, 153–155 protocol, 3 DES, 150–151 Dotfuscator, 143–144 Electronic Code Book, 153 dotnet binary, 81 HTTP connection to, 108 downgrade attack, 176 key, 146 DSA (Digital Signature Algorithm), 165 libraries, 132 DS register, 116, 118 magic constants, 133 dSYM (debugging symbols one-time pad, 148 package), 131 padding, 155 Dtrace, 16–18 public key. See asymmetric key Dynamic Host Configuration Protocol cryptography (DHCP), 63, 66 RSA, 155, 161 packets, 71–72 substitution ciphers, 147 spoofing, 71–74 TLS, 175 –176, 200 –206 dynamic libraries, 130, 195–196 Triple DES, 151 dynamic linking, 113–114, 121 XOR, 108–109, 148–149, 153–154 dynamic reverse engineering encryption libraries, 132 breakpoints, 135, 137 endianness, 41–42 defined, 134 errno, 262 general purpose registers, 136 errors codes, 262 E detecting and correcting, 2 off-by-one, 213 EAX register, 116, 123, 242, 258, 270 verbose, 221–222 EBP register, 116–117, 124 ES register, 116, 118 EBX register, 116, 124 ESI register, 116, 124 ECDH (Elliptic Curve Diffie– ESP register, 116–117, 124, 136, 270 Hellman), 202 eth0, 180 ECX register, 116, 124 Ethernet, 3 EDI register, 116–117, 124 ARP poisoning, 74–75 EDX register, 116, 123–124 frame, 6, 8 EFAULT, 262 MAC addresses, 6, 74 EFLAGS register, 117, 119, 136 network routing, 7–8

Index 297 Ethernet (continued) redirecting traffic to, 30–31 passive network capture, 12–13 simple implementation of, 30–31 simple network, 6 fragmentation, 51–52 Ettercap, 72–75, 287–288 FreeBSD, 16 executable codes FreeCAP, 27 address space layout free-list, 251 randomization, 272 frequency analysis, 147 file formats, 119–120 FS register, 116, 118 function calls in, 123 FTP (), 24, 28 memory corruption and, 210, 246 function monitors, 111 partial overwrites, 272 fuzz testing repurposing, 188–199 defined, 234 in .NET applications, 189–193 mutation fuzzer, 235 in Java applications, 193–195 simplest, 234 ROP gadgets, 269 test cases, 235–236 system calls, 259 tools unmanaged, 195–199 American Fuzzy Lop, 285–286 executable file formats, 119–120, 137 Kali , 286 Executable Linking Format (ELF), 120, Metasploit, 286 131, 144 Scapy, 287 .exe extension, 120, 137–138, 189 Sulley, 287 exit system call, 260–261 Extensible Markup Language G (XML), 58 Extensible Messaging and Presence Galois Counter Mode (GCM), 155 Protocol (XMPP), 58 gateway configuring, 66–67 ARP poisoning, 74–77 F DHCP spoofing, 71–74 false, 55 default, 8, 66 fd argument, 261 forwarding traffic to, 71–77 Federal Information Processing hops, 65 Standard (FIPS), 151 nodes, 64 Feistel network, 151 routing tables on, 65–66 File Transfer Protocol (FTP), 24, 28 GB2312, 44 FILETIME (Windows), 50 GCC compiler, 196 Financial Information Exchange (FIX) GCM (Galois Counter Mode), 155 protocol, 56 GDB (debugger), 236–241 finished packet, 176 General Public License, 14 fixed-length buffer overflows, 211–213 general purpose registers, 116–117, 136 floating-point data, 40–41 GET request, 8, 29 Follow Stream button (Wireshark), 85 GetConstructor method (.NET), 191 Follow TCP Stream view (Wireshark), getDeclaredConstructor() (Java), 195 88–89 GetMethod() method (.NET), 192–193 footers, 4–5 Google, 170, 176 –177 format string vulnerability, 227 GS register, 116, 118 forward slash (/), 81, 220 guard pages, 245 forwarding HTTP proxy. See also GUI registry editor, 67 reverse HTTP proxy GVSP protocol, 182 advantages and disadvantages of, 31 gzip, 217

298 Index H I handshake, 172 IBM, 151 hardcoded credentials, 218 ICS (Internet Connection Sharing), 69 hash table, 225 IDA Pro, 289 hashed message authentication codes analyzing stack variables and (HMAC), 168–169 arguments in, 128 hashing algorithms analyzing strings in, 132 collision resistance, 164 debugger windows, 135–136 cryptographic, 164–165 EIP window, 135 nonlinearity of, 164 ESP window, 136 pre-image resistance, 164 disassembly window, 127–128 secure, 164–165, 202 extracting symbolic information in, SHA-1, 133, 165–166 129–131 SHA-2, 165 free version, 125–128 SHA-3, 168 graph view, 126 HEAD, 29 identifying automated code in, Header, , 4–5 133–134 C, 17, 262 Imports window, 131–132 Ethernet, 6 main interface, 127 HTTP, 24, 32–34 viewing imported libraries in, IP, 6 131–132 system call number, 260 windows, 126–127 TCP, 5, 87 IEEE format, 40–41 UDP, 5 IEEE Standard for Floating-Point heap buffer overflows, 248–249 Arithmetic (IEEE 754), 40 heap implementations, 250–251 ILSpy, 138, 290 heap memory storage, 253 analyzing type in, 140–141 Hellman, Martin, 162 main interface, 139 Hex Dump (Wireshark), 86–95 Search window, 139 determining protocol structure in, implicit-length data, 48–49 88–89 in-band method, 253 information columns in, 87 inbound bytes, 89–92 viewing individual packets in, 87 inbound data, 92 hex editor, 125 INC instruction, 115 hex encoding, 59–60 incorrect resource access, 220–223 Hex Rays, 125 canonicalization, 220–221 high privileges, 254–255 verbose errors, 221–222 HMAC (hashed message authentication inet_pton, 122–123 codes), 168–169 information disclosure, 209 Hopper, 289–290 initialization vector, 154 hops, 65 inner padding block, 168 host header, 24, 32–33 instruction set architecture (ISA), host order, 42 114–116 hosts file, 23, 34 integer overflows, 214–215 Hping, 282 integers HTTP (HyperText Transport Protocol), signed, 39 3, 56 text protocols, 55 host header, 24 unsigned, 38 network protocol analysis, 8–10 variable-length, 39–40 proxies. See also protocols Intel, 114 forwarding, 29–31 Intel syntax, 116 reverse, 32–35 Internet Connection Sharing (ICS), 69

Index 299 Internet layer, 3 K Internet Protocol (IP), 2 , 286 Internet Protocol Suite (IPS) kernel mode, 14 data encapsulation, 4–7 key-scheduling algorithm, 151 data transmission, 6–7 Korean characters, 44 defined, 3 Krypto Analyzer, 134 layers, 3 network routing, 7–8 interpreted languages, 112 L interpreters, 112 least significant bit (LSB), 38 Invoke() method (.NET), 192–193 length-extension attacks, 166–168 IP (Internet Protocol), 2 length-prefixed data, 48 IP address lengths, 107 32-bit, 24 LibPCAP, 278–279 ARP poisoning, 74–77 line feed, 56 data transmission, 6–7 line oriented protocols, 56 destination, 18, 22 linking, 113–114 DNAT, 69–71 link layer, 3, 6 DNS spoofing, 34 Linux, 120 hosts file, 34 ASLR implementation flaws in, 272 NAT, 68 configuring SNAT on, 69 network routing, 7–8 cron jobs, 254 reverse shell, 266 debug symbols, 129 SNAT, 68 debugger, 236–241 SOCKS connection, 25 dynamic libraries, 196 ipconfig command, 69 enabling routing on, 67 iptables command, 69 error codes, 262 IPS. See Internet Protocol Suite (IPS) executable file format, 131 IPv4, 3, 5, 24, 52, 122 loading library on, 197 IPv6, 3, 5–6, 25, 52, 67 SOCKS proxy, 27 ISA (instruction set architecture), strace, 16 114–116 little endian, 42, 122 LLDB (debugger), 236–241 J Load() method (.NET), 190 LoadFrom() method (.NET), 190 Japanese characters, 44 local variables, corrupting, 274–275 Java, 112, 210 localhost, 12 applications, 141–142 low-privileged file writes, 255 reflection types, 194 Lua, 95–103 repurposing codes in, 193–195 Java archive (JAR), 141, 193–194 Java byte code, 137 M Java Decompiler, 288 MAC (Media Access Control) Java Runtime, 27 addresses, 6–7, 8, 74–77 JavaScript, 252 machine code, 112–114, 120, 125 JavaScript Object Notation (JSON), macOS, 16, 27–28, 120 57–58 debug symbols, 129 Java TCP client, 27 debugger, 236–241 Jcc instruction, 115 dynamic libraries, 196 JD-GUI, 142 enabling routing on, 67 JMP instruction, 115, 119 Mach-O format, 120, 131, 144

300 Index MACs. See message authentication off-by-one error, 213 codes (MACs) out-of-bounds buffer indexing, magic constants, 132 216–217 mail application, 3 memory exhaustion attacks, 222–223 main thread, 121 memory index registers, 117 Mallory, 281–282 memory sections, 120 malware, 23 memory-safe languages, 210 man 2 syscall_name command, 16 memory-unsafe languages, 210 managed languages Message Analyzer, 278 Java, 141–142 message authentication codes (MACs) .NET applications, 137–141 collision attacks, 166–168 reverse engineering, 137–144 hashed, 168–169 man-in-the-middle proxy, 20, 201 length-extension attacks, 166–168 masquerading, 68 signature algorithms, 166–168 master secret (TLS), 175 Message command, 101–102 MD algorithm. See message digest message digest (MD) algorithm, 164 (MD) algorithm MD4, 165 Media Access Control (MAC) MD5, 133, 165–167 addresses, 6–7, 8, 74–77 message packet, 100–103 memory Metasploit, 286 arbitrary writing of, 253–254 accessing payloads, 265 heap memory storage, 253 advantages and disadvantages of, information disclosure 265–266 vulnerabilities, 270–271 executing payloads, 266 wasted, 250 generating shell code with, memory canaries (cookies) 265–266 bypassing by corrupting local MethodInfo type (.NET), 192 variables, 274–275 Microsoft, 170 bypassing with stack buffer Microsoft Message Analyzer, 278 underflow, 275–276 MIME (Multipurpose Internet Mail detecting stack overflows with, Extensions), 56–57 273–276 minus sign (-), 55 memory corruption. See also MIPS, 42, 137 vulnerabilities Mitmproxy, 284–285 buffer overflows, 210–215 mnemonic instruction, 114 data expansion attack, 217 modulo arithmetic, 214 dynamic memory allocation modulus, 161, 214 failures, 217 mono binary, 80 exploit mitigations, 266–276 Mono Project, 137 address space layout most significant bit (MSB), 38 randomization, 270–273 MOV instruction, 115 data execution prevention, Mozilla Firefox, 26 266–267 MSCORLIB, 141 return-oriented programming, MS-DOS, 119 268–270 msfvenom tool, 265–266 exploiting, 245–253 multibyte character sets, 44 heap buffer overflows, 248–249 multiplexing, 51–52 stack buffer overflows, 246–248 Multipurpose Internet Mail Extensions memory-safe vs. memory-unsafe (MIME), 56–57 languages, 210 multitasking, 120

Index 301 N newInstance() method (Java), 195 Nmap, 282–283 namespace, 193 NNTP (Network News Transfer name-value pairs (XML), 58 Protocol), 59 nasm assembler, 256, 258, 263 nodes, 1 NAT. See network address gateway, 64 translation (NAT) identifying through addressing, 2 .NET applications no-execute (NX) mitigation, 267 base class library, 141 nonlinearity, 165 file formats, 137–138 nonpersistent denial-of-service, 208 ILSpy, 138–141 NULL, 263–264 reflection binding types, 192 numeric data reflection types, 190 decimal numbers, 55 repurposing codes in, 189–193 floating-point data, 40–41 repurposing executable codes in integers, 55 assembly loading, 190–193 signed integers, 39 using Reflection , 190 text protocols, 55 .NET Core, 80 unsigned integers, 38 .NET Reflector, 290–291 variable-length integers, 39–40 Netcat, 180–182, 234, 282 NX (no-execute) mitigation, 267 NetClientTemplate class, 184–185 netstat -r command, 65 Netwide Assembler, 256 O network, 1 OAEP (Optimal Asymmetric connectivity and protocol testing Encryption Padding), 162 tools obfuscation, 143–144 Hping, 282 octets, 38–40 Netcat, 282 octet-stream, 57 Nmap, 282–283 off-by-one error, 213 monitoring connections with offer (DHCP packet), 71 DTrace, 16–18 one-time pad encryption, 148 proxies, 20–35 open system call, 18 routing, 7–8 OpenSSL, 132 network address, 7, 20, 22, 52–53, 66, operands, 115 71, 123 network address translation (NAT), application binary interface, 68–71 123–124 defined, 68 executable file formats, 119–120 destination, 24, 68 networking interface, 121–124 source, 68–69 processes, 120–121 network communication, 4 sections, 120 Berkeley Sockets model, 15 threads, 120–121 layers, 3 Optimal Asymmetric Encryption man-in-the-middle attack on, 20 Padding (OAEP), 162 symmetric ciphers, 150 OR instruction, 115 user-to-kernel, 15 outbound bytes, 89 network interface, 121–124 outbound traffic, 89 client connection to TCP , 122 outer padding block, 168 TCP client connection to server, out-of-band method, 253 121–122 out-of-bounds buffer indexing, 216–217 Network News Transfer Protocol output feedback mode, 159 (NNTP), 59 overflow flag, 117 network order, 42

302 Index P path, 220 $pc, 239 package-private scoped classes, 193 PDB (program database) file, 129–131 packets, 6 PDP-11, 42 calculating checksum of, 93–94 PDU (protocol data unit), 4 capturing, 83–84 PE (Portable Executable) format, 120, finding, 87–88 134, 144 identifying structure with Hex PEiD, 134 Dump, 86–95 PEM format, 202 sniffing, 12–14 percent encoding, 60 viewing, 87–88 perfect forward secrecy, 177 packing tools, 134 permutation boxes (P-Box), 152 padded data, 49 persistent denial-of-service, 208 padding PGP (Pretty Good Privacy), 169 block ciphers, 155–156 PHP, 255 decryption, 155–157 PKI. See public key infrastructure (PKI) encryption, 155 plain, 57 inner block, 168 plaintext, 146 OA EP, 162 plus sign (+), 54 oracle attack, 156–158 Point-to-Point Protocol (PPP), 3 outer block, 168 POP3 ( 3), 4 RSA encryption, 155, 162 POP instruction, 115 Page Heap, 244–245 port, 2 parity flag, 117 port numbers, 5 Parser class, 106, 185 Portable Executable (PE) format, 120, parser.csx script, 183–184 134, 144 parsing port-forwarding proxy. See also proxies binary conversion and, 90 advantages and disadvantages of, decimal numbers and, 55 23–24 endianness of data and, 41 binding to network addresses, 22 HTTP header, 33 redirecting traffic to, 22–23 message command, 101–102 simple implementation of, 21–22 message packet, 100–103 POSIX, 15 mutation fuzzer and, 235 POSIX/Unix time, 50 protocol, 107–108 POST, 29 Python script for, 91 Post Office Protocol 3 (POP3), 4 traffic, 183 PowerPC, 38 URL, 230 PPP (Point-to-Point Protocol), 3 variable-length integers, 40 Practical Packet Analysis, 14 partial overwrites, 272–273 pre-image resistance (hashing passive network capture algorithm), 165 advantages and disadvantages of, pre-master secret (TLS), 175 19–20 Pretty Good Privacy (PGP), 169 Dtrace, 16–18 printable characters (ASCII), 43 packet sniffing, 12–14 printf function, 227 Process Monitor tool, 17–18 private Connect() method (.NET), 192 strace, 16 private exponent, 161 system call tracing, 14–16 private key, 161, 165 tools PRNGs (pseudorandom number LibPCAP, 278–279 generators), 149 Microsoft Message Analyzer, 278 Process() method, 275–276 TCPDump, 278–279 Process Monitor tool, 17–18 Wireshark, 12–13, 279–280

Index 303 processes, 120–121 public key encryption. See asymmetric processor architectures, 42 key cryptography program database (PDB) file, 129–131 public key infrastructure (PKI), program flow, 118–119 169–172 ProGuard, 143–144 certificate chain verification, , 12 170 –172 PROT_EXEC flag, 257 defined, 169 protocol data unit (PDU), 4 web of trust, 169 protocol stack, 3 X.509 certificates, 169–170 protocols PublicClass class, 189 analysis, 8–10, 105–106 PublicMethod() method, 189 binary, 38–49 PUSH instruction, 115 changing behavior of, 108–109 Python, 210 checksum, 93–94 binary conversions, 90–92 dates, 49–50 calling functions with, 199 determining structure of, 88–89 ctypes library, 195 fragmentation, 51–52 data types, 198 functions of, 2 dissecting protocol with, 90–95 multiplexing, 51–52 loading library with, 197 network address, 52–53 resending captured UDP traffic network connectivity and protocol with, 182–183 testing struct library, 90–92 Hping, 282 Netcat, 282 Q Nmap, 282–283 parsing, 107–108 quoted string, 47–48 security, 145 –178 structured binary formats, 53–54 R tag, length, value (TLV) pattern, 50–51 rand() function, 149 text, 54–58 random number generators, 149 times, 49–50 RAX register, 257–260 unknown parts, 93 RC4 stream cipher, 176 proxies RDP (Remote Desktop Protocol), 51 HTTP, 29 –35 read system call, 15, 18, 122 man-in-the-middle, 20 read_bytes() function, 91 port-forwarding, 21–24 ReadData() function, 108 protocol analysis with, 105–106 ReadOutbound() function, 109 setting up, 103–105 Real Time Messaging Protocol SOCKS, 24–29, 103 (RTMP), 29 traffic analysis with, 103–110 Receive() method (.NET), 193 Proxifier, 27 recv system call, 15, 122–123 pseudo registers, 239 recvfrom system call, 15 pseudorandom number generators reflection, 189 (PRNGs), 149 registers public Connect() method (.NET), 192 control, 117 public exponent, 161 CS, 116, 118 public key, 160–161, 165 DS, 116, 118 Public Key Cryptography Standard EAX, 116, 123, 242, 258, 270 #1.5, 162 EBP, 116–117, 124 Public Key Cryptography Standard #7 EBX, 116, 124 (PKCS#7), 155–156 ECX, 116, 124

304 Index EDI, 116–117, 124 Rijndael, 152 EDX, 116, 123–124 Rivest, Ron, 160 EFLAGS, 117, 119, 136 RMI (Remote Method Invocation), 29 EIP, 116–117, 135 root certificate, 170 ES, 116, 118 ROP (return-oriented programming), ESI, 116, 124 268–270 ESP, 116–117, 124, 136, 270 route print command (Windows), 65 FS, 116, 118 , 7–8 general purpose, 116–117, 136 ARP poisoning, 75–77 GS, 116, 118 configuring, 66–67 memory index, 117 defined, 64 pseudo, 239 enabling DNAT, 70 RAX, 257–260 enabling SNAT, 68–69 scratch, 123 routing selector, 118 on Linux, 67 SS, 116 on macOS, 67 x86 architecture, 116–118 on Windows, 66 remote code execution, 208 routing table, 8, 65–66 Remote Desktop Protocol (RDP), 51 RPC (Remote Procedure Call), 22 Remote Method Invocation (RMI), 29 RSA encryption, 149 Remote Procedure Call (RPC), 22 algorithm, 160–162 request (DHCP packet), 72 padding, 155, 162 Request for Comments (RFCs), 42, signature algorithm, 165 56–57 RSS (Rich Site Summary), 58 request line, 30 Ruby, 210 rerouting traffic, 64–66 Run() function, 187 RESP field, 25 runtime, 137 RET instruction, 115 Ret2Libc, 269 S RETN instruction, 115 return-oriented programming (ROP), say_hello() method, 197 268–270 say_string() method, 197 reverse engineering say_struct() function, 199 dynamic, 134–137 Scan for Hosts (Ettercap), 76 managed languages, 137–144 Scapy, 287 obfuscation, 143–144 scratch registers, 123 resources, 144 scripting languages, 112 static, 125–134 sections (memory), 120 tools secure hashing algorithm (SHA), 164 Hopper, 289–290 SHA-1, 133, 165–166 IDA Pro, 289 SHA-2, 165 ILSpy, 290 SHA-3, 168 Java Decompiler, 288 Secure Sockets Layer (SSL). .NET Reflector, 290–291 See Transport Layer reverse HTTP proxy. See also Security (TLS) forwarding HTTP proxy security, 145 –178 advantages and disadvantages of, 35 encryption, 146 –149 host header, 32–33 public key infrastructure (PKI), redirecting traffic to, 34 169–172 simple implementation of, 33 random number generators, 149 reverse shell, 266 requirements, 145–146 Rich Site Summary (RSS), 58 signature algorithms, 164–169

Index 305 security (continued) Simple Mail Transport Protocol symmetric key cryptography, (SMTP), 3–4, 56, 59 149–159 Simple Network Management Protocol Transport Layer Security, 172–177 (SNMP), 53 segment, 5, 87 sketches, 150 SELECT statement, 229 sniffing, 12–14, 73 selector registers, 118 sockaddr_in structure, 17, 122 self-signed certificate, 170 socket system call, 15 Send() method (.NET), 192–193 SOCKS proxy, 103. See also proxies send system call, 15, 122–123 advantages and disadvantages of, sendfrom system call, 15 28–29 Serpent, 152 Firefox proxy configuration, 26 server random (TLS), 173 Java TCP client, 27 session key, 162 overview, 24 session state, 2 redirecting traffic to, 26–27 set detach-on-fork off command, 237 simple implementation of, 25–26 setAccessible() (Java), 195 versions, 24–25 SGML (Standard Generalized Markup socksProxyHost system property, 27 Language), 58 socksProxyPort system property, 27 SHA. See secure hashing SOH (Start of Header), 56 algorithm (SHA) Solaris, 16, 120 Shamir, Adi, 160 source address, 5 shared key, 163 source code, 112 shell code source network address translation accessing payloads, 265 (SNAT) debugging technique, 258–259 configuring on Linux, 69 generating with Metasploit, 265–266 enabling, 68–69 relative address on 32- and 64-bit $sp, 239 systems, 263 SPARC architecture, 42, 118, 137 reverse shell, 266 spoofing setting breakpoint on, 258–259 DHCP, 71–74 system calls, 259 DNS, 34 exit, 260–261 tools, 287–288 write, 261–263 sprintf string function, 212 writing, 255–266 SQL. See Structured Query shell_bind_tcp, 265 Language (SQL) Shift-JIS, 44 SS register, 116 SHL instruction, 115, 119 stack buffer overflows, 246–248, SHR instruction, 115 273–276 sign flag, 117 stack buffer underflow, 275–276 signature algorithms, 146, 164–169 stack trace, 239–240 asymmetric, 165 stack variables, 128 cryptographic hashing algorithms, Standard Generalized Markup 164–165 Language (SGML), 58 DSA, 165 start address, 120 message authentication codes, Start of Header (SOH), 56 166–168 static linking, 113–114 RSA, 165 static reverse engineering, 125–134. See symmetric, 166 also reverse engineering signed integers, 39 analyzing strings in, 133 simple checksum, 93–94 extracting symbolic information in, 129–131

306 Index identifying key functionality in, system API, 268 129–134 System assembly, 141 stack variables and arguments, 128 system calls stdcall, 199 accept, 123 storage exhaustion attacks, 223–224 bind, 15 strace, 16 connect, 15 strcat string function, 212 exit, 260–261 strcpy string function, 212 open, 18 strcpy_s string function, 212 read, 15, 18, 122 stream ciphers, 158–159. See also block recv, 15, 122–123 ciphers recvfrom, 15 strings, 42–46 send, 15, 122–123 analyzing, 132 sendfrom, 15 ASCII standard, 42–44 shell code, 259–262 Strip tool, 131 socket, 15 struct library (Python), 90 tracing, 14–19 Structure class, 199 Unix-like systems, 15–16, 122 structured binary formats, 53–54 write, 15, 18, 122, 261–263 Structured Query Language (SQL) system function, 228 injection, 228–229 System.Activator class (.NET), 191 Server, 229 System.Reflection.Assembly class structured text formats, 56–58 (.NET), 190 SUB instruction, 115 System.Reflection.ConstructorInfo class subroutine calling, 118–119 (.NET), 190 substitution boxes (S-Box), 152 System.Reflection.FieldInfo class substitution ciphers, 147 (.NET), 190 substitution-permutation network, 152 System.Reflection.MethodInfo class Sulley, 287 (.NET), 190 SuperFunkyChat System.Reflection.PropertyInfo class analysis proxy (.NET), 190 captured traffic, 183–187 System.Type class (.NET), 190 simple network client, 184–186 simple server, 186–188 T ChatClient, 81, 83–84, 106, 200 ChatServer, 80, 106 tag, length, value (TLV) pattern, commands, 81 50–51, 89, 94–95 communicating between clients, 81 TCP. See Transmission Control dissectors, 95–103 Protocol (TCP) parser code for, 107 TCPDump, 278–279 starting clients, 80–81 TCP/IP, 2, 9–10, 121, 262 starting the server, 80 TCP/IP Guide, 16 UDP mode, 97 TcpNetworkListener (ILSpy), 140 switch device, 6 terminated data, 47–48 symbolic information, 129–131 terminated text, 56 symmetric key cryptography, 149. TEST instruction, 115, 119 See also asymmetric key testy virtual buffer (TVB), 99 cryptography text protocols, 54 block ciphers, 150–159 Booleans, 55 stream ciphers, 158–159 dates, 55 symmetric signature algorithms, 166 numeric data, 55 synchronize flag (SYN), 41 structured text formats, 56–58

Index 307 text protocols (continued) traffic times, 55 analysis using proxy, 103 variable-length data, 55 capturing text-encoding character replacement, active method, 20 229–231 HTTP, 29 –35 threads, 120–121 man-in-the-middle, 20 times, 49–50, 55 passive method, 12–20 TLS. See Transport Layer Security (TLS) port-forwarding, 21–24 TLS Record protocol, 172 proxies, 20–35 TLV (tag, length, value) pattern, SOCKS, 24–29 50–51, 89, 94–95 system call tracing, 14–19 ToDataString() method, 186 capturing tools token, 56 Dtrace, 17–18 tools Netcat, 180–182 for active network capture and Process Monitor tool, 18–19 analysis strace, 16 Canape, 280–281 generating, 83–84 Canape Core, 281 outbound, 89 Mallory, 281–282 Transmission Control Protocol (TCP), fuzz testing 2–3, 21 American Fuzzy Lop, 285–286 bit flags, 41 Kali Linux, 286 client connection to server, 121–123 Metasploit, 286 header, 5, 87 Scapy, 286 HTTP proxy, 30 Sulley, 286 packets, 87–88 network connectivity and protocol port numbers, 5 testing port-forwarding proxy, 21–22, 201 Hping, 282 reading contents of sessions, 85–86 Netcat, 282 reverse shell, 265–266 Nmap, 282–283 SOCKS proxy, 24–28 for network spoofing and stream, 13–14 redirection transport layer, 3, 6, 8–10 DNSMasq, 287 Transport Layer Security (TLS) Ettercap, 287–288 certificate pinning, 177 for passive network capture and client certificate, 175 analysis decryption, 201–202 LibPCAP, 278–279 encryption, 175–176, 200–201 Microsoft Message Analyzer, 278 endpoint authentication, 174–175 TCPDump, 278–279 forcing TLS 1.2, 202 reverse engineering handshake, 172–173 Hopper, 289–290 initial negotiation, 173 IDA Pro, 289 perfect forward secrecy, 177 ILSpy, 290 replacing certificate in, 202–206 Java Decompiler, 288 security requirements, 176 –177 .NET Reflector, 290–291 TLS Record protocol, 172 for web application testing trapdoor functions, 160 Burp Suite, 283–284 Triple DES, 151 Mitmproxy, 284–285 true, 55 Zed Attack Proxy, 284 trusted root certification traceconnect.d file, 16 authorities, 204 traceroute, 64–65 Tshark, 180–182 tracert (Windows), 64–65 TVB (testy virtual buffer), 99

308 Index Twofish, 152 user interface (UI), 4 two’s complement, 39 user mode, 14 user-after-free vulnerability, 249–250 U UTF (Unicode Transformation Format), 44–45 UCS (Universal Character Set), 44–45 UTF-8, 45–46 UDP. See (UDP) UI (user interface), 4 V uname command, 263–264 variable binary length data Unicode implicit-length data, 48–49 character encoding, 44–45 length-prefixed data, 48 character mapping, 44–45 padded data, 49 UCS-2/UTF-16, 45 terminated data, 47–48 UCS-4/UTF-32, 45 variable-length buffer overflows, 211, Unicode Transformation Format 213–214 (UTF), 44–45 variable-length data, 56 Unified Sniffing mode (Ettercap), 76 variable-length integers, 39–40 Uniform Request Identifier (URI), verbose errors, 221–222 30, 32 Verisign, 170 uninitialized data, 120 virtual function table, 242, 248–249 Universal Character Set (UCS), 44–45 virtual hosts, 24 Unix-like systems, 5 virtual machine, 137 ASLR implementation flaws in, 272 VirtualAlloc, 250 AT&T syntax, 116 Visual C++, 129 command injection, 228 vulnerabilities command line utilities on, 31 authentication checking, 226 configuring DNAT on, 70 classes Dtrace, 16 authentication bypass, 209 enabling routing on, 67 authorization bypass, 209–210 error codes, 262 denial-of-service, 208 executable format, 120 information disclosure, 209 hosts file, 23 remote code execution, 208 read and write calls, 122 command injection, 228 routing tables on, 65 CPU exhaustion attacks system calls, 15–16, 122 algorithmic complexity, traceroute, 64 224–225 Unk2 value, 93–95 configurable cryptography, unmanaged executables, 195–199 224–225 dynamic libraries, 195–196 default or hardcoded unsafe keyword, 210 credentials, 218 unsigned integers, 38 exploiting UPX, 134 arbitrary writing of memory, URI (Uniform Request Identifier), 253–254 30, 32 defined memory pool User Datagram Protocol (UDP), 3 allocations, 252–253 captured traffic, 182–183 heap layout manipulation, dissectors, 98–99 249–250 payload and header, 5 heap memory storage, 253 port forwading, 21 high-privileged file writes, socket, 122 254–256 user enumeration, 218–219 low-privileged file writes, 255

Index 309 memory corruption, 245–253 Winsock library, 121 user-after-free vulnerability, XP SP2, 270 249–250 WinDump, 278 format string, 227 WinPcap, 278 fuzz testing, 234–236 Winsock, 121 incorrect resource access Wireshark, 12–14, 81, 279–280 canonicalization, 220–221 basic analysis, 84–85 verbose errors, 221–222 capture interfaces dialog, 82–83 memory corruption Conversations window, 84–85 buffer overflows, 210–215 dissectors, 95–103 data expansion attack, 217 generating network traffic in, dynamic memory allocation 83–84 failures, 217 Hex Dump view, 86–95 exploit mitigations, 267–268 main window, 82 memory-safe vs. memory-unsafe reading contents of TCP sessions languages, 210 in, 85–86 out-of-bounds buffer indexing, Tshark command line version, 216–217 180–182 memory exhaustion attacks, WOT (web of trust), 169 222–223 write system call, 15, 18, 122, 261–263 shell code, 255–266 WriteData() function, 108 SQL injection, 228–229 WritePackets() method, 22 storage exhaustion attacks, ws2_32.dll Windows network library, 223–224 130–131 text-encoding character replacement, 229–231 X triaging, 236–245 user enumeration, 218–219 X.509 certificates, 53–54, 169–171, 173 X.680 series, 53 x86 architecture, 42, 125 W history, 114 W3C, 58 instruction mnemonics, 115 web application testing tools, 283–285 instruction set architecture, Burp Suite, 283–284 114–116 Mitmproxy, 284–285 mnemonic forms, 115 Zed Attack Proxy, 284 program flow, 118–119 web of trust (WOT), 169 registers, 116–118 wget, 31 xcalc, 228 windll, 199 XML Schema, 58 Windows XOR encryption, 108–109, 148–149, ASLR implementation flaws in, 272 153–154 calling functions with Python XOR instruction, 115 on, 199 XOR parameter, 108–109 certificate manager, 203 xp_cmdshell function, 229 debug symbols, 129 xxd tool, 90, 181 debugger, 236–241, 244–245 dynamic link libraries, 196 Z enabling routing on, 67 FILETIME, 50 Zed Attack Proxy (ZAP), 284 loading library on, 197 zero flag, 117 Page Heap, 244–245 ZLib compression library, 132 registry, 67

310 Index