Symbols & Numbers A

INDEX

Symbols & Numbers Accumulator (EAX) register, 24, 346 zeroing, 368 & (ampersand) ACK flag, 223 for address-of operator, 45 filter for, 260 for background process, 347 active sniffing, 239–251 < > (angle brackets), for include add instruction, 293 file, 91 Address Resolution Protocol (ARP), = (assignment operator), 12 219, 240 * (asterisk), for pointers, 43 cache poisoning, 240 \ (backslash), for escaped redirection, 240 character, 180 reply messages, 219 { } (curly braces), for set of spoofing, 243 instructions, 8, 9 request messages, 219 $ (dollar sign qualifier), and direct address-of operator, 45, 47, 98 parameter access, 180 addressof.c program, 46 == (equal to operator), 14 addressof2.c program, 47 ! (exclamation point), 14 addr_struct.c file, 348–349 > (greater than operator), 14 administrator account, 88. See also >= (greater than or equal to root, user operator), 14 AES (Rijndael), 398 < (less than operator), 14 AF_INET, socket address structure <= (less than or equal to operator), 14 for, 201–202 != (not equal to operator), 14 aircrack, 448–449 ! (not operator), 14 AirSnort, 439 % (percent sign), for format algorithm, efficiency of, 398 parameter, 48 algorithmic run time, 397–398 " (quotation marks), for include ampersand (&) files, 91 for address-of operator, 45 ; (semicolon), for instruction end, 8 for background process, 347 $1 variable, 31 amplification attacks, 257 8-by-8 S-box, 435 AND bitwise operation, 366 32-bit addressing scheme, 22 and instruction, 293 64-bit addressing scheme, 22 AND operator, 14–15 404 HTTP response, 213 < > (angle brackets), for include file, 91 A application layer (OSI), 196 accept() function, 199, 206 argument vector, 59 access mode for file, 84 arithmetic operators, 12–14 ARP. See Address Resolution Protocol Base Pointer (EBP) register, 24, 31, (ARP) 70, 73, 344–345 arp_cmdline() function, 246 saving current values, 342 ARPhdr structure, 245–246 BASH shell, 133–150, 332 arp_initdata() function, 246 command substitution, 254 arp_send() function, 249 investigations with, 380–384 arpspoof.c program, 249–250, 408 for loops, 141–142 arp_validatedata() function, 246 script to send ARP replies, 243–244 arp_verbose() function, 246 BB84, 396 arrays in C, 38 bc calculator program, 30 artistic expression, programming as, 2 beauty, in mathematics, 3 ASCII, 33–34 Bennett, Charles, 396 function for converting to Berkeley Packet Filter (BPF), 259 integer, 59 big-endian byte order, 202 for IP address, conversion, 203 big-oh notation, 398 ASLR, 379–380, 385, 388 bind call, host_addr structure for, 205 aslr_demo.c program, 380 bind() function, 199 aslr_execl.c program, 389 bind_port.c program, 303–304 aslr_execl_exploit.c program, bind_port.s program, 306–307 390–391 bind_shell.s program, 312–314 assembler, 7 bind_shell1.s program, 308 assembly language, 7, 22, 25–37 /bin/sh, 359 GDB examine command to display system call to execute, 295 instructions, 30 birthday paradox, 437 if-then-else structure in, 32 bitwise operations, 84 Linux system calls in, 284–286 bitwise.c program, 84–85 for shellcode, 282–286 block cipher, 398 syntax, 22 Blowfish, 398 assignment operator (=), 12 Bluesmack, 256 asterisk (*), for pointers, 43 Bluetooth protocol, 256 asymmetric encryption, 400–405 bootable LiveCD. See LiveCD asymptotic notation, 398 botnet, 258 AT&T syntax for assembly bots, 258 language, 22 BPF (Berkeley Packet Filter), 259 atoi() function, 59 Brassard, Gilles, 396 auth_overflow.c program, 122–125 breakpoint, 24, 27, 39, 342, 343 auth_overflow2.c program, 126–133 broadcast address, for amplification attacks, 257 B brute-force attacks, 436–437 exhaustive, 422–423 backslash (\), for escaped bss segment, 69, 77 character, 180 for C variable storage, 75 backtrace bt command, 40 of nested function calls, 66 buffer overflows, 119–133, 251 of stack, 40, 61, 274 command substitution and Perl to bandwidth, ping flood to generate, 134–135 consume, 257 in memory segments, 150–167 Base (EBX) register, 24, 344–345 notesearch.c program vulner- saving current values, 342 ability to, 137–142 stack-based vulnerabilities, 122–133

456 INDEX buffer overrun, 119 close() function, file descriptor for, 82 buffers, 38 closed ports, response with SYN/ACK program restrictions on, 363–376 packets, 268 buildarp() function, 246 cmp operation, 26, 32, 310, 311 byte, 21 code segment, 69 byte counter, incrementing, 177 CodeRed worm, 117, 319 byte order of architecture, 30 command line, Perl to execute conversion, 238 instructions, 133 command prompt, indicator of back- C ground jobs, 332 command-line arguments, 58–61 C compilers, 19 commandline.c program, 58–59 free, 20 commands variable data types and, 58 running single as root user, 88 C programming language substitution and Perl to generate address-of operator, 45 buffer overflows, 134–135 arithmetic operators shorthand, 13 comments, in C program, 19 vs. assembly language, 282 comparison operators, 14–15 Boolean operations, 15 compiled code, 20 comments, 19 compiler, 7 control structures, 309–314 computational power, vs. storage file access in, 81–86 space, 424 functions in, 16 computational security, 396 memory segments, 75–77 conditional probability, 114 programmer responsibility for data conditional statements, integrity, 119 variables in, 14 call instruction, 287 confusion, 399 null bytes from, 290 connect() function, 199, 213, 314 callback function, 235 connect-back shellcode, 314–318 carriage return, for line termination connectback-shell.s program, in HTTP, 209 314–315 caught_packet() function, 236, 237 connectivity, ICMP to test for, 221 CD with book. See LiveCD constants, 12 cdq instruction, 302 constructors (.ctors), table char data type, 12, 43 sections for, 184–188 character array (C), 38 convert.c program, 59–60 char_array executable binary, 38 Copyright Act, 118 char_array.c program, 38 core dump, 289 check_authentication() function, Counter (ECX) register, 24 122, 125 countermeasures stack frame for, 128–129 for attack detections, 320 child process, spawning root shell buffer restrictions, 363–376 with, 346 hardening, 376 chmod command, 88 log files and, 334–336 chown command, 90 nonexecutable stack, 376–379 chsh command, 89 overlooking obvious, 336–347 cleanup() function, 184 system daemons, 321–328 client_addr_ptr, 348, 349 tools, 328–333 and crash, 353 crackers, 3

INDEX 457 crash, 61, 128 stack variable, 76 from buffer overflow, 120 variables, 12 and client_addr_ptr, 353 decode_ethernet() function, 237 by DoS attacks, 251 decode_ip() function, 237 from out-of-bound memory decode_sniff.c file, 235–239 addresses, 60 decode_tcp() function, 236, 237 CRC32 (cyclic redundancy checksum) decoherence, 399 function, 434 default gateway, ARP redirection criminal activity, 451–452 and, 241 crypt() function, 153, 418 Denial of Service (DoS), 251–258 salt values, 423 amplification attacks, 257 cryptanalysis, 393 distributed DoS flooding, 258 crypt_crack.c program, 420 ping flooding, 257 cryptography, 393 ping of death, 256 laws restricting, 3 SYN flooding, 252–256 cryptology, 393 teardrop, 256 crypt_test.c program, 418 dereference operator, 47 .ctors (constructors), table sections loading address of, 297 for, 184–188 DES, 398 curly braces ({ }), for set of Destination Index (EDI) register, 24 instructions, 8, 9 destructors (.dtors) current_time variable, 97 displaying contents, 185 custom signal handlers, 322 overwriting section with address of cut command, 143–144 injected shellcode, 190 cyclic redundancy checksum table sections for, 184–188 (CRC32) function, 434 Deutsch, Peter, 2 Cynosure, 118 dictionary attacks, 419–422 dictionary tables, IV-based D decryption, 438 diffusion, 399 daemon() function, 321 Digital Millennium Copyright Act daemons, 321 (DCMA) of 1998, 3 Data (EDX) register, 24, 361 direct parameter access, 180–182 data integrity, programmer responsi- directory, for include files, 91 bility for, 119 Dissembler, 454 data segment, 69 distributed DoS flooding, 258 for C variable storage, 75 division, remainder after, 12 data types, of variables, 12 DNS (Domain Name Service), 210 datafile buffer, 151–152 dollar sign qualifier ($), and direct datagram socket, 198 parameter access, 180 data-link layer (OSI), 196, 197 DoS. See Denial of Service (DoS) for web browser, 217, 218–219 dotted-number notation, 203 datatype_sizes.c program, 42–43 double word (DWORD), 29 DCMA (Digital Millennium Copy- converting to quadword, 302 right Act) of 1998, 3 drop_privs.c program, 300 debuggers, 23–24 dsniff program, 226, 249, 454 declaring .dtors (destructors) destructor function, 184 displaying contents, 185 functions with data type of return overwriting section with address of value, 16–17 injected shellcode, 190 heap variable, 76 table sections for, 184–188

458 INDEX dtors_sample.c program, 184 ESI (Source Index) register, 24 dump() function, 204 ESP (Stack Pointer) register, 24, 33, dup2 system call, 307 70, 73 DWORD (double word), 29 shellcode and, 367 converting to quadword, 302 /etc/passwd file, 89, 153 /etc/services file, default ports in, E 207–208 ETHERhdr structure, 245–246 EAX (Accumulator) register, 24, Ethernet, 218, 230 312, 346 header for, 230 zeroing, 368 length of, 231 EBP (Base Pointer) register, 24, 31, Euclidean algorithm, 400–401 70, 73, 344–345 extended, 401–402 saving current values, 342 Euler’s totient function, 400, 403 EBX (Base) register, 24, 312, 344–345 examine command (GDB) saving current values, 342 for ASCII table lookup, 34–35 ec_malloc() function, 91 to display disassembled ECX (Counter) register, 24 instructions, 30 EDI (Destination Index) register, 24 display unit size for, 28–29 EDX (Data) register, 24, 361 for memory, 27–28 EFLAGS register, 25 exclamation point (!), 14 EIP register. See Instruction Pointer execl() function, 149, 389, 390 (EIP) register execle() function, 149 elegance, 2, 6 exec_shell.c program, 296 encapsulation, 196 exec_shell.s program, 297 encoded_sockreuserestore_dbg.s file, executable binaries, 21 360–361 creating from assembly code, 286 encryption, 393 execute permission, 87 asymmetric, 400–405 execution flow, controlling, 118 maximum allowable key size in execution of arbitrary code, 118 exported software, 394 execve() function, 295–296, 388–389 symmetric, 398–400 structure for, 298 wireless 802.11b, 433–436 exhaustive brute-force attacks, env command, 142 422–423 environment variables, 142 exit, automatically executing displaying location, 146 function on, 184 for exploiting, 148 exit() function, 191, 286 PATH, 172 address of, 192 placing shellcode in, 188 exploit buffer, 332 randomization of stack exploit programs, 329 location, 380 exploit scripts, 328–333 for storing string, 378 exploit tools, 329 epoch, 97 exploitation, 115 equal to operator (==), 14 with BASH, 133–150 error checking, for malloc(), 79, 80–81 buffer overflows, 119–133 errorchecked_heap.c program, 80–81 format strings, 167–193 errors, off-by-one, 116–117 direct parameter access, escape sequences, 48 180–182 escaped character, backslash (\) reading from arbitrary memory for, 180 addresses, 172

INDEX 459 exploitation, continued fingerprints format strings, continued fuzzy, 413–417 with short writes, 182–183 host, for SSH, 410–413 vulnerability, 170–171 firewalls, and port-binding writing to arbitrary memory shellcode, 314 addresses, 173–179 first-in, last-out (FILO) ordering, 70 general techniques, 118 firstprog.c program, 19 heap-based overflow, 150–155 float data type, 12, 13, 43 jackpot() function as target, flood services, by DoS attacks, 251 160–166 flow of execution, operations overflowing function pointers, controlling, 26 156–167 Fluhrer, Mantin, and Shamir (FMS) overwriting global offset table, attack, 439–449 190–193 fms.c program, 443–445 without log file, 352–354 fmt_strings.c program, 48–49 exploit_notesearch.c program, 121 fmt_uncommon.c program, 168 exploit_notesearch_env.c program, fmt_vuln.c program, 170–171 149–150 fopen() function, 419 extended Euclidian algorithm, for loops, 10–11 401–402 with assembly instructions, 309–310 to fill buffer, 138 F foreground (fg) command, 158, 332 fatal errors, displaying, 228 forging source address, 239 fatal() function, 83, 91 fork() function, 149, 346 fcntl_flags.c program, 85–86 format parameters, 48 fcntl.h file, 84 format strings, 167–193 Feistel network, for DES, 399 memory for, 171 Felten, Edward, 3 for printf() function, 48–51 fencepost error, 116 short writes for exploits, 182–183 ffp, 454 simplifying exploits with direct fg (foreground) command, 158, 332 parameter access, 180–182 fgets() function, 419 vulnerability, 170–171 field-width option, for format FP (frame pointer), 70 parameter, 49 fprintf() function, for error file access, in C, 81–86 messages, 79 file descriptors, 81 fraggle attacks, 257 duplicating standard, 307–309 fragmenting packets, 221 in Unix, 283 IPv6, 256 File Not Found HTTP response, 213 frame pointer (FP), 70 file permissions, 87–88 free() function, 77, 79, 152 File Transfer Protocol (FTP), 222 free speech, 4 server, 226 FTP (File Transfer Protocol), 222 filestreams, 81 server, 226 FILO (first-in, last-out) ordering, 70 funcptr_example.c program, 100 filter, for packets, 259 functionality, expansion, and FIN scans, 264–265 errors, 117 after kernel modification, 268 functions, 16–19 before kernel modification, automatically executing on 267–268 exit, 184 find_jmpesp.c program, 386 breakpoint in, 24

460 INDEX declaring as void, 17 gethostbyname() function, 210, 211 for error checking, 80–81 getuid() function, 89, 92 libraries of, 19 Glen, Peter, 454 local variables for, 62 glibc, heap memory management, 152 memory, string pointer global offset table (GOT), referencing, 228 overwriting, 190–193 pointers, 100–101 global variables, 63, 64, 75 calling without overwriting, 157 memory addresses, 69 overflowing, 156–167 memory segment for, 69 prologue, 27, 71, 132 GNU Compiler Collection (GCC), 20. saving current register See also GDB debugger values, 342 compiler, GDB access to source prototype, 17 code, 26 for string manipulation, 39 objdump program, 21, 184, 185 fuzzy fingerprints, 413–417 Goldberg, Ian, 394 GOT (global offset table), G overwriting, 190–193 greater than operator (>), 14 game_of_chance.c program, 102–113, greater than or equal to 156–167 operator (>=), 14 gateway, 241 greatest common divisor (GCD), 401 GCC. See GNU Compiler Collection Greece, ancient, 3 (GCC) grep command, 21, 143–144 GCD (greatest common divisor), 401 to find kernel code sending reset GDB debugger, 23–24 packets, 267 address-of operator, 45 Grimes, Mark, 242, 454 analysis with, 273–275 groups, file permissions for, 87 to control running tinywebd Grover, Lov, 399–400 process, 350–352 to debug daemon child process, 330–331 H disassembly syntax, 25 Hacker Ethic, 2 displaying local variables in stack hacking, 272–280 frame, 66 analysis with GDB, 273–275 examine command attitudes toward, 451 for ASCII table lookup, 34–35 and compiled program, 21 to display disassembled cycle of innovation, 319 instructions, 30 essence of, 1–2 for memory, 27–28 origins, 2 investigating core with, 289–290 port-binding shellcode, 278–280 investigations with, 380–384 as problem solving, 5 print command, 31 and program crash control, 121 shorthand commands, 28 hacking.h file, adding to, 204 stepi command, 384 hacking-network.h file, 209–210, 231, .gdbinit file, 25 232, 272–273 general-purpose registers, 24 hacks, 6 GET command (HTTP), 208 half-open scan, 264 getenv() function, 146 handle_connection() function, 216, 342 getenvaddr.c program, 147–148, 172 breakpoint in function, 274–275 geteuid() function, 89 handle_shutdown() function, 328

INDEX 461 hardware addresses, 218 if-then-else structure, 8–9 hash lookup table, 423–424 in assembly language, 32 head command, 143–144 in_addr structure, 203 HEAD command (HTTP), 208 connection IP address in, 315–316 heap, 70 inc operation, 25, 36 allocation function for, 75 include file, for functions, 91 buffer overflows in, 150–155 incoming connection growth of, 75 C function to accept, 199 memory allocation, 77 listening for, 316 variable incrementing variable values, 13–14 declaring, 76 inet_aton() function, 203 space allocated for, 77 inet_ntoa() function, 203, 206 heap_example.c program, 77–80 info register eip command, 28 Heisenberg uncertainty principle, 395 information theory, 394–396 “Hello, world!”, program to print, 19 initialization vector (IV) helloworld1.s program, 287–288 gathering, 449 helloworld3.s program, 294 for WEP, 434, 437, 440 helloworld.asm program, 285–286 decryption dictionary tables helloworld.c, rewrite in assembly, 285 based on, 438 Herfurt, Martin, 256 input, length check or hexadecimal dump, of standard restriction on, 120 shellcode, 368 input size, for algorithm, 397 hexadecimal notation, 21 input validation, 365 high-level languages, conversion to input.c program, 50 machine language, 7 input_name() function, 156 Holtmann, Marcel, 256 Instruction Pointer (EIP) register, 25, host fingerprints, for SSH, 410–413 27, 40, 43, 69, 73 host key, retrieving from servers, 414 assembly instructions and, 287 host_addr structure, for bind call, 205 crash from attempt to restore, 133 hostent structure, 210–211 examining memory for, 28 host_lookup.c file, 211–212 as pointer, 43 htonl() function, 202 program execution and, 69 htons() function, 203, 205 shellcode and, 367 HTTP (Hypertext Transfer Protocol), int data type, 12 197, 207–208, 222 int instruction, 285 hybrid ciphers, 406–417 integers, function for converting Hypertext Transfer Protocol (HTTP), ASCII to, 59 197, 207–208, 222 Intel syntax for assembly language, 22, 23, 25 I Internet Control Message Protocol (ICMP), 220–221 ICMP. See Internet Control Message amplification attacks with Protocol (ICMP) packets, 257 id command, 88 echo messages, 256 idle scanning, 265–266 Echo Request, 221 IDS (intrusion detection systems), Internet Datagram header, 232 4, 354 Internet Explorer, zero-day VML if statement, in BASH, 381 vulnerability, 119 ifconfig command, 316 Internet Information Server for promiscuous mode setting, 224 (Microsoft IIS), 117

462 INDEX Internet Protocol (IP), 220 L addresses, 197, 220 LaMacchia, David, 118 conversion, 203 LaMacchia Loophole, 117–118 data-link layer and, 218–219 Laurie, Adam, 256 in logs, 348 LB (local base) pointer, 70 redirection, 438–439 lea (Load Effective Address) spoofing logged, 348–352 instruction, 35, 296 IDs, predictable, 265 least significant byte, 174, 178 structure, 231 leave instruction, 132 interrupt 0x80, 285 less than operator (<), 14 intrusion detection systems (IDS), less than or equal to operator (<=), 14 4, 354 libc, returning into, 376–377 intrusion prevention systems libc function, finding location, (IPS), 354 377–378 intrusions libnet library (C), 244 log files and detection, 334–336 documentation for functions, overlooking obvious, 336–347 248–249 IP. See Internet Protocol (IP) release, 254 IPS (intrusion prevention structures, 263 systems), 354 libnet_build_arp() function, 248–249 iptables command, 407 libnet_build_ethernet() function, 248 IPv6 packets, fragmented, 256 libnet_close_link_interface() IV. See initialization vector (IV) function, 249 libnet-config program, 254 J libnet_destroy_packet() function, 249 jackpot() function, as exploit target, libnet_get_hwaddr() function, 251 160–166 libnet_get_ipaddr() function, 251 jle operation, 32, 310 libnet_get_prand() function, 252 jmp esp instruction, 385 libnet_host_lookup() function, 251 predictable address for, 388 libnet_init_packet() function, 248 jmp short instruction, 292 libnet_open_link_interface() jobs command, 332 function, 248 John the Ripper, 422, 454 libnet_seed_prand() function, 252 jumps in assembly language, 26 libpcap sniffer, 228–230, 235, 260 conditional, 310 libraries unconditional, 36 documentation, 251 of functions, 19 Linux environment, 19 K booting from CD, 4 Key Scheduling Algorithm (KSA), nonexecutable stack, 376 435, 440–442 system calls in assembly, 284–286 keystream, 398 linux-gate reuse, 437–438 bouncing off, 384–388 kill command, 323, 324 execution jump to, 386 knowledge, and morality, 4 linux/net.h include file, 304–305 known_hosts file, 410 listen() function, 199, 206 KSA (Key Scheduling Algorithm), little-endian byte order, 29, 93, 316 435, 440–442

INDEX 463 LiveCD, 4, 19 mark_break.s file, 342–343 John the Ripper, 422 mark_restore.s file, 345 Nemesis, 242 mark.s file, 339 /usr/src/mitm-ssh, 407 mathematics, beauty in, 3 Load Effective Address instruction Maxwell, James, 321 (lea), 35, 296 Media Access Control (MAC) local base (LB) pointer, 70 addresses, 218 local variables, 62 memcpy() function, 139 displaying in stack frame, 66 memory, 21–22 memory addresses, 69 addresses memory saved for, 130 hexadecimal notation for, 21 localtime_r() function, 97 order of, 75 log files reading from arbitrary, 172 exploitation without, 352–354 writing to arbitrary, 173–179 and intrusion detection, 334–336 allocation for void pointer, 57 logic, as art form, 2 corruption, 118 long keyword, 42 efficiency, vs. time for coding, 6 loopback address, 217, 317–318 for format string, 171 loopback_shell_restore.s file, 346–347 GDB debugger to examine, 27–28 loopback_shell.s file, 318 instructions to set up, 27 looping for local variables, 130 for, 10–11 predicting address, 147 while/until, 9–10 segmentation, 69–81, 285 lseek() function, 95 segments, 60 LSFR (stream cipher), 398 buffer overflows in, 150–167 in C, 75–77 M for variables, 119 violation, 60 MAC (Media Access Control) memory_segments.c program, 75–77 addresses, 218, 230 memset() function, 138 machine language, 7 Microsoft, IIS webserver, 117 control structures, 309 MIT model railroad club, 2 converting assembly to, 288 MitM (man-in-the-middle) attacks, viewing for main() function, 21 406–410 main() function, 19 mitm-ssh package, 407, 454 command-line argument modulo reduction, 12 access in, 58 morality, and knowledge, 4 disassembly of, 27 mov instruction, 25, 33, 285 viewing machine code for, 21 variations, 292 malloc() function, 75, 76, 77, 79 error checking for, 80–81 man page N for arpspoof, 249 %n format parameter, 48, 168–169, 173 for ASCII, 33–34 nasm assembler, 286, 288, 454 for daemon(), 321 Nathan, Jeff, 242, 454 for exec(), 388 nc program, 279 for libnet, 248, 251 ndisasm tool, 288 for write(), 283 negative numbers, 42 man-in-the-middle (MitM) attacks, Nemesis, 242–248, 454 406–410

464 INDEX nemesis_arp() function, 245 sockets, 198–217 nemesis-arp.c file, 244–245 address conversion, 203 nemesis.h file, 245–246 addresses, 200–202 nemesis-proto_arp.c file, 246–248 functions, 199–200 nested function calls, 62 network byte order, 202–203 netcat program, 279, 309, 316, 332 server example, 203–207 netdb.h file, 210 tinyweb server, 213–217 netinet/in.h file, 201–202 web client, 207–213 netstat program, 309 TCP/IP hijacking, 258–263 Netwide Assembler (NASM), 454 RST hijacking, 259–263 network byte order, 202–203, 316 newline character, for HTTP line network layer (OSI), 196, 197 termination, 209 for web browser, 217, 220–221 Newsham, Tim, 436–437 network sniffing, 224–251, 393 nexti (next instruction) command, 31 active sniffing, 239–251 NFS (number field sieve), 404 decoding layers, 230–239 nm command, 159, 184, 185 libpcap sniffer, 228–230 nmap (port scanning tool), 264 raw socket sniffer, 226–227 No Electronic Theft Act, 118 networking, 195 nonorthogonal quantum states, in abnormal traffic detection, photons, 395 354–359 nonprintable characters, printing, 133 Denial of Service, 251–258 NOP (no operation) sled, 140, 145, amplification attacks, 257 275, 317, 332, 390 distributed DoS flooding, 258 hiding, 362–363 ping flooding, 257 between loader code and ping of death, 256 shellcode, 373 SYN flooding, 252–256 not equal to operator (!=), 14 teardrop, 256 not operator (!), 14 hacking, 272–280 notesearch.c program, 93–96 analysis with GDB, 273–275 exploitation, 386–387 port-binding shellcode, 278–280 format string vulnerability, network sniffing, 224–251 189–190 active sniffing, 239–251 vulnerability to buffer overflow, decoding layers, 230–239 137–142 libpcap sniffer, 228–230 notetaker.c program, 91–93, 150–155 raw socket sniffer, 226–227 note-taking program, 82 OSI layers for web browser, ntohl() function, 203 217–224 ntohs() function, 203, 206 data-link layer, 218–219 null bytes, 38–39, 290 network layer, 220–221 and exploit buffer, 335 transport layer, 221–224 filling exploit buffer with, 275 OSI model, 196–198 removing, 290–295 port scanning, 264–272 NULL pointer, 77 FIN, X-mas, and null scans, null scans, 264–265 264–265 number field sieve (NFS), 404 idle scanning, 265–266 numbers, pseudo-random, 101–102 proactive defense, 267–272 numerical values, 41–43 spoofing decoys, 265 Nyberg, Claes, 407, 454 stealth SYN scan, 264

INDEX 465 O pads, 395 password file, 153 O_APPEND access mode, 84 password probability matrix, 424–433 objdump program, 21, 184, 185 passwords O_CREAT access mode, 84, 87 cracking, 418–433 off-by-one error, 116–117 dictionary attacks, 419–422 one-time pads, 395 exhaustive brute-force attacks, one-time password, 258 422–423 one-way hashing algorithm, for pass- hash lookup table, 423–424 word encryption, 153 length of, 422 open files, file descriptor to one-time, 258 reference, 82 PATH environment variable, 172 open() function, 87, 336–337 payload smuggling, 359–363 file descriptor for, 82 pcalc (programmer’s calculator), flags used with, 84 42, 454 length of string, 83 pcap libraries, 229 OpenBSD kernel pcap_fatal() function, 228 fragmented IPv6 packets, 256 pcap_lookupdev() function, 228 nonexecutable stack, 376 pcap_loop() function, 235, 236 OpenSSH, 116–117 pcap_next() function, 235 openssh package, 414 pcap_open_live() function, 229, 261 optimization, 6 pcap_sniff.c program, 228 or instruction, 293 percent sign (%), for format OR operator, 14–15 parameter, 48 for file access flags, 84 Perl, 133 O_RDONLY access mode, 84 permissions for files, 87–88 O_RDWR access mode, 84 perror() function, 83 OSI model, 196–198 photons, nonorthogonal quantum layers for web browser, 217–224 states in, 395 data-link layer, 218–219 physical layer (OSI), 196, 197 network layer, 220–221 for web browser, 218 transport layer, 221–224 pigeonhole principle, 425 O_TRUNC access mode, 84 ping flooding, 257 outbound connections, firewalls ping of death, 256 and, 314 ping utility, 221 overflow_example.c program, 119 plaintext, for protocol structure, 208 overflowing function pointers, play_the_game() function, 156–157 156–167 PLT (procedure linkage table), 190 overflows. See buffer overflows pointer, to sockaddr structure, 201 O_WDONLY access mode, 84 pointer arithmetic, 52–53 owner, of file, 87 pointer variables dereferencing, 53 P typecasting, 52 packet injection tool, 242–248 pointer.c program, 44 packet-capturing programs, 224 pointers, 24–25, 43–47 packets, 196, 198 function, 100–101 capturing, 225 to structs, 98 decoding layers, 230–239 pointer_types.c program, 52 inspecting, 359 pointer_types2.c program, 53–54 size limitations, 221 pointer_types3.c program, 55

466 INDEX pointer_types4.c program, 56 product ciphers, 399 pointer_types5.c program, 57 programming polymorphic printable ASCII access to heap, 70 shellcode, 366–376 as artistic expression, 2 pop instruction, 287 basics, 6–7 and printable ASCII, 368 control structures, 8–11 popping, 70 if-then-else, 8–9 port scanning, 264–272 while/until loops, 9–10 FIN, X-mas, and null scans, variables, 11–12 264–265 programs, results from, 116 idle scanning, 265–266 promiscuous mode, 224 proactive defense, 267–272 capturing in, 229 spoofing decoys, 265 pseudo-code, 7, 9 stealth SYN scan, 264 Pseudo-Random Generation Algo- port scanning tool (nmap), 264 rithm (PRGA), 435, 436 port-binding shellcode, 278–280, pseudo-random numbers, 101–102 303–314 public key, 400 ports, root privileges for binding, 216 punch cards, 2 position-independent code, 286 push instruction, 287, 298 PowerPC processor architecture, 20 and printable ASCII, 368 ppm_crack.c program, 428–433 pushing, 70 ppm_gen.c program, 426–428 Pythagoreans, 3 presentation layer (OSI), 196 PRGA (Pseudo-Random Generation Q Algorithm), 435, 436 print command (GDB), 31 quadword, converting print error, 83 doubleword to, 302 printable ASCII shellcode, quantum factoring algorithm, polymorphic, 366–376 404–405 printable characters, program to quantum key distribution, 395–396 calculate, 369 quantum search algorithm, 399–400 printable_helper.c program, 369–370 quotation marks ("), for include printable.s file, 371–372 files, 91 printf() function, 19–20, 35, 37, 47 format strings for, 48–51, 167 R printing nonprintable characters, 133 RainbowCrack, 433 print_ip() function, 254 rand() function, 101 private key, 400 rand_example.c program, 101–102 privileges, 273, 299 random numbers, 101–102 priv_shell.s program, 301 randomization, execl() function and, probability, conditional, 114 390, 391 problem solving randomized stack space, 379–391 with hacking, 1–2 raw socket sniffer, 226–227 hacking as, 5 raw_tcpsniff.c program, 226–227 procedure linkage table (PLT), 190 RC4 (stream cipher), 398, 434, procedure prologue, 71 435–436 process, suspending current, 158 read() function, file descriptor for, 82 process hijacking, 118 read permission, 87 processor, assembly language read-only permission, for text specificity for, 7 segment, 69

INDEX 467 Recording Industry Association of spawning, 192 America (RIAA), 3 spawning with child process, 346 recv() function, 199, 206 user, 88 recv_line() function, 209, 273, RSA Data Security, 394, 400, 404 335, 342 RST hijacking, 259–263 redirection attack, 240–241 rst_hijack.c program, 260–263 registers, 23, 285, 292 modification, 268 displaying, 24 run time of simple algorithm, 397 for x86 processor, 23 zeroing, with polymorphic S shellcode, 366 relatively prime numbers, 400 %s format parameter, 48, 172 remainder, after division, 12 Sadmind worm, 117 remote access, to root shell, 317 salt value, 153–154 remote targets, 321 for password encryption, 419 Request for Comments (RFC) Sasser worm, 319 768, on UDP header, 224 saved frame pointer (SFP), 70, 791, on IP headers, 220, 232 72–73, 130 793, on TCP header, 222–223, S-box array, 435 233–234 scanf() function, 50 ret instruction, 132, 287 scope of variables, 62–69 ret2libc, 376–377 scope.c program, 62 return address, 70 scope2.c program, 63–64 finding exact location, 139 scope3.c program, 64–65 overwriting, 135 script kiddies, 3 in stack frame, 131 Secure Digital Music Initiative return command, 267 (SDMI), 3 Return Material Authorization Secure Shell (SSH) (RMA), 221 differing host fingerprints, return value of function, declaring 410–413 function with data type of, protections against identity 16–17 spoofing, 409–410 RFC. See Request for Comments Secure Sockets Layer (SSL), 393 (RFC) protections against identity RIAA (Recording Industry Associa- spoofing, 409–410 tion of America), 3 security Rieck, Konrad, 413, 454 changing vulnerabilities, 388 RMA (Return Material computational, 396 Authorization), 221 impact of mistakes, 118 Ronnick, Jose, 454 unconditional, 394 root seed number, for random sequence privileges, 153, 273 of numbers, 101 to bind port, 216 segmentation fault, 60, 61 shell to restore, 301 semicolon (;), for instruction end, 8 shell send() function, 199, 206 obtaining, 188 send_string() function, 209 overflow to open, 122 seq command, 141 remote access, 317 sequence numbers, for TCP, 222, 224 socket reuse, 355–359 server example, displaying packet data, 204

468 INDEX session layer (OSI), 196 Simple Mail Transfer Protocol for web browser, 217 (SMTP), 222 set disassembly intel command, 25 simplenote.c program, 82–84 set user ID (setuid) permission, 89 simple_server.c file, 204–207 seteuid() function, 299 sizeof() function, 58 setresuid() system call, 300–301 sizeof() macro (C), 42 setsockopt() function, 205 Sklyarov, Dmitry, 3–4 SFP (saved frame pointer), 70 SMTP (Simple Mail Transfer Shannon, Claude, 394 Protocol), 222 shell command, executing like smurf attacks, 257 function, 134 sniffing packets shellcode, 137, 281 active, 239–251 argument as placement option, 365 in promiscuous mode, 225 assembly language for, 282–286 sockaddr structure, 200–202, 305, 306 connect-back, 314–318 pointer to, 201 creating, 286–295 sockaddr_in structure, 348 jump to, 386 socket() function, 199, 200, 205, 314 memcpy() function to copy, 139 socketcall() system call (Linux), 304 memory location for, 142 socket_reuse_restore.s file, 357 overwriting .dtors section with sockets, 198–217, 307 address of injected, 190 address conversion, 203 placing in environment addresses, 200–202 variable, 188 file descriptor for accepted polymorphic printable ASCII, connection, 206 366–376 functions, 199–200 port-binding, 278–280, 303–314 reuse, 355–359 proof of functioning, 336 server example, 203–207 reducing size, 298 tinyweb server, 213–217 restoring tinyweb daemon web client, 207–213 execution, 345 software piracy, 118 shell-spawning, 295–303 Solar Designer, 422, 454 and webserver, 332 Song, Dug, 226, 249, 454 zeroing registers, 294 source address, manipulating, 239 shellcode.s program, 302–303 Source Index (ESI) register, 24 Shor, Peter, 404–405 Sparc processor, 20 short keyword, 42 spoofing, 239–240 short writes, for format string logged IP address, 348–352 exploits, 182–183 packet contents, 263 shorthand expressions, for arith- sprintf() function, 262 metic operators, 13–14 srand() function, 101 shroud.c program, 268–272 SSH. See Secure Shell (SSH) sigint_handler() function, 323 SSL (Secure Sockets Layer), 393 SIGKILL signal, 324 protections against identity signal() function, 322 spoofing, 409–410 signal_example.c program, 322–323 stack, 40, 70, 128 signal_handler() function, 323 arguments to function call in, 339 signals, for interprocess communica- assembly instructions using, tion in Unix, 322–324 287–289 signed numerical values, 41

INDEX 469 stack, continued strncasecmp() function, 213 frame, 70, 74, 128 strstr() function, 216 displaying local variables in, 66 structs, 96–100 instructions to set up and access to elements, 98 remove structures, 341 su command, 88 growth of, 75 sub instruction, 293, 294 memory in, 77 sub operation, 25 nonexecutable, 376–379 sudo command, 88, 90 randomized space, 379–391 superposition, 399–400 role with format strings, 169 suspended process, returning to, 158 segment, 70 switched network environment, variables packets in, 239 declaring, 76 symmetric encryption, 398–400 and shellcode reliability, 356 SYN flags, 223 Stack Pointer (ESP) register, 24, 33, SYN flooding, 252–256 70, 73 preventing, 255 shellcode and, 367 SYN scan stack_example.c program, 71–75 preventing information leakage Stallman, Richard, 3 with, 268 standard error, 307 stealth, 264 standard input, 307, 358 syncookies, 255 standard input/output (I/O) synflood.c file, 252–254 library, 19 sys/stat.h file, 84 standard output, 307 bit flags defined in, 87 static function memory, string pointer system calls, manual pages for, 283 referencing, 228 system daemons, 321–328 static keyword, 75 system() function, 148–149 static variables, 66–69 returning into, 377–379 memory addresses, 69 memory segment for, 69 T static.c program, 67 static2.c program, 68 TCP. See Transmission Control status flags, cmp operation to set, 311 Protocol (TCP) stderr argument, 79 tcpdump, 224, 226 stdio header file, 19 BPFs for, 259 stealth, by hackers, 320 source code for, 230 stealth SYN scan, 264 tcphdr structure (Linux), 234 stepi command (GDB), 384 TCP/IP, 197 storage space, vs. computational connection, telnet to power, 424 webserver, 208 strace program, 336–338, 352–353 hijacking, 258–263 strcat() function, 121 stack, SYN flood attempt to exhaust strcpy() function, 39–41, 365 states, 252 stream ciphers, 398 tcp_v4_send_reset() function, 267 stream sockets, 198, 222 teardrop, 256 string.h, 39 telnet, 207, 222 strings, 38–41 to open TCP/IP connection to concatenation in Perl, 134 webserver, 208 encoding, 359–362 temporary variable, from print strlen() function, 83, 121, 209 command, 31

470 INDEX text segment, of memory, 69 uid_demo.c program, 90 then keyword, 8–9 ulimit command, 289 th_flags field, of tcphdr structure, 234 uname command, 134 time() function, 97 unary operator time_example.c program, 97 address-of operator, 45 time_example2.c program, 98–99 dereference operator, 47, 50 time_ptr variable, 97 unconditional jumps, in assembly time/space trade-off attack, 424 language, 36 timestamp() function, 352 unconditional security, 394 tiny_shell.s program, 298–299 unencrypted data transmission, 226 tinyweb.c program Unicode character set, 117 converting to system daemon, 321 Unix systems as daemon, 324–328 manual pages, 283 exploit for, 275 signals for interprocess vulnerability in, 273 communication, 322–324 tinywebd.c program, 325–328, 355 time on, 97 exploit tool, 329–333 unsigned keyword, 42 log file, 334 unsigned numerical values, 41 tinyweb_exploit.c program, 275 integer for pointer address, 57 tinyweb_exploit2.c program, 278 unswitched network, 224 tm time struct, 97 until loop, 10 translator, for machine language, 7 update_info.c file, 363–364 Transmission Control Protocol usage() function, 82 (TCP), 198, 222 User Datagram Protocol (UDP), connection for remote shell access, 198–199, 222, 224 308–309 echo packets, amplification attacks flags, 222 with, 257 opening connection, 314 user IDs, 88–96 packet header, 233–234 displaying notes written by, 93 sniffing, with raw sockets, 226 setting effective, 299 structure, 231 users, file permissions for, 87 transport layer (OSI), 196, 197 user-supplied input, length check or for web browser, 217, 221–224 restriction on, 120 Triple-DES, 399 /usr/include/asm-i386/unistd.h file, two’s complement, 42, 49 284–285 to remove null bytes, 291 /usr/include/asm/socket.h file, 205 typecasting, 51–58 /usr/include/bits/socket.h file, from tm struct pointer to integer 200, 201 pointer, 98 /usr/include/if_ether.h file, 230 typecasting.c program, 51 /usr/include/linux/if_ethernet.h typedef, 245 file, 230 typeless pointers, 56 /usr/include/netinet/ip.h file, types. See data types 230, 231–232 /usr/include/netinet/tcp.h file, 230, U 233–234 /usr/include/stdio.h file, 19 UDP (User Datagram Protocol), /usr/include/sys/sockets.h file, 199 198–199, 222, 224 /usr/include/time.h file, 97 echo packets, amplification attacks /usr/include/unistd.h file, 284 with, 257 /usr/src/mitm-ssh, 407

INDEX 471 V where command, 61 while/until loops, 9–10 values Wired Equivalent Privacy (WEP), 433, assigning to variable, 12 434–435 returned by function, 16 attacks, 436–449 variables, 11–12 wireless 802.11b encryption, 433–436 arithmetic operators for, 12–14 word, 28–29 C compiler and data type, 58 worms, 119 comparison operators for, 14–15 Wozniak, Steve, 3 scope, 62–69 WPA wireless protocol, 448 structs, 96–100 write() function, 83 temporary, from print file descriptor for, 82 command, 31 manual page for, 283 typecasting, 51–58 pointer for, 92 void keyword, 56 write permission, 87 for declaring function, 17 for text segment, 69 void pointer (C), 56, 57 vuln.c program, 377 vulnerabilities X format strings, 170–171 %x format parameter, 171, 173 in software, 451–452 field-width option, 179 stack-based, 122–133 x/3xw command, 61 in tinyweb.c program, 273 x86 processor, 20, 23–25 zero-day VML, 119 assembly instructions for, 285 xchg (exchange) instruction, 312 W X-mas scans, 264–265 xor instruction, 293, 294 warnings, about pointer data type, 54 xtool_tinywebd_reuse.sh script, 358 web browser, OSI layers for, 217–224 xtool_tinywebd.sh script, 333 web client, 207–213 xtool_tinywebd_silent.sh script, web requests, processing after 353–354 intrusion, 336 xtool_tinywebd_spoof.sh script, webserver 349–350 telnet for TCP/IP xtool_tinywebd_stealth.sh script, 335 connection to, 208 tinyweb server, 213–217 webserver_id.c file, 212–213 Z WEP (Wired Equivalent Privacy), 433, zeroing registers, 294 434–435 EAX (Accumulator) register, 368 attacks, 436–449 with polymorphic shellcode, 366

472 INDEX