Carnegie Mellon Carnegie Mellon

Today

¢ Linking Linking ¢ Case study: interposioning

15-213 / 18-213: Introducon to Computer Systems 12th Lecture, Feb. 23, 2012

Instructors: Todd C. Mowry & Anthony Rowe

1 2

Carnegie Mellon Carnegie Mellon

Example C Program Stac Linking

¢ Programs are translated and linked using a driver: main.c swap.c § > gcc -O2 -g -o p main.c swap.c int buf[2] = {1, 2}; extern int buf[]; § unix> ./p

int main() int *bufp0 = &buf[0]; { static int *bufp1; main.c swap.c Source files swap(); return 0; void swap() Translators Translators } { (cpp, cc1, as) (cpp, cc1, as) int temp;

Separately compiled bufp1 = &buf[1]; main.o swap.o relocatable object files temp = *bufp0; *bufp0 = *bufp1; (ld) *bufp1 = temp; } Fully linked object file p (contains code and for all funcons defined in main.c and swap.c)

3 4

1 Carnegie Mellon Carnegie Mellon

Why Linkers? Why Linkers? (cont)

¢ Reason 1: Modularity ¢ Reason 2: Efficiency

§ Program can be wrien as a collecon of smaller source files, § Time: Separate compilaon rather than one monolithic mass. § Change one source file, compile, and then relink. § No need to recompile other source files. § Can build libraries of common funcons (more on this later) § e.g., Math library, standard C library § Space: Libraries § Common funcons can be aggregated into a single file... § Yet executable files and running memory images contain only code for the funcons they actually use.

5 6

Carnegie Mellon Carnegie Mellon

What Do Linkers Do? What Do Linkers Do? (cont)

¢ Step 1. Symbol resoluon ¢ Step 2. Relocaon

§ Programs define and reference symbols (variables and funcons): § Merges separate code and data secons into single secons § void swap() {…} /* define symbol swap */ § swap(); /* reference symbol a */ § Relocates symbols from their relave locaons in the .o files to § int *xp = &x; /* define symbol xp, reference x */ their final absolute memory locaons in the executable.

§ Symbol definions are stored (by compiler) in . § Updates all references to these symbols to reflect their new § Symbol table is an array of structs posions. § Each entry includes name, size, and locaon of symbol.

§ Linker associates each symbol reference with exactly one symbol definion.

7 8

2 Carnegie Mellon Carnegie Mellon

Three Kinds of Object Files (Modules) Executable and Linkable Format (ELF)

¢ Relocatable object file (.o file) ¢ Standard binary format for object files § Contains code and data in a form that can be combined with other ¢ Originally proposed by AT&T System V Unix relocatable object files to form executable object file. § Later adopted by BSD Unix variants and § Each .o file is produced from exactly one source (.c) file ¢ One unified format for § Relocatable object files (.o), ¢ Executable object file (a.out file) § Executable object files (a.out) § Contains code and data in a form that can be copied directly into § Shared object files (.so) memory and then executed.

¢ Generic name: ELF binaries ¢ Shared object file (.so file) § Special type of relocatable object file that can be loaded into memory and linked dynamically, at either load me or run-me. § Called Dynamic Link Libraries (DLLs) by Windows

9 10

Carnegie Mellon Carnegie Mellon ELF Object ELF Format (cont.) ¢ Elf header § Word size, byte ordering, file type (.o, ¢ .symtab secon 0 0 , .so), machine type, etc. ELF header § Symbol table ELF header § Procedure and stac variable names ¢ Segment header table Segment header table Segment header table § Secon names and locaons § Page size, virtual addresses memory segments (required for ) (required for executables) (secons), segment sizes. .text secon ¢ .rel.text secon .text secon § Relocaon info for .text secon ¢ .text secon .rodata secon § Addresses of instrucons that will need to be .rodata secon § Code .data secon modified in the executable .data secon § Instrucons for modifying. ¢ .rodata secon .bss secon .bss secon § Read only data: jump tables, ... ¢ .rel.data secon .symtab secon § Relocaon info for .data secon .symtab secon ¢ .data secon .rel.txt secon § Addresses of pointer data that will need to be .rel.txt secon § Inialized global variables modified in the merged executable .rel.data secon .rel.data secon ¢ .bss secon ¢ .debug secon .debug secon .debug secon § Uninialized global variables § Info for symbolic (gcc -g) § “Block Started by Symbol” Secon header table ¢ Secon header table Secon header table § “Beer Save Space” § Offsets and sizes of each secon § Has secon header but occupies no space 11 12

3 Carnegie Mellon Carnegie Mellon

Linker Symbols Resolving Symbols

Global External Local ¢ Global symbols Global § Symbols defined by module m that can be referenced by other modules. int buf[2] = {1, 2}; extern int buf[]; § E.g.: non-static C funcons and non-static global variables. int main() int *bufp0 = &buf[0]; { static int *bufp1; ¢ External symbols swap(); § Global symbols that are referenced by module m but defined by some return 0; void swap() Global other module. } main.c { int temp;

¢ Local symbols External Linker knows bufp1 = &buf[1]; § Symbols that are defined and referenced exclusively by module m. nothing of temp temp = *bufp0; § E.g.: C funcons and variables defined with the static aribute. *bufp0 = *bufp1; *bufp1 = temp; § Local linker symbols are not local program variables } swap.c

13 14

Carnegie Mellon Carnegie Mellon Relocang Code and Data Relocaon Info (main)

Relocatable Object Files Executable Object File main.c main.o int buf[2] = 0000000

: {1,2}; 0: 8d 4c 24 04 lea 0x4(%esp),%ecx 4: 83 e4 f0 and $0xfffffff0,%esp System code .text 0 Headers int main() 7: ff 71 fc pushl 0xfffffffc(%ecx) a: 55 push %ebp System data .data System code { b: 89 e5 mov %esp,%ebp swap(); d: 51 push %ecx main() return 0; e: 83 ec 04 sub $0x4,%esp .text main.o } 11: e8 fc ff ff ff call 12 swap() 12: R_386_PC32 swap main() .text 16: 83 c4 04 add $0x4,%esp More system code 19: 31 c0 xor %eax,%eax int buf[2]={1,2} .data 1b: 59 pop %ecx System data 1c: 5d pop %ebp swap.o int buf[2]={1,2} .data 1d: 8d 61 fc lea 0xfffffffc(%ecx),%esp int *bufp0=&buf[0] 20: c3 ret swap() .text int *bufp1 .bss int *bufp0=&buf[0] .data .symtab .debug Disassembly of section .data: static int *bufp1 .bss Source: –r -d 00000000 : Even though private to swap, requires allocaon in .bss 0: 01 00 00 00 02 00 00 00 15 16

4 Carnegie Mellon Carnegie Mellon Relocaon Info (swap, .text) Relocaon Info (swap, .data) swap.c swap.o swap.c extern int buf[]; Disassembly of section .text: extern int buf[]; Disassembly of section .data: 00000000 : int 00000000 : *bufp0 = &buf[0]; 0: 8b 15 00 00 00 00 mov 0x0,%edx int *bufp0 = 2: R_386_32 buf &buf[0]; 0: 00 00 00 00 6: a1 04 00 00 00 mov 0x4,%eax static int *bufp1; static int *bufp1; 0: R_386_32 buf 7: R_386_32 buf b: 55 push %ebp void swap() c: 89 e5 mov %esp,%ebp void swap() { e: c7 05 00 00 00 00 04 movl $0x4,0x0 { int temp; 15: 00 00 00 int temp; 10: R_386_32 .bss bufp1 = &buf[1]; 14: R_386_32 buf bufp1 = &buf[1]; 18: 8b 08 mov (%eax),%ecx temp = *bufp0; temp = *bufp0; 1a: 89 10 mov %edx,(%eax) *bufp0 = *bufp1; *bufp0 = *bufp1; 1c: 5d pop %ebp *bufp1 = temp; *bufp1 = temp; 1d: 89 0d 04 00 00 00 mov %ecx,0x4 } 1f: R_386_32 buf } 23: c3 ret

17 18

Carnegie Mellon Carnegie Mellon

0: 8b 15 00 00 00 00 mov 0x0,%edx Executable Before/Aer Relocaon (.text) 2: R_386_32 buf 0000000

: 6: a1 04 00 00 00 mov 0x4,%eax . . . 0x8048396 + 0x1a 7: R_386_32 buf e: 83 ec 04 sub $0x4,%esp ... 11: e8 fc ff ff ff call 12 = 0x80483b0 e: c7 05 00 00 00 00 04 movl $0x4,0x0 12: R_386_PC32 swap 15: 00 00 00 16: 83 c4 04 add $0x4,%esp 10: R_386_32 .bss . . . 14: R_386_32 buf . . . 08048380
: 1d: 89 0d 04 00 00 00 mov %ecx,0x4 8048380: 8d 4c 24 04 lea 0x4(%esp),%ecx 1f: R_386_32 buf 8048384: 83 e4 f0 and $0xfffffff0,%esp 23: c3 ret 8048387: ff 71 fc pushl 0xfffffffc(%ecx) 804838a: 55 push %ebp 080483b0 : 804838b: 89 e5 mov %esp,%ebp 80483b0: 8b 15 20 96 04 08 mov 0x8049620,%edx 804838d: 51 push %ecx 80483b6: a1 24 96 04 08 mov 0x8049624,%eax 804838e: 83 ec 04 sub $0x4,%esp 80483bb: 55 push %ebp 8048391: e8 1a 00 00 00 call 80483b0 80483bc: 89 e5 mov %esp,%ebp 8048396: 83 c4 04 add $0x4,%esp 80483be: c7 05 30 96 04 08 24 movl $0x8049624,0x8049630 8048399: 31 c0 xor %eax,%eax 80483c5: 96 04 08 804839b: 59 pop %ecx 80483c8: 8b 08 mov (%eax),%ecx 804839c: 5d pop %ebp 80483ca: 89 10 mov %edx,(%eax) 804839d: 8d 61 fc lea 0xfffffffc(%ecx),%esp 80483cc: 5d pop %ebp 80483a0: c3 ret 80483cd: 89 0d 24 96 04 08 mov %ecx,0x8049624 80483d3: c3 ret 19 20

5 Carnegie Mellon Carnegie Mellon

Executable Aer Relocaon (.data) Strong and Weak Symbols

¢ Program symbols are either strong or weak Disassembly of section .data: § Strong: procedures and inialized globals 08049620 : § Weak: uninialized globals 8049620: 01 00 00 00 02 00 00 00

08049628 : 8049628: 20 96 04 08 p1.c p2.c strong int foo=5; int foo; weak

strong p1() { p2() { strong } }

21 22

Carnegie Mellon Carnegie Mellon

Linker’s Symbol Rules Linker Puzzles int x; Link me error: two strong symbols (p1) ¢ Rule 1: Mulple strong symbols are not allowed p1() {} p1() {} § Each item can be defined only once § Otherwise: Linker error int x; int x; References to x will refer to the same p1() {} p2() {} uninialized int. Is this what you really want?

¢ Rule 2: Given a strong symbol and mulple , int x; double x; int y; p2() {} Writes to x in p2 might overwrite y! choose the strong symbol p1() {} Evil! § References to the weak symbol resolve to the strong symbol int x=7; double x; Writes to x in p2 will overwrite y! int y=5; p2() {} Nasty! p1() {} ¢ Rule 3: If there are mulple weak symbols, pick an arbitrary one int x=7; int x; References to x will refer to the same inialized § Can override this with gcc –fno-common p1() {} p2() {} variable.

Nightmare scenario: two idencal weak structs, compiled by different with different alignment rules. 23 24

6 Carnegie Mellon Carnegie Mellon

Role of .h Files Running Preprocessor global.h global.h c1.c c1.c #ifdef INITIALIZE #include "global.h" #ifdef INITIALIZE #include "global.h" int g = 23; int g = 23;

static int init = 1; int f() { static int init = 1; int f() { #else return g+1; #else return g+1; int g; } int g; } static int init = 0; static int init = 0; #endif #endif c2.c -DINITIALIZE #include #include "global.h" no inializaon

int main() { int g = 23; int g; if (!init) static int init = 1; static int init = 0; g = 37; int f() { int f() { int t = f(); return g+1; return g+1; printf("Calling f yields %d\n", t); } } return 0; } #include causes C preprocessor to insert file verbam 25 26

Carnegie Mellon Carnegie Mellon

Role of .h Files (cont.) Global Variables global.h c1.c #ifdef INITIALIZE ¢ Avoid if you can #include "global.h" int g = 23; static int init = 1; int f() { #else ¢ Otherwise return g+1; int g; } static int init = 0; § Use static if you can #endif § Inialize if you define a global variable c2.c § Use extern if you use external global variable #include What happens: #include "global.h" gcc -o p c1.c c2.c

int main() { ?? if (!init) gcc -o p c1.c c2.c \ g = 37; -DINITIALIZE int t = f(); ?? printf("Calling f yields %d\n", t); return 0; } 27 28

7 Carnegie Mellon Carnegie Mellon

Packaging Commonly Used Funcons Soluon: Stac Libraries

¢ How to package funcons commonly used by ? ¢ Stac libraries (.a archive files) § Math, I/O, , string manipulaon, etc. § Concatenate related relocatable object files into a single file with an index (called an archive).

¢ Awkward, given the linker framework so far: § Enhance linker so that it tries to resolve unresolved external references § Opon 1: Put all funcons into a single source file by looking for the symbols in one or more archives. § Programmers link big object file into their programs

§ Space and me inefficient § If an archive member file resolves reference, link it into the executable. § Opon 2: Put each funcon in a separate source file

§ Programmers explicitly link appropriate binaries into their programs § More efficient, but burdensome on the

29 30

Carnegie Mellon Carnegie Mellon

Creang Stac Libraries Commonly Used Libraries libc.a (the ) § 8 MB archive of 1392 object files. atoi.c printf.c random.c § I/O, memory allocaon, signal handling, string handling, data and me, random numbers, integer math Translator Translator ... Translator libm.a (the C math library) § 1 MB archive of 401 object files. atoi.o printf.o random.o § floang point math (sin, cos, tan, log, exp, sqrt, …)

% -t /usr/lib/libc.a | sort % ar -t /usr/lib/libm.a | sort Archiver (ar) unix> ar rs libc.a \ atoi.o printf.o … random.o … … .o e_acos.o … e_acosf.o libc.a C standard library fprintf.o e_acosh.o fpu_control.o e_acoshf.o fputc.o e_acoshl.o freopen.o e_acosl.o fscanf.o e_asin.o ¢ Archiver allows incremental updates fseek.o e_asinf.o ¢ Recompile funcon that changes and replace .o file in archive. fstab.o e_asinl.o … …

31 32

8 Carnegie Mellon Carnegie Mellon Linking with Stac Libraries Using Stac Libraries

addvec.o multvec.o ¢ Linker’s algorithm for resolving external references: § Scan .o files and .a files in the command line order. § During the scan, keep a list of the current unresolved references. main2.c vector.h Archiver § (ar) As each new .o or .a file, obj, is encountered, try to resolve each unresolved reference in the list against the symbols defined in obj. Translators § If any entries in the unresolved list at end of scan, then error. (cpp, cc1, as) libvector.a libc.a Stac libraries

and any other ¢ Problem: Relocatable main2.o addvec.o printf.o object files modules called by printf.o § Command line order maers! § Moral: put libraries at the end of the command line. Linker (ld) unix> gcc -L. libtest.o -lmine unix> gcc -L. -lmine libtest.o Fully linked p2 libtest.o: In function `main': executable object file libtest.o(.text+0x4): undefined reference to `libfun'

33 34

Carnegie Mellon Carnegie Mellon Loading Executable Object Files Shared Libraries Memory outside 32-bit Executable Object File Kernel virtual memory 0 address space ¢ Stac libraries have the following disadvantages: ELF header 0x100000000 User stack § Duplicaon in the stored executables (every funcon need std libc) Program header table (created at runme) (required for executables) %esp § Duplicaon in the running executables (stack .init secon pointer) § Minor bug fixes of system libraries require each applicaon to explicitly relink .text secon Memory-mapped region for shared libraries .rodata secon 0xf7e9ddc0 .data secon ¢ Modern soluon: Shared Libraries

.bss secon brk § Object files that contain code and data that are loaded and linked into .symtab Run-me heap an applicaon dynamically, at either load-me or run-me (created by malloc) .debug § Also called: dynamic link libraries, DLLs, .so files Read/write segment Loaded .line (.data, .bss) from the .strtab Read-only segment executable Secon header table (.init, .text, .rodata) file (required for relocatables) 0x08048000 Unused 0 35 36

9 Carnegie Mellon Carnegie Mellon

Shared Libraries (cont.) Dynamic Linking at Load-me main2.c vector.h unix> gcc -shared -o libvector.so \ addvec.c multvec.c ¢ Dynamic linking can occur when executable is first loaded and run (load-me linking). Translators (cpp, cc1, as) libc.so § Common case for Linux, handled automacally by the libvector.so (ld-linux.so). Relocatable main2.o Relocaon and symbol § Standard C library (libc.so) usually dynamically linked. object file table info Linker (ld) ¢ Dynamic linking can also occur aer program has begun (run-me linking). Parally linked p2 § In Linux, this is done by calls to the dlopen() interface. executable object file § Distribung soware. § High-performance web servers. libc.so (execve) libvector.so § Runme library interposioning. Code and data ¢ Shared library rounes can be shared by mulple processes. Fully linked § More on this when we learn about virtual memory executable Dynamic linker (ld-linux.so) in memory 37 38

Carnegie Mellon Carnegie Mellon Dynamic Linking at Run-me Dynamic Linking at Run-me

#include ... #include /* get a pointer to the addvec() function we just loaded */ int x[2] = {1, 2}; addvec = dlsym(handle, "addvec"); int y[2] = {3, 4}; if ((error = dlerror()) != NULL) { int z[2]; fprintf(stderr, "%s\n", error); exit(1); int main() } { void *handle; /* Now we can call addvec() just like any other function */ void (*addvec)(int *, int *, int *, int); addvec(x, y, z, 2); char *error; printf("z = [%d %d]\n", z[0], z[1]);

/* dynamically load the shared lib that contains addvec() */ /* unload the shared library */ handle = dlopen("./libvector.so", RTLD_LAZY); if (dlclose(handle) < 0) { if (!handle) { fprintf(stderr, "%s\n", dlerror()); fprintf(stderr, "%s\n", dlerror()); exit(1); exit(1); } } return 0; }

39 40

10 Carnegie Mellon Carnegie Mellon

Today Case Study: Library Interposioning

¢ Linking ¢ Library interposioning : powerful linking technique that ¢ Case study: Library interposioning allows programmers to intercept calls to arbitrary funcons ¢ Interposioning can occur at: § Compile me: When the source code is compiled § Link me: When the relocatable object files are stacally linked to form an executable object file § Load/run me: When an executable object file is loaded into memory, dynamically linked, and then executed.

41 42

Carnegie Mellon Carnegie Mellon

Some Interposioning Applicaons Example program

¢ Security #include ¢ Goal: trace the addresses #include § Confinement (sandboxing) #include and sizes of the allocated § Interpose calls to libc funcons. and freed blocks, without § Behind the scenes encrypon int main() { modifying the source code. § Automacally encrypt otherwise unencrypted network free(malloc(10)); connecons. printf("hello, world\n"); exit(0); ¢ Three soluons: interpose ¢ Monitoring and Profiling } on the lib malloc and § Count number of calls to funcons hello.c free funcons at compile § Characterize call sites and arguments to funcons me, link me, and load/ § Malloc tracing run me. § Detecng memory leaks § Generang address traces

43 44

11 Carnegie Mellon Carnegie Mellon

Compile-me Interposioning Compile-me Interposioning

#ifdef COMPILETIME #define malloc(size) mymalloc(size, __FILE__, __LINE__ ) /* Compile-time interposition of malloc and free using C #define free(ptr) myfree(ptr, __FILE__, __LINE__ ) * preprocessor. A local malloc.h file defines malloc (free)

* as wrappers mymalloc (myfree) respectively. void *mymalloc(size_t size, char *file, int line); */ void myfree(void *ptr, char *file, int line);

malloc.h #include #include linux> helloc /* gcc -O2 -Wall -DCOMPILETIME -c mymalloc.c * mymalloc - malloc wrapper function gcc -O2 -Wall -I. -o helloc hello.c mymalloc.o */ linux> make runc void *mymalloc(size_t size, char *file, int line) ./helloc { hello.c:7: malloc(10)=0x501010 void *ptr = malloc(size); hello.c:7: free(0x501010) printf("%s:%d: malloc(%d)=%p\n", file, line, (int)size, hello, world ptr); return ptr; } mymalloc.c 45 46

Carnegie Mellon Carnegie Mellon

Link-me Interposioning Link-me Interposioning

linux> make hellol #ifdef LINKTIME gcc -O2 -Wall -DLINKTIME -c mymalloc.c /* Link-time interposition of malloc and free using the gcc -O2 -Wall -Wl,--wrap,malloc -Wl,--wrap,free \ static linker's (ld) "--wrap symbol" flag. */ -o hellol hello.c mymalloc.o linux> make runl #include ./hellol malloc(10) = 0x501010 void *__real_malloc(size_t size); free(0x501010) void __real_free(void *ptr); hello, world

/* * __wrap_malloc - malloc wrapper function ¢ The “-Wl” flag passes argument to linker */ ¢ Telling linker “--wrap,malloc ” tells it to resolve void *__wrap_malloc(size_t size) { references in a special way: void *ptr = __real_malloc(size); § Refs to malloc should be resolved as __wrap_malloc printf("malloc(%d) = %p\n", (int)size, ptr); § Refs to __real_malloc should be resolved as malloc return ptr; } mymalloc.c 47 48

12 Carnegie Mellon Carnegie Mellon #ifdef RUNTIME /* Run-time interposition of malloc and free based on * dynamic linker's (ld-linux.so) LD_PRELOAD mechanism */ Load/Run-me Interposioning #define _GNU_SOURCE #include linux> make hellor #include gcc -O2 -Wall -DRUNTIME -shared -fPIC -o mymalloc.so mymalloc.c #include Load/Run-me gcc -O2 -Wall -o hellor hello.c void *malloc(size_t size) Interposioning linux> make runr { (LD_PRELOAD="/usr/lib64/libdl.so ./mymalloc.so" ./hellor) static void *(*mallocp)(size_t size); malloc(10) = 0x501010 char *error; free(0x501010) void *ptr; hello, world

/* get address of libc malloc */ if (!mallocp) { mallocp = dlsym(RTLD_NEXT, "malloc"); ¢ The LD_PRELOAD tells the dynamic if ((error = dlerror()) != NULL) { fputs(error, stderr); linker to resolve unresolved refs (e.g., to malloc)by looking exit(1); in libdl.so and mymalloc.so first. } § } libdl.so necessary to resolve references to the dlopen ptr = mallocp(size); funcons. printf("malloc(%d) = %p\n", (int)size, ptr); return ptr; } mymalloc.c 49 50

Carnegie Mellon

Interposioning Recap

¢ § Apparent calls to malloc/free get macro-expanded into calls to mymalloc/myfree ¢ Link Time § Use linker trick to have special name resoluons § malloc à __wrap_malloc § __real_malloc à malloc ¢ Compile Time § Implement custom version of malloc/free that use dynamic linking to load library malloc/free under different names

51

13