Dynamic Tracing with DTrace SystemTap Sergey Klyaus Copyright © 2011-2016 Sergey Klyaus This work is licensed under the Creative Commons Attribution-Noncommercial-ShareAlike 3.0 License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/3.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. Table of contents Introduction 7 Foreword . 7 Typographic conventions . 9 TSLoad workload generator . 12 Operating system Kernel . 14 Module 1: Dynamic tracing tools. dtrace and stap tools 15 Tracing . 15 Dynamic tracing . 16 DTrace . 17 SystemTap . 19 Safety and errors . 22 Stability . 23 Module 2: Dynamic tracing languages 25 Introduction . 25 Probes . 27 Arguments . 33 Context . 34 Predicates . 35 Types and Variables . 37 Pointers . 40 Strings and Structures . 43 Exercise 1 . 44 Associative arrays and aggregations . 44 Time . 48 Printing and speculations . 48 Tapsets translators . 50 Exercise 2 . 52 Module 3: Principles of dynamic tracing 54 Applying tracing . 54 Dynamic code analysis . 55 Profiling . 61 Performance analysis . 65 Pre- and post-processing . 66 Vizualization . 70 Module 4: Operating system kernel tracing 74 Process management . 74 3 Exercise 3 . 86 Process scheduler . 87 Virtual Memory . 105 Exercise 4 . 116 Virtual File System . 116 Block Input-Output . 122 Asynchronicity in kernel . 131 Exercise 5 . 132 Network Stack . 134 Synchronization primitives . 138 Interrupt handling and deferred execution . 143 Module 5: Application tracing 146 Userspace process tracing . 146 Unix C library . 149 Exercise 6 . 152 Java Virtual Machine . 153 Non-native languages . 160 Web applications . 165 Exercise 7 . 172 Appendix A. Exercise hints and solutions 173 Exercise 1 . 173 Exercise 2 . 175 Exercise 3 . 176 Exercise 4 . 184 Exercise 5 . 187 Exercise 6 . 190 Exercise 7 . 191 Appendix B. Lab setup 193 Setting up Operating Systems . 193 iSCSI . 194 Web application stack . 195 Appendix C. Cheatsheet 200 Cheatsheet . 200 4 SystemTap example scripts scripts/stap/wstat.stp . 47 scripts/stap/tapset/lstat.stp . 51 scripts/stap/lstat.stp . 52 scripts/stap/callgraph.stp . 58 scripts/stap/dumptask.stp . 75 scripts/stap/proc.stp . 84 scripts/stap/cfstrace.stp . 100 scripts/stap/pagefault.stp . 113 scripts/stap/scsitrace.stp . 128 scripts/stap/wqtrace.stp . 141 scripts/stap/pthread.stp . 150 scripts/stap/hotspot.stp . 155 scripts/stap/pymalloc.stp . 162 scripts/stap/pycode.stp . 165 scripts/stap/web.stp . 169 scripts/stap/opentrace.stp . 174 scripts/stap/openaggr.stp . 176 scripts/stap/dumptask-lab3.stp . 176 scripts/stap/forktime.stp . 182 scripts/stap/pfstat.stp . 184 scripts/stap/kmemstat.stp . 187 scripts/stap/deblock.stp . 187 scripts/stap/readahead.stp . 188 scripts/stap/mtxtime.stp . 191 scripts/stap/topphp.stp . 191 DTrace example scripts scripts/dtrace/wstat.d . 47 scripts/dtrace/stat.d . 50 scripts/dtrace/callgraph.d . 60 scripts/dtrace/dumptask.d . 80 scripts/dtrace/proc.d . 85 scripts/dtrace/tstrace.d . ..