Introduction to low-level profiling and tracing EuroPython 2019 / Basel 2019-07-11 Christian Heimes Principal Software Engineer [email protected] / [email protected] @ChristianHeimes Our systems block every 5 minutes. We are loosing money. Fix it! Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Is it me, or the GIL? Christoph Heer EuroPython 2019 Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 requests.get('https://www.python.org') Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 2 ½ use case for tracing tools Who am I? ● from Hamburg/Germany ● Python and C developer ● Python core contributor since 2008 ● maintainer of ssl and hashlib module ● Python security team Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Professional life ● Principal Software Engineer at Red Hat ● Security Engineering ● FreeIPA Identity Management ● Dogtag PKI Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Agenda & Goals Agenda ● introduction ● ptrace (strace, ltrace) ● kernel & hardware tracing tools ● Summary ● ~ 5 minutes Q&A Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Special thanks ● Brendan Gregg (Netflix) ● Victor Stinner (CPython, Red Hat) https://vstinner.readthedocs.io/benchmark.html ● Dmitry Levin (strace) ”Modern Strace”, DevConf.CZ 2019 ● Michal Sekletár (Red Hat) ”Tracing Tools for Systems Engineers”, DevConf.CZ 2019 Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Introduction Terminology Debugging – The process of identifying and removing bugs. ● active, expensive, intrusive, slow-down ● deploy new version ● attach debugger Tracing – observing and monitoring behavior ● passive, non-intrusive, and fast* Profiling – gathering and analyzing metrics ● byproduct of tracing with counting and timing Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Methodology Application level tracing ● debug builds ● performance logs (MySQL Slow Query Log, PHP xdebug) ● trace hooks (Python: sys.settrace()) User space tracing ● LD_PRELOAD, ptrace Kernel space tracing ● ftrace, eBPF, systemtap, LTTng, ... Hardware tracing ● hardware performance counter, PMU, ... Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Prerequisites ● installation of extra packages ● permissions ● root or special group ● CAP_SYS_PTRACE, CAP_SYS_ADMIN, CAP_SYS_MODULE ● recent Kernel (BCC, eBPF) ● debug symbols dnf debuginfo-install package, apt install package-dbg ● compiler/linker flags (-g, -fno-omit-framepointer, BIND_LAZY) ● disable secure boot (stap) Kernel Live Patching or dynamic modules Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 “Lies, damned lies, and statistics” ”Wer misst, misst Mist.” Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 benchmarks / statistics ● average, median, standard deviation, percentile ● observational error ● systematic error ● random error ● systemic bias, cognitive bias (human factor) ● misleading presentation (Vatican City has 2.27 popes / km²) ● sampling profiler: quantization error, Nyquist–Shannon theorem “Producing Wrong Data Without Doing Anything Obviously Wrong!” (Mytkowicz, Diwan, Hauswirth, Sweeney) Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Computers are noisy ● CPU power states (P-states, C-states, TurboBoost, thermal throttling) ● caches (L1 CPU, file system, JIT warm-up, DNS/HTTP) ● hardware IRQs, Address Space Layout Randomization (ASLR) https://vstinner.readthedocs.io/benchmark.html Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 env vars, path length, hostname, username BenchmarkBenchmark withoutwithout PythonPython virtualvirtual environmentenvironment %% timetime secondsseconds usecs/callusecs/call callscalls errorserrors syscallsyscall ------------ ---------------------- ---------------------- ------------------ ------------------ -------------------------------- 28.7828.78 0.0004380.000438 00 14401440 readread 27.3327.33 0.0004160.000416 11 440440 2525 statstat 9.729.72 0.0001480.000148 11 144144 mmapmmap ...... 0.790.79 0.0000120.000012 11 1111 munmapmunmap BenchmarkBenchmark insideinside aa virtualvirtual environmentenvironment %% timetime secondsseconds usecs/callusecs/call callscalls errorserrors syscallsyscall ------------ ---------------------- ---------------------- ------------------ ------------------ -------------------------------- 57.1257.12 0.0990230.099023 22 6147161471 munmapmunmap 41.8741.87 0.0725800.072580 11 6161861618 mmapmmap 0.230.23 0.0003950.000395 11 465465 2727 statstat ● https://mail.python.org/pipermail/python-dev/2019-February/156522.html ● https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Let's profile importimport timetime startstart == time.time()time.time() withwith open('/etc/os-release')open('/etc/os-release') asas f:f: lineslines == f.readlines()f.readlines() print(time.time()print(time.time() -- start)start) Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 Examples ● shell ● cat ● Python ● open + read file ● HTTPS request with requests Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 ptrace ptrace ● process trace syscall ● introduced in Unix Version 6 (~1985) ● user-space tracing ● used by debuggers and code analysis tools ● gdb ● strace ● ltrace ● code coverage ● ... Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 ltrace – A library call tracer $$ ltraceltrace -e-e ''SSL_CTX*@*SSL_CTX*@*'' \\ python3python3 -c-c ''importimport requests;requests; requests.get("https://www.python.org")requests.get("https://www.python.org")'' _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_newSSL_CTX_new(0x7fa4a1a77120,(0x7fa4a1a77120, 0,0, 0,0, 0)0) == 0x55636123a1000x55636123a100 _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_get_verify_callbackSSL_CTX_get_verify_callback(0x55636123a100,(0x55636123a100, -2,-2, 0x7fa4a1acd5e0,0x7fa4a1acd5e0, 0x7fa4a1acd5f8)0x7fa4a1acd5f8) == 00 _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_set_verifySSL_CTX_set_verify(0x55636123a100,(0x55636123a100, 0,0, 0,0, 0x7fa4a1acd5f8)0x7fa4a1acd5f8) == 00 _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_set_optionsSSL_CTX_set_options(0x55636123a100,(0x55636123a100, 0x82420054,0x82420054, 0,0, 0x7fa4a1acd5f8)0x7fa4a1acd5f8) == 0x825200540x82520054 _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_ctrlSSL_CTX_ctrl(0x55636123a100,(0x55636123a100, 33,33, 16,16, 0)0) == 2020 _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_set_session_id_contextSSL_CTX_set_session_id_context(0x55636123a100,(0x55636123a100, 0x7fa4af500494,0x7fa4af500494, 7,7, 0)0) == 11 ...... _ssl.cpython-37m-x86_64-linux-gnu.so->_ssl.cpython-37m-x86_64-linux-gnu.so->SSL_CTX_freeSSL_CTX_free(0x55636123a100,(0x55636123a100, 0x7fa4af4b1e10,0x7fa4af4b1e10, -3,-3, 0x7fa4a12734c8)0x7fa4a12734c8) == 00 Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 ltrace – count memory allocations >>>>>> importimport os,os, time,time, requestsrequests >>>>>> os.getpid()os.getpid() 65046504 ## attachattach ltraceltrace >>>>>> startstart == time.time();time.time(); rr == requests.get("https://www.python.org");requests.get("https://www.python.org"); print(time.time()print(time.time() -- start)start) ## withoutwithout ltrace:ltrace: 0.5660.566 secsec ## withwith ltrace:ltrace: 3.3963.396 secsec $$ ltraceltrace -e-e ''malloc+free+realloc@*malloc+free+realloc@*'' -c-c -p-p 65046504 %% timetime secondsseconds usecs/callusecs/call callscalls functionfunction ------------ ---------------------- ---------------------- ------------------ ---------------------------------------- 53.4953.49 1.1048041.104804 3737 2979629796 freefree 45.7845.78 0.9455650.945565 3737 2552325523 mallocmalloc 0.730.73 0.0150690.015069 4848 309309 reallocrealloc ------------ ---------------------- ---------------------- ------------------ ---------------------------------------- 100.00100.00 2.0654382.065438 5562855628 totaltotal Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 strace – trace system calls and signals ● Paul Kranenburg in 1991 ● Dmitry Levin (current maintainer) Ring 3 Ring 2 Least privileged Ring 1 Ring 0 syscall Kernel Most privileged Device drivers Device drivers Applications Hertzsprung at English Wikipedia [CC BY-SA 3.0] Profiling and Tracing, EuroPython 2019, @ChristianHeimes, CC BY-SA 4.0 strace “open” syscall $$ stracestrace -e-e openopen catcat /etc/os-release/etc/os-release >/dev/null>/dev/null ++++++ exitedexited withwith 00 ++++++ $$ manman 22 openopen ...... TheThe open()open() systemsystem callcall opensopens thethe filefile specifiedspecified byby pathname.pathname. $$ stracestrace -c-c catcat /etc/os-release/etc/os-release >/dev/null>/dev/null %% timetime secondsseconds usecs/callusecs/call callscalls errorserrors syscallsyscall ------------ ---------------------- ---------------------- ------------------ ------------------ -------------------------------- 31,4031,40 0,0005910,000591 590590 11 execveexecve 13,5113,51 0,0002540,000254 2828 99 mmapmmap 8,678,67 0,0001630,000163 4040 44 mprotectmprotect 7,207,20 0,0001350,000135 2222 66 readread 6,956,95 0,0001310,000131 3232 44