Dtrace: “total surveillance” Marcin Kula Campus Ambassador [email protected] www.ee.pw.edu.pl/~kulam Agenda Who am I Introduction DTrace Demo Next steps Who am I • Student of Warsaw University of Technology • Campus Ambassador • I am interested in > OS > Solaris > FreeBSD > ... What is DTrace? Part of OpenSolaris OpenSolaris ZettaByte Filesystem Dynamic Tracing (DTrace) Solaris Containers Image Packaging System Predictive Self-Healing Secure Execution 188 Open Source Apps Compatibility Guarantee Solaris and OpenSolaris • Enterprise class • Web economy • Unmatched • Latest Solaris innovations level of support • Short release cycle • Long release cycle (every 6 months) (3-5 years) How did you analyse these? Example Dynamic Languages Java, PHP User Executable /usr/bin/* Libraries /usr/lib/* Syscall Interface man -s2 Kernel ZFS,TCP Hardware disk data controller How did you analyse these? Example Before DynamicDynamic Languages Languages Java, PHP debuggers UserUser Executable Executable /usr/bin/* truss -ua.out apptrace, /usr/lib/* LibrariesLibraries sotruss SyscallS yscall Interface Interface man -s2 truss KernelKernel ZFS,TCP mdb Hardware disk data kstat Hardware controller How did you analyse these? Example Before DTrace DynamicDynamic Languages Languages Java, PHP debuggers Yes UserUser Executable Executable /usr/bin/* truss -ua.out Yes apptrace, /usr/lib/* Yes LibrariesLibraries sotruss SyscallS yscall Interface Interface man -s2 truss Yes KernelKernel ZFS,TCP mdb Yes No Hardware disk data kstat Hardware controller (Indirectly, yes) What is DTrace? • Introduced in Solaris 10 • Powerful infrastructure > diagnose system and application behavior • Infrastructure includes > A tool > A DTrace language, “D”, with an interpreter > An instrumentation framework • Which Operating Systems has DTrace ? > ... What is DTrace? • Introduced in Solaris 10 • Powerful infrastructure • Which Operating Systems has DTrace ? > Solaris 10 > OpenSolaris > FreeBSD > MacOS X Leopard > QNX > ... DTrace Features • Dynamic Instrumentation > Zero probe effect when disabled • Unified Instrumentation > Instrument both kernel and applications > Can follow flow-control across boundaries • Actions & Predicates • High-level Control Language > C-like language describes predicates & actions > All ANSI C operators (add, subtract, etc) > Access to kernel variables and data types • Data aggregation • Data Integrity > Errors always reported DTrace • “Probes” are instrumentation points • Probe names consist of > Provider DTrace kernel module > Module Name of a kernel module or the name of a user library > Function The name of the program function in which the probe is located > Name It gives you some idea of the probe’s semantic meaning Provider Module Function Name Probe: syscall::open:entry Predicate: / pid == 123 / Action: { trace(1); } Providers • fbt • pid • fsinfo • profile • io • sched • lockstat • syscall • mib • sysinfo / vminfo • proc • ... # dtrace -l Providers • pid > probes for tracing any function in a user process > entry : arg0, arg1, … are the values of the arguments > return: arg1 holds the return value • fbt > probes associated with most functions in the kernel > entry : arg0,.. are the values of the arguments > return: arg1 holds the return value fbt:::{ printf(“%s:%s:%s\n”,probemod,probefunc,probename); } Providers • syscall > probes associated with system call in the system > entry : arg0 .. argn are the arguments to the system call. > return: both arg0 and arg1 contain the return value – A non-zero value in the D variable errno indicates system call failure. > ex. syscall::open:entry { printf("%s\n",copyinstr(arg0)); } • io > provider makes available probes related to disk input and output > probes: start, done, wait-start, wait-done > arg[0]: bufinfo_t *, arg[1]: devinfo_t *, arg[2]: fileinfo_t * > ex. /* DEVICE, FILE, RW */ io:::start{ printf("%10s %58s %2s\n", args[1]->dev_statname,args[2]->fi_pathname, args[0]->b_flags & B_READ ? "R" : "W" );} Providers • sched > probes related to CPU scheduling > on-cpu, off-cpu, enqueue,.. • proc > probes pertaining to the following activities: > process creation and termination > LWP creation and termination proc:::exec-success > executing new program images {trace(curpsinfo->pr_psargs);} > sending and handling signals > Ex. create, exec, exit, signal-send, signal-handle • profile > probes associated with a time-based interrupt firing every fixed, specified time interval Providers • sysinfo / vminfo > kernel statistics classified by the name sys / vm • mib > counters in the Solaris management information bases • ip ( since OpenSolaris build 93) > probes for tracing both IPv4 and IPv6 protocols. • lockstat > allow to understand virtually any aspect of locking behavior • ... Providers • sysinfo / vminfo > kernel statistics classified by the name sys / vm • mib > counters in the Solaris management information bases • ip ( since OpenSolaris build 93) > probes for tracing both IPv4 and IPv6 protocols. • lockstat > allow to understand virtually any aspect of locking behavior • ... Free your mind When to use DTrace System monitoring • Probing kernel subsystems > file systems > networks > zones > etc. • Collecting information from whole system > DExplorer tool • Tuning system and applications When to use DTrace Web-development • Server-side • Tools > Ruby > Apache > Java > Glassfish > PHP > MySQL > Groovy > PostgreSQL > Perl > Python • Client-side > Java-script When to use DTrace Software development • Debugging • Profiling > Memory leaks > Lost CPU cycles > Function calls > etc. Gold mind “It needs to borrow your brain to do the thinking” Brendan Gregg demo Get Next Steps Involved > Join Community Open Source University Meetup Change (y)our world http://osum.sun.com/group/pw Next Steps Resources • DTrace community page > http://www.opensolaris.org/os/community/dtrace • Solaris dynamic tracing guide > http://docs.sun.com/app/docs/doc/817-6223 • Wiki DTrace > http://wikis.sun.com/display/DTrace • NetBeans DTrace GUI plugin > http://www.netbeans.org/kb/docs/ide/NetBeans_DTrace_GUI_Plugin_0_4.html • D-Light > http://developers.sun.com/sunstudio/documentation/tutorials/d_light_tutorial/index.html Next Steps TechTalk • OpenSolaris 2008.1 1 • ZFS Questions ? Thank you! Marcin Kula Campus Ambassador [email protected].