Operating systems and Infrastructure Services
Systemtap at work. From a systemtap module to a deployable rpm.
Thomas Oulevey IT-OIS [email protected] / @thomasnomas Table of content
Part 1 Part 2 Systemtap : Corner case : ● Introduction ● Vulnerability ● Scripts ● Packaging ● More scripts ● Feedback Introduction
SystemTap provides the infrastructure to monitor the running Linux system for detailed analysis. Introduction
The essential idea is to name events, and to give them handlers. When SystemTap runs the script, SystemTap monitors for the event; once the event occurs, the Linux kernel then runs the handler as a quick sub- routine, then resumes. Events
There are several kind of events: A synchronous event occurs when any process executes an instruction at a particular location in kernel code. – Entering/exiting a function An Asynchronous events are not tied to a particular instruction or location in code.timer expiration, – Session (begin, end) – Counters, timers. Handlers
A handler is a series of script language statements that specify the work to be done whenever the event occurs.
● Extracting data from the event context, ● Storing them into internal variables, ● Printing results. Probes
An event and its corresponding handler is collectively called a “probe”.
function function_name(arguments) { statements } probe event, event2 {function_name(arguments)} Tapsets
Tapsets are scripts that form a library of pre-written probes and functions to be used in SystemTap scripts.
$ ls /usr/share/systemtap/tapset/ Guru mode
● Code and data memory reference protection are removed ● Guru mode is set by passing the -g option to the stap command. ● It accepts C code enclosed between “%{'' and ”%}” Installation
yum install \ systemtap systemtap-runtime \ kernel-debuginfo \ kernel-debuginfo-common-arch \ kernel-devel Architecture
stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
1: First, SystemTap checks the script against the existing tapset library. 2: SystemTap then translates the script to C to create a kernel module from it. 3: SystemTap loads the module, then enables all the probes. 4: As the events occur, their corresponding handlers are executed. 5: The probes are disabled, and the kernel module is unloaded. My Hello World !
global msg=”Hello World”
probe timer.s(4) { printf("%s!\n",msg) }
$ stap hello.stp -x PID $ stap hello.stp -c command Useful functions
● tid() The ID of the current thread.
● uid() The ID of the current user.
● cpu() The current CPU number.
● gettimeofday_s() The number of seconds since UNIX epoch (January 1, 1970).
● ctime() Convert number of seconds since UNIX epoch to date.
● pp() A string describing the probe point currently being handled.
● thread_indent() "indentation counter". Useful functions thread_indent() Useful function: hist_log Analyze system performance
● IO
● Network
● CPU
● Filesystem
Correlate the needed information for your specific problem.
● http://sourceware.org/systemtap/examples/ A simple example: ttyspy.stp
void tty_audit_add_data (struct tty_struct *tty, unsigned char *data, size_t size, unsigned icanon)
probe kernel.function("tty_audit_add_data") { major=$tty->driver->major; minor=$tty->driver->minor_start + $tty->index; pgrp=$tty->pgrp ; data=kernel_string_n($data,$size); uid=uid() activity_time[major,minor,pgrp,uid] = gettimeofday_s(); activity_log[major,minor,pgrp,uid] = nice_print_f(activity_log[major,minor,pgrp,uid],data,40); } ttyspy in action Corner case : Real world 0-day
“perf_swevent_enabled array out-of-bound access”
● CVE-2013-2094
● https://bugzilla.redhat.com/show_bug.cgi?id=962792
● Local escalation PoC released before CVE (semtex.c)
● Fixed upstream, back ported to Red Hat kernel :( Problems ? Only solutions
lxplus.cern.ch service has many interactive users, around 50.000 ! -> We need a fix now not in 12 hours! -> Redhat published a .stap to sanitize the variable. -> How to publish it to =~ 100 servers. The faulty code Sanitize event, no overflow
#!/usr/bin/env stap %{ #include
function sanitize_config:long (event:long) % { struct perf_event *event; event = (struct perf_event *) (unsigned long) STAP_ARG_event; event->attr.config &= INT_MAX; %} probe kernel.function("perf_swevent_init").call { sanitize_config($event); } Let's package it.
rpm to package, yum to distribute, puppet to apply.
● Three steps :
● Build the systemtap script on a development system for all available kernels.
● Distribute the resulting kernel module to all affected systems.
● Run it and survive to reboot. cve_2013_2094.spec
● yumdownloader –source cve_2013_2094 Red Hat fix !
● Red hat fix available in next days, ● Easy to remove the workaround, ● Small user interaction if needed, ● Only Standard tools were used, ● Easy to automatize for future fixes. => WIN! OIS Questions
?
Eclipse
● The SystemTap Plugin
● http://www.eclipse.org/linuxtools/projectPages/systemtap/
● http://wiki.eclipse.org/Linux_Tools_Project/Systemtap/User_Guide#The_SystemTap_Plugin