Operating systems and Infrastructure Services

Systemtap at work. From a systemtap module to a deployable rpm.

Thomas Oulevey IT-OIS [email protected] / @thomasnomas Table of content

Part 1 Part 2 Systemtap : Corner case : ● Introduction ● Vulnerability ● Scripts ● Packaging ● More scripts ● Feedback Introduction

SystemTap provides the infrastructure to monitor the running system for detailed analysis. Introduction

The essential idea is to name events, and to give them handlers. When SystemTap runs the script, SystemTap monitors for the event; once the event occurs, the then runs the handler as a quick sub- routine, then resumes. Events

There are several kind of events: A synchronous event occurs when any process executes an instruction at a particular location in kernel code. – Entering/exiting a function An Asynchronous events are not tied to a particular instruction or location in code.timer expiration, – Session (begin, end) – Counters, timers. Handlers

A handler is a series of script language statements that specify the work to be done whenever the event occurs.

● Extracting data from the event context, ● Storing them into internal variables, ● Printing results. Probes

An event and its corresponding handler is collectively called a “probe”.

function function_name(arguments) { statements } probe event, event2 {function_name(arguments)} Tapsets

Tapsets are scripts that form a library of pre-written probes and functions to be used in SystemTap scripts.

$ ls /usr/share/systemtap/tapset/ Guru mode

● Code and data memory reference protection are removed ● Guru mode is set by passing the -g option to the stap command. ● It accepts code enclosed between “%{'' and ”%}” Installation

yum install \ systemtap systemtap-runtime \ kernel-debuginfo \ kernel-debuginfo-common-arch \ kernel-devel Architecture

stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

1: First, SystemTap checks the script against the existing tapset library. 2: SystemTap then translates the script to C to create a kernel module from it. 3: SystemTap loads the module, then enables all the probes. 4: As the events occur, their corresponding handlers are executed. 5: The probes are disabled, and the kernel module is unloaded. My Hello World !

global msg=”Hello World”

probe timer.s(4) { printf("%s!\n",msg) }

$ stap hello.stp -x PID $ stap hello.stp -c command Useful functions

● tid() The ID of the current thread.

● uid() The ID of the current user.

● cpu() The current CPU number.

● gettimeofday_s() The number of seconds since UNIX epoch (January 1, 1970).

● ctime() Convert number of seconds since UNIX epoch to date.

● pp() A string describing the probe point currently being handled.

● thread_indent() "indentation counter". Useful functions thread_indent() Useful function: hist_log Analyze system performance

● IO

● Network

● CPU

● Filesystem

Correlate the needed information for your specific problem.

● http://sourceware.org/systemtap/examples/ A simple example: ttyspy.stp

void tty_audit_add_data (struct tty_struct *tty, unsigned char *data, size_t size, unsigned icanon)

probe kernel.function("tty_audit_add_data") { major=$tty->driver->major; minor=$tty->driver->minor_start + $tty->index; pgrp=$tty->pgrp ; data=kernel_string_n($data,$size); uid=uid() activity_time[major,minor,pgrp,uid] = gettimeofday_s(); activity_log[major,minor,pgrp,uid] = nice_print_f(activity_log[major,minor,pgrp,uid],data,40); } ttyspy in action Corner case : Real world 0-day

“perf_swevent_enabled array out-of-bound access”

● CVE-2013-2094

● https://bugzilla.redhat.com/show_bug.cgi?id=962792

● Local escalation PoC released before CVE (semtex.c)

● Fixed upstream, back ported to kernel :( Problems ? Only solutions

lxplus.cern.ch service has many interactive users, around 50.000 ! -> We need a fix now not in 12 hours! -> Redhat published a .stap to sanitize the variable. -> How to publish it to =~ 100 servers. The faulty code Sanitize event, no overflow

#!/usr/bin/env stap %{ #include %}

function sanitize_config:long (event:long) % { struct perf_event *event; event = (struct perf_event *) (unsigned long) STAP_ARG_event; event->attr.config &= INT_MAX; %} probe kernel.function("perf_swevent_init").call { sanitize_config($event); } Let's package it.

rpm to package, yum to distribute, puppet to apply.

● Three steps :

● Build the systemtap script on a development system for all available kernels.

● Distribute the resulting kernel module to all affected systems.

● Run it and survive to reboot. cve_2013_2094.spec

● yumdownloader –source cve_2013_2094 Red Hat fix !

● Red hat fix available in next days, ● Easy to remove the workaround, ● Small user interaction if needed, ● Only Standard tools were used, ● Easy to automatize for future fixes. => WIN! OIS Questions

?

Eclipse

● The SystemTap Plugin

● http://www.eclipse.org/linuxtools/projectPages/systemtap/

● http://wiki.eclipse.org/Linux_Tools_Project/Systemtap/User_Guide#The_SystemTap_Plugin