SEE TEXT CADETSONLY Blending Tracing & Security on FreeBSD
The FreeBSD operating system has been used in many products that require strong security proper- ties—from storage appli- ances, to network switches and routers as well as gaming consoles. Over the 20-plus years of FreeBSD’s By Jonathan Anderson, development, there have George V. Neville-Neil, been significant additions Arun Thomas, and to the system in the area Robert N. M. Watson of security.
12 FreeBSD Journal he Jails [1] system introduced what about ‘hello, world.’” [6] Using print statements today would be considered a lightweight for a logging system in code that is supposed to Tversion of virtualization. The Mandatory have, otherwise, low overhead, is not an option, Access Control (MAC) framework was and so most logging systems are enabled or dis- added in 2002 to provide fine-grained, system- abled at either compile time, using an wide control over what users and programs could #ifdef/#endif statement, or at runtime, by do within the system [2]. The Audit subsystem wrapping the logging in an if statement. added the ability to track—on a system-call-by-sys- An example of this idiom in the C language is tem-call basis—what actions were occurring on shown below. the system and who or what was initiating those actions [3]. More recently Capsicum has intro- #ifdef LOGGING duced capabilities into the operating system that if (log) are the basis for application sandboxes [4]. printf("You have written %d bytes", len); As part of a new research project, the Causal #endif /* LOGGING */ Adaptive Distributed, and Efficient Tracing System (CADETS), we are adding new security primitives The next problem with print-based logging sys- to FreeBSD as well as leveraging existing primitives tems is that they are both static and prone to to produce an operating system with the maxi- errors. If the programmer did not expose a piece mum amount of transparency. Apart from the of information via the logging system before the security implications, having better visibility into code was built, then it will not be possible, with- systems will help us to improve overall perform- out modifying the code, to learn about any other ance and provide new runtime debugging tools. data upon which the same function might be In this article, we’ll cover our work with DTrace operating. and the Audit system, which form the core com- Together, these two problems mean that most ponents of the work, and talk about the chal- high-performance systems, such as operating sys- lenges of using DTrace as an always on tracing tems, are shipped without logging enabled, and system to track security and other events. when logging is enabled, the data that is available is limited to whatever the original programmer Starting at the Beginning wished to expose. As open-source developers, we are used to the One of the main components of FreeBSD we are idea that we can “just recompile the code,” but in exploiting in CADETS is DTrace. Originally devel- production systems, that’s not always possible. oped for Sun’s Solaris operating system in the early Imagine you have sold a system to a large bank. years of the 21st century [5], DTrace was meant to At 3 a.m., the system has a fault of some sort and solve the following problem: most software sys- logs an error. Someone in the IT department gets a tems have some sort of logging framework, which message reporting the fault, they call support, sup- usually depends on the ubiquitous printf func- port calls a programmer, the programmer then tion and formats text and sends it to some form says, “Stop the system, rebuild the code, and of console. rerun it with logging turned on.” Handling errors in that way is terrible both for the customer and printf("Hello world!"); for the developer. The customer will be annoyed at having to rebuild and restart their system, and the All C programmers know the code shown developer is unlikely to get helpful information above and every programmer knows the equiva- because the error is now far in the past. Enter lent idiom in their own language, whether it’s Dynamic Tracing. Python, Rust, Go, or PHP. There are a few prob- DTrace was designed to always be available for lems with using print statements for logging sys- use, without reducing system performance and tems. The first problem is that print statements without running the risk of outright crashing the have a high performance overhead. If you’ve ever system. The clearest way to think about DTrace is wondered about how much work is done on the as a runtime debugger with significant logging programmer’s behalf by printf then see Brooks and statistical capabilities. DTrace avoids the over- Davis’s “Everything you ever wanted to know head of printf based logging systems by using a
May/June 2017 13 few tricks to subvert the execution of compiled introduced by tracing gets too high, then the ker- code. The complete details are covered in The nel will terminate the tracing as a form of self- Design and Implementation of the FreeBSD preservation. The second tenet of the DTrace sys- Operating System [7], but, briefly, DTrace can tem, that the tracing system must not unduly tax override the entry or exit point of any function the system, is one of the key challenges of using that is compiled into the kernel or into a user- DTrace as a security technology. An attacker that space program. The way in which functions are knows there might be tracing will first cause collected into libraries and programs is a well- there to be a great deal of irrelevant load on the defined process, and each function has a set of system, causing the tracing system to exit, and instructions that indicate where the function then they will go about attacking the system. begins. When DTrace traces a function, it Another component of the CADETS project looks replaces a few instructions with some of its own at the provenance of traces in order to thwart so that when the code reaches that point, such attacks on the tracing system itself. DTrace’s code gets called first, and it can collect The majority of the current use cases for and process the function’s argument. The practi- DTrace involve using it as a runtime debugger, cal upshot of this is that when DTrace is not in turning on tracing when someone suspects there use, it has zero overhead. is a problem on a system, rather than leaving the tracing running all the time. A small subset of Tracing for Security users have built complex telemetry systems How can we apply DTrace to the problems of around DTrace, including Fishworks [8], but in security? One aspect of security is finding where these first attempts at always-on tracing the the bad actor lies in a system. A key question to number of things being traced was a small subset ask when looking at a system is “Who did what of the possible tracepoints. To build a system that to whom and when?” We’ll refer to this as has complete visibility, we need to not only have “Question 1.” Establishing the tree of operations always-on tracing, but to increase the perform- such that we can trace it back to its root is one ance of the tracing system such that the over- part of forensic analysis that can help us find our head collecting the data does not overwhelm the bad actor and also show us what we need to system’s ability to do productive work. Another change to prevent future security breaches. requirement of a tracing system targeted at secu- Imagine that, regardless of the performance rity is that trace records cannot be dropped or cost, we had complete transparency into every lost due to high load. DTrace was designed in operation performed on a computer system. such a way that under high load it was accept- With sufficient time, analysis, and tooling, we able to drop trace records, long before the kernel dtrace would be able to take the output generated by might terminate the collection process the tracing system and find out the answer to due to the system being unresponsive. A system Question 1. that is trying to collect trace data for later foren- DTrace gives us the basis on which to build sic analysis turns that concept on its head. such a system, but there remain some challenges. Any system where tracing is always on will gen- In the previous section, we stated that using erate a lot of data, and that data will need to be some clever tricks to replace certain instructions analyzed to track down attackers and how they at runtime, DTrace would have zero overhead are able to compromise the system. There are sev- when not in use. That feature of DTrace was eral systems for taking arbitrary textual output what made it viable to ship it, by default, with from various tools and trying to make some sense Solaris, and the FreeBSD and Mac OS in the first of it, including Splunk [9]. The goal of the CADETS place. It was not until there was a way to ship a project is to produce data for use by such tools. It tracing system that didn’t have a performance will be much easier for consumers if the data is in impact on a running system that such a system a machine-readable format. could be fielded. What happens when DTrace begins tracing? Depending on what is being Trace Records for traced and how much data is being collected, the Software Tools overall performance of the system will be impact- As we began our work using DTrace as an ed to a greater or lesser degree. If the overhead 14 FreeBSD Journal always-on tracing system, we quickly realized that format. The machine-readable output tags each parsing the voluminous output generated by the element, starting with the probe, which is an system would present a problem not only for object that contains several elements, including the human analysts, but also for any tools that would cpu, id, func and name, all of which appeared, be consuming the data. One of the first features untagged, in the machine-unreadable output of we added to DTrace for CADETS was machine-read- Example 1. One addition for machine-readable out- able output. Using the libxo library, we converted put is the timestamp element, which reports the DTrace to not only produce plain text, but also time that the probe fired, in nanoseconds, since the XML, JSON, and HTML, the three output formats UNIX epoch. While a DTrace script can output the supported by libxo. At the time that we added time using either the timestamp or walltime machine-readable output, the Illumos version of stamp variables, we believed that having a time DTrace did have a way of outputting JSON, but this stamp in every machine-readable record would sim- was via a print-like operation rather than a perva- plify building tools for security forensics. While sive change. In the CADETS version of DTrace, a probes may fire in parallel on different cores, indicat- single command-line option changes all the output ed by the cpu variable, modern Intel-based systems have a synchronized ``` timestamp, which # dtrace -n 'syscall::write:entry' DTrace uses, meaning dtrace: description 'syscall::write:entry' matched 2 probes that the time stamps CPU ID FUNCTION:NAME are an excellent indica- 0 59780 write:entry tion of the order of 0 59780 write:entry operations on a single Example 1``` system. ``` DTrace and # dtrace -O json -n 'syscall::write:entry' Audit dtrace: description 'syscall::write:entry' matched 2 probes audit CPU ID FUNCTION:NAME The subsys- { tem has been a part "probe": { of FreeBSD since "timestamp": 3594774042481656, 2004 and is an "cpu": 1, optional kernel com- "id": 59780, ponent, along with a "func": "write", user space daemon auditd "name": "entry" , which } implements a “fine- Example 2 grained, configurable logging of security related events [10]. the user sees from DTrace from plain text into a It was built to meet the “Common Criteria (CC) machine-readable format. Common Access Protection Profile (CAPP) evalua- The code samples above show the differing output tion,” a security standard set out by the U.S. between the plain, textual output, and machine- Government [11]. The audit subsystem adds a set of readable output. Our example asks DTrace to trace handcrafted tracepoints via C macros to parts of the all calls into the write system call. Example 1 kernel where access to data and resources take place. shows the default, textual output from the For example, in a system with audit enabled, any dtrace command, which is arranged in columns. time a file descriptor is accessed, a note is made in an To write a tool that parses the output requires audit record. Audit records are periodically flushed to knowing quite a bit about the output, because the permanent storage. columns are only labeled at the start of the output. Our recent work with DTrace and security led Example 2 shows the same tracepoint, but with the us to desire a bridge between the audit system addition of the -O json command line argument, and DTrace. Robert Watson added an audit instructing dtrace to give us all output in JSON provider to the DTrace system in FreeBSD. A
May/June 2017 15 DTrace provider gives access to a set of tracepoints nect(2), and then filter those events based on the from within DTrace. Some well-known examples IP addresses they contain. Performing this data of providers are those dealing with Function reduction as close to the source of the information Boundary Tracepoints (fbt), System Calls (syscall), as possible not only reduces the overall load on and network protocols (tcp, udp, ip). The audit the system, but it also reduces the amount of data provider gives DTrace the ability to record informa- a human analyst or software tool has to look at tion about audit events that occur on the system during the later analysis phase. while also applying DTrace features, such as filter- ing events via predicates and collecting statistical OpenDTrace: information via aggregations. The Future of DTrace Using the audit system in the absence of DTrace, Besides Illumos, two other operating systems proj- we did not have a convenient way to write runtime ects have ported and adopted DTrace. FreeBSD analysis scripts that allowed us to more finely target has had a port of DTrace since 2008 and Apple’s the processes that we wanted to investigate. The Mac OS since 2007, where it is also integrated audit system will target a particular process, but we into the Instruments performance-analysis tool. wanted to be able to collect data only when that With the demise of Sun Microsystems in 2010, the process took a particular action. development of DTrace split, with some work Consider a scenario where we want to see who being done within Oracle, but much of it moved is talking to a web server; we might decide that onto Illumos, the fully open-source follow-on to we only want to know about connections that are OpenSolaris. Both FreeBSD and Mac OS continued coming from a specific set of Internet addresses, to bring in changes from Illumos, but these were, perhaps because we know those addresses have for the most part, bug fixes rather than large new already been identified as part of a botnet. With features. All three groups have done what they the Audit Provider we can write a simple D script could to share code among themselves, but there that asks only for the audit events relating to con-
Premier VPS Hosting RootBSD has multiple datacenter locations, and offers friendly, knowledgeable support staff. Starting at just $20/mo you are granted access to the latest FreeBSD, full Root Access, and Private Cloud options. www.rootbsd.net
16 FreeBSD Journal are significant differences among all three kernels. unified upstream for DTrace code that can then DTrace, although it was written in a good, be easily imported into other operating systems, portable style, requires many specialized hooks including FreeBSD, Mac OS, and Illumos, as well into the operating system, and this has resulted in as others. The approach is similar to that taken by some level of code drift. For example, the FreeBSD OpenBSM [12] and OpenZFS [13]. With this uni- version of DTrace works on both ARMv8 and fied code base in place, we can then add features ARMv7 processors, but this is not yet true on that apply to all the downstream OS consumers of Illumos. As part of our CADETS work, we realized DTrace far more quickly than we do today. . that we wanted to add new features to DTrace and also to further abstract the code from any Acknowledgment particular operating system, which led us to create The authors thank Ripduman Sohan for com- OpenDTrace. ments that greatly improved the manuscript. • The aim of OpenDTrace is to provide a single,
JONATHAN ANDERSON is an Assistant Professor in degree in computer science at Northeastern Memorial University of Newfoundland's Department of University in Boston, Massachusetts, and is a mem- Electrical and Computer Engineering, where he works ber of ACM, the Usenix Association, and IEEE. He is at the intersection of operating systems, security, and an avid bicyclist and traveler and currently lives in software tools such as compilers. He is a FreeBSD com- New York City. mitter and is always looking for new graduate students ARUN THOMAS is a researcher at BAE Systems R&D with similar interests. and the principal investigator of the the Causal, GEORGE V. NEVILLE-NEIL works on networking Adaptive, Distributed, and Efficient Tracing System and operating system code for fun and profit. He (CADETS) project. also teaches courses on various subjects related to DR ROBERT N. M. WATSON is a Senior Lecturer programming. His areas of interest are code (Associate Professor) at the University of Cambridge spelunking, operating systems, networking, and time Computer Laboratory, where he leads research span- protocols. He is the coauthor with Marshall Kirk ning operating systems, security, and computer archi- McKusick and Robert N. M. Watson of The Design and tecture. He is a FreeBSD developer, member of the Implementation of the FreeBSD Operating System. For FreeBSD Foundation Board of Directors, and coauthor over 10 years he has been the columnist better of The Design and Implementation of the FreeBSD known as Kode Vicious. He earned his bachelor’s Operating System (second edition).
This work has been sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA8650-15-C-7558. The views, opinions, and/or findings contained in this paper are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of the Department of Defense or the U.S. Government. REFERENCES [1] “Jails: Confining the omnipotent root.” Poul-Henning Kamp
May/June 2017 17