THE SEARCH FOR A NEW DEBUGGER GPL v3 of LLDB, and our plan is that it will become the standard debugger in the FreeBSD base system. As with other GNU projects, in 2007 GDB’s license changed to version 3 of the GNU Public License (GPLv3). The GPLv3 includes some LLDB DESIGN restrictions that major FreeBSD contributors and LLDB is built as a set of modular components on consumers find unpalatable, and to date the top of the LLVM compiler infrastructure project project has avoided including any GPLv3- and the Clang front-end. Reusing Clang and licensed code in the base system. LLVM components allows for a great deal of functionality with lower effort and smaller code SEARCH FOR A NEW size compared to other debuggers. For example, LLDB has a full Clang compiler built in, used for DEBUGGER its expression parser. If an expression is accept - With a stale version of GDB in the base system able in a project’s source code, it will also be and no clear path forward, it was clear FreeBSD handled by LLDB’s expression parser, allowing a needed a new debugger. Several open-source user to examine complex classes and data types debugger projects were formed, both within and in great detail and with confidence. LLVM pro - outside of the FreeBSD community. None of them vides LLDB’s processor-specific support, including reached critical mass to sustain development and disassembly support and CPU-specific function - produce a viable debugger we could use. ality. The modular design also provides the foun - Then at their 2010 World-Wide Developer dation for straightforward support of new LL Conference, Apple announced they had their processors, languages, and platforms. own debugger project, LLDB. It was released as A number of overall goals guide LLDB’s open source in June of that year, and the next design, and many of these derive from Clang year it became the default debugger for Xcode, and LLVM. To achieve high performance and Apple’s IDE. LLDB is provided under the reduce memory usage, LLDB attempts to parse University of Illinois/NCSA license which is a per - only the debugging information required to per - IN F ree BSD missive, BSD-like license that is an ideal match form an action. Threaded, high-performance for the FreeBSD project’s licensing philosophy. classes provided by LLVM also contribute to by Ed Maste LLDB has since grown beyond an Apple proj - LLDB’s speed (see chart below). ect, with major contributions coming from LLDB aims to allow customization throughout ne attribute that distinguDishes FreeBSD fBrom other open- open-source groups within companies like Intel so that a user can tailor the debugging experi - source operating systems is the concept of the “base and Google, and from FreeBSD, Debian, and ence. Variable and value display, type formatters, O other independent, open-source projects. summary information, commands, prompts, and Osystem,” an integrated core that is developed, maintained, Several people contributed to the FreeBSD port aliases can all be configured. tested, and released as an integrated whole. Some major lldb Python IDE components of the base system are the kernel, userland libraries, system binaries, and development toolchain –and a key component of the toolchain is the debugger. The GNU debugger, GDB, has long served as This work produced a growing set of LLDB APIs the base system’s debugger. In 1993, the very changes, and the effort increased with each LLDB Core first FreeBSD release included GDB 3.5. Every new import. Project members attempted to EXPRESSION PARSER EXPRESSION PARSER PLUGINS release since has included a version of GDB. have the changes incorporated to the upstream ABI OPERATING SYSTEM

The project followed GDB’s development for GDB project, but met resistance from the pro - COMMANDS DISASSEMBLER PLATFORM more than a decade, and a number of different ject’s maintainers. Eventually this growing main - HOST ABSTRACTION DYNAMIC LOADER PROCESS contributors incorporated new versions into the tenance effort wore on the team, and the last INTERPRETER INSTRUCTION SYMBOL FILE FreeBSD source tree. import was GDB 6.1.1 in June 2004. SCRIPT SYMBOL INTERPRETER JIT LOADER SYMBOL VENDOR TARGET LANGUAGE RUNTIME 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 SYSTEM RUNTIME OBJECT CONTAINER UNWIND ASSEMBLY 4.16 4.18 5.0 5.2 5.2.1 6.1.1 6.3 6.4 6.5 6.6 6.7 7.0 DATA FORMATTERS UTILITY OBJECT FILE

LLVM or CLANG-provided components 12 FreeBSD Journal Sept/Oct 2014 13 (Commands for Comparison continued) GDB LLDB Like Clang and LLVM, LLDB inherently sup - interface is also available for external use; a Launch a process in a new terminal window ports multiple CPU architectures and platforms Python script can create a debugger object and n/a (lldb) process launch --tty -- args within the same debugger binary. For example, then use it to examine and control a debuggee’s the same debugger could be used to locally state, evaluate expressions, and so on. Attach to a process by pid debug a native FreeBSD/amd64 application, (gdb) attach pid (lldb) process attach --pid pid examine a core file from a FreeBSD/MIPS sys - (lldb) attach -p pid tem, and remotely debug an application run - LLDB USE LLDB’s command interpreter is designed with a ning under a debug server on Linux. It’s also Source level single step consistent, structured syntax. Commands gen - possible to have multiple debug sessions run - (lldb) thread step-in erally follow the pattern “noun verb”—for ning simultaneously in the same debugger. (gdb) step (lldb) step example “thread list” or “breakpoint set ”. The As a debugger framework, LLDB is designed (gdb) s (lldb) s command syntax is somewhat more verbose to be embedded in, or used by, other projects. than GDB, and adapting may take some effort Source level single step over function calls IDEs and graphical front ends can easily incorpo - for longtime GDB users. The benefit is that the (lldb) thread step-over rate and build on LLDB’s functionality, using a command set is discoverable and regular; tar - (gdb) next (lldb) next C++ API. A full scripting API is also provided, geted autocompletion can provide relevant (gdb) n (lldb) n currently supporting Python bindings. Python is options to the user. As with GDB, commands usable from within LLDB, at the command line, may be abbreviated to the shortest unique Source level single step out of current function for controlling execution after breakpoints, and prefix. An example of starting a debug (lldb) thread step-out for implementing new commands. The scripting (gdb) finish (lldb) finish session may look like: Instruction level single step t (gdb) stepi (lldb) thread step-inst (gdb) si (lldb) si % lldb (lldb) target create /bin/ls Instruction level single step over function calls Current executable set to ‘/bin/ls’ (x86_64). (gdb) nexti (lldb) thread step-inst-over (lldb) breakpoint set -name main (gdb) ni (lldb) ni Breakpoint 1: where = ls`main + 33 at ls.c:163, address = 0x00000000004023f1 (lldb) process launch Return immediately from current frame (gdb) return return expression (lldb) thread return return expression LLDB has a powerful support for command aliases, and includes a built-in set of aliases for many GDB commands. Using the GDB aliases, the same result as above could be achieved with: Set a breakpoint at all functions by name (lldb) breakpoint set --name name t (gdb) break name (lldb) br s -n name % lldb /bin/ls (gdb) b name (lldb) b name Current executable set to ‘/bin/ls’ (x86_64). (lldb) b main Set a breakpoint by file and line Breakpoint 1: where = ls`main + 33 at ls.c:163, address = 0x00000000004023f1 (lldb) breakpoint set --file file --line line (lldb) run (gdb) break file :line (lldb) br s -f file -l line (lldb) b file: line

The built-in aliases are limited though, and some of the more esoteric overloaded functionality Set a breakpoint in all C++ methods by name provided by GDB commands is not available through the aliases. This is particularly true for break - (lldb) breakpoint set --method name points—in GDB the breakpoint command argument may be a line number, file name, function, or (gdb) break name (lldb) br s -M name address, with sometimes overlapping or conflicting meaning. Migrating to LLDB’s syntax and relying assuming no C functions have the same name on the substring match to allow more concise commands is likely to be most effective. List breakpoints (gdb) info break (lldb) breakpoint list t (lldb) br l Delete a breakpoint Some GDB and LLDB Commands for Comparison (gdb) delete 1 (lldb) breakpoint delete 1 GDB LLDB (lldb) br del 1 Launch a process with no arguments (lldb) process launch (gdb) run (lldb) run Show arguments and local variables (gdb) r (lldb) r (gdb) info args (lldb) frame variable and (lldb) fr v Launch a process with arguments args (gdb) info locals (gdb) run args (lldb) process launch -- args (gdb) r args (lldb) r args 14 FreeBSD Journal Sept/Oct 2014 15 (continues) The command (previous page chart) can create multiple targets in a provide short forms for relative frame selection. “target create” With a frame selected the debugging session with showing the active ones: “frame vari - “target list” command will display function argu - able” t ments and local variables that are in scope. (lldb) target list Current targets: CONTROLLING THE target #0: /bin/app1 ( arch=x86-64-unknown-freebsd10.1, platform=host ) target #1: /bin/app2 ( arch=x86-64-unknown-freebsd10.1, platform=host, pid=81166, state=exited ) DEBUGGEE LLDB groups the single-stepping process control To switch between them use . commands under the top-level thread com - “target select ” mand. steps a single “thread step-in” source line, continuing into function calls. BREAKPOINTS also steps a single “thread step-over” Breakpoints are set with the “breakpoint set” existing breakpoint specification results in new source line, but does not stop inside of a func - command. Breakpoints may be set at a given locations being added to that breakpoint. tion call. continues until “thread step-out” LL address, filename and line number, function or Similarly, unloading a shared library may the program returns from the current function. method name, or upon language-specific remove breakpoint locations. A breakpoint The command “thread until ” exceptions. Breakpoints may also be restricted remains set, but in an unresolved state after continues until the program reaches the speci - to a specified thread or shared library. unloading all of its locations. fied source file line, or it returns from the cur - Breakpoints are maintained as logical Whenever the debuggee process stops, rent function. breakpoints within LLDB, which then resolve LLDB prints relevant information: the thread The stepping commands have aliases to to one or more locations. The logical break - that stopped, process location information match GDB: or for thread step-in, SCRIPTING “s” “step” point and each resolved location are given including the address, filename, and line num - or for thread step-over, and LLDB’s scripting interface may be accessed in “n” “next” “f” multiple ways. The most basic is the integer identifiers, joined with a dot. For ber, the current function and its arguments, or for thread step-out. “script” example, if the third breakpoint matches two and the stop reason. The reported stop reason “finish” comDmand, which invokes thBe embedded inter - function names and thus resolves to two loca - may be a breakpoint, watchpoint, signal, preter and may be used to query current pro - tions, they will be called “3.1.” and “3.2”. address exception, or one of a number of tar - DATA FORMATTERS gram state through a set of convenience vari - LLDB has built-in support for a number of high- Breakpoints remain live throughout the get- or language-specific reasons. Finally, a ables. Python may also be used to implement level data structure formats used by language debugging session, so loading a shared library small portion of the source code at the cur - new LLDB commands. runtimes and libraries. These take the internal with a function or method that matches an rent address is shown. LLDB can also invoke a script after hitting representation of a variable and display it in a a breakpoint. The script can then control pro - t convenient user-facing format, as might be used gram state (for example, continuing the in source code. process), allowing for very complex breakpoint * thread #1: tid = 100641, 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe760) For FreeBSD, the C++ runtime library format - conditions. + 33 at ls.c:163, name = 'ls', stop reason = breakpoint 1.1 ters are likely the most valuable. LLDB includes Finally, LLDB can be used entirely from a frame #0: 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe760) + 33 at ls.c:163 support for the two of interest in FreeBSD: Python script, without involving the stand-alone 160 #ifdef COLORLS libc++ and GNU libstdc++. lldb binary. A script can cre - “import lldb”, 161 char termcapbuf[1024]; /* termcap definition buffer */ As an example, a will show ate a debugger instance and then a target, set std::string 162 char tcapbuf[512]; /* capability buffer */ just the string contents by default, when the breakpoints, launch, single step, and continue -> 163 char *bp = tcapbuf; type formatter is enabled: the target, and examine variables or evaluate 164 #endif expressions. 165 t 166 (void)setlocale(LC_ALL, ""); (lldb) (lldb) expression str LLDB IN F ree BSD ROADMAP (string) $1 = “This is a string.” Ongoing development effort on the LLDB Formatters may be disabled in order to see the FreeBSD port takes place directly in the LLDB repository. It currently works well for the amd64 EXAMINING DEBUGEE STATE sequent commands. full details of the data structure, if necessary: To obtain a stack backtrace use the architecture, for both live After encountering a breakpoint or stopping command, also t and core file-based userland for another reason, LLDB selects the most rele - “thread backtrace” available with the alias. By default the lldb) type category disable gnu-libstdc++ debugging. Core file debug - vant thread. This will be the one that encoun - “bt” current thread’s backtrace is shown, but a dif - ging support also exists for tered a breakpoint, received a signal, per - (lldb) expression str ferent thread index may be provided as an the MIPS architecture. formed an invalid memory access, or otherwise (string) $2 = { argument, or to show the stack for _M_dataplus = {_M_p = “This is a string.”} Between CPU support cur - triggered the stop. “all” each thread. rently in progress by Google The command lists all } “thread list” While examining a backtrace the and work ongoing in the active threads in the debuggee, with “frame “thread command may be used to choose a FreeBSD community, we choosing the desired thread for sub - select” select” specific frame. The and aliases expect to support 64- and “up” “down” 16 FreeBSD Journal Sept/Oct 2014 17 THE SEARCH FOR A NEW DEBUGGER 32-bit versions of the x86, ARM, MIPS, and LLDB snapshot is currently built without script - PowerPC CPU architectures. ing support. It remains functional, but as a A Google Summer of Code (GSoC) 2014 result some of the more interesting and project delivered a proof-of-concept for advanced features are not available. We are FreeBSD kernel debugging support; additional evaluating different approaches to address this, work is required to refine this before it can be likely migrating the scripting interface to a run- integrated into LLDB. time rather than compile-time option. This One key component under development in would allow all of LLDB’s Python capabilities to the LLDB project is a remote debugging stub. be enabled by simply installing the Python This allows an LLDB instance running on one package or port. computer to access and control a process run - We expect to update Clang and LLVM in the ning on another. This is especially important FreeBSD base system to version 3.5 in the near for debugging on small embedded devices, future. LLDB will be updated at the same time, which may lack the memory or computing and we then expect to enable building it by power to accommodate a full-featured debug - default. • ger. The debugging stub is initially being implemented for Linux, but the port to FreeBSD is relatively straightforward. Ed Maste manages project development for Snapshots of the LLDB tree are imported the FreeBSD Foundation and works in an engi - into FreeBSD-Current on an occasional basis. neering support role with the University of LLDB is not yet built by default, but may Cambridge Computer Laboratory. He is also be enabled by adding WITH_LLDB=yes a member of the elected FreeBSD Core Team. to before running “ /etc/src.conf make Aside from FreeBSD and LLDB, he is a contrib - as described in the FreeBSD buildworld” utor to a number of other open-source proj - Handbook. ects, including QEMU and Open vSwitch. He One complication in the FreeBSD base sys - lives in Kitchener, Canada, with his wife, tem is that it does not include Python, so the Anna, and sons, Pieter and Daniel.

18 FreeBSD Journal