David R. Cheriton School of Computer Science, University of Waterloo, Ontario, Canada A Program Extractor Suite for C and C++: Choosing the Right Tool for the Job Jingwei Wu 1 and Richard Holt 2 Software Architecture Group (SWAG) David R. Cheriton School of Computer Science University of Waterloo Waterloo, ON, Canada Abstract This report describes a suite of program extractors which we developed by adopt- ing and extending existing program parsing and extraction techniques or tools. This suite is called CX because it is mainly targeted at extracting facts from C and C++ programs. This suite is currently composed of four extractors: CPPX, BFX, LDX and CTSX. The main goal of creating CX is to provide a convenient set of program extractors that can complement each other and work in a systematic manner. The benefits of this extractor suite will be discussed in terms of two practical applica- tions: (1) creating program comprehension pipelines to support various understand- ing tasks, and (2) building an open source software evolution database (EvolDB) to support empirical research on software evolution. 1 Introduction Program extraction is important to program comprehension and maintenance tasks [FSG04][MN96]. For example, an architect may need to monitor changes made to a software system to identify changes violating the design constraints of the system. In this case, a program extractor is needed to collect structural information to support continual maintenance. Empirical research on software evolution often requires using a robust and efficient extractor to collect struc- tural artifacts from a large number of versions, sometime up to several hundred versions. A researcher may also be interested in deriving a domain reference architecture by studying several systems from the same domain (e.g.,the com- piler domain and the web browser domain). In these various examples, one needs to choose a program extractor by making an appropriate tradeoff among 1 Email: [email protected] 2 Email: [email protected] requirements on accuracy, efficiency and robustness. In an empirical study of software evolution, a researcher is not likely to use an accurate extractor which spends months in extracting structural artifacts from many historical versions of a long-lived system (e.g., Linux [Lin04]). A robust and efficient extractor, though it may be less accurate, is more likely to prevail in such a case. No single extraction technique can meet the highest standards on accu- racy, efficiency and robustness and in the meanwhile support as many tasks as possible [SHE02]. Instead, a very small set of complementary extractors should be developed to support a wide range of tasks and meet extraction requirements to varying degrees. This paper describes our effort in developing a suite of program extractors for the C and C++ programming language. The main design goals of this suite are as follows. ² It should be simple and convenient to use. ² It should scale up to handle large programs. ² It should support diverse program analysis tasks. ² It should cover the entire build process of software. ² It should efficiently handle large numbers of versions. CPPX [CPP02] is the first extractor we added to the CX suite. CPPX is a C/C++ source code extractor based on the front end of GCC [GCC02]. It was developed by the SWAG group at the University of Waterloo. CPPX outputs detailed information of a program at the abstract syntax graph level. However, CPPX is neither efficient nor robust in handling very large programs over many versions. This motivated us to develop several lightweight extractors to complement it. These new extractors and CPPX form a suite that provides a simple and cost-effective solution for a wide variety of program analysis tasks (see Section 3). The rest of this paper is organized as follows. Section 2 describes the four extractors from the CX suite and their pros and cons. Section 3 describes two major applications of the CX suite. Lessons learned from each application are also summarized. Section 4 further compares four CX extractors and discusses their role in supporting systematic extraction of C/C++ programs. Section 5 considers related work. Section 6 draws the conclusion. 2 The CX Suite The CX suite consists of four program extractors which are created by means of adapting free open source tools, including GNU Compiler Collection (GCC) [GCC02], GNU Binutils (binary utilities) [BUM02], Ctags [Cta04] and Cscope [Csc04]. These extractors are briefly summarized below. ² CPPX is a C/C++ source code extractor based on the GCC frontend. It relies on the preprocessing, parsing, and semantic analysis of GNU g++ 2 and can produce program facts as detailed as abstract syntax graph (ASG). ² BFX is a binary code extractor built on the Binary File Descriptor (BFD) library. It parses binary code to locate definitions of functions and variables and outputs symbol references to these definitions. ² LDX is a binary code extractor based on the GNU code linker LD. It reuses BFX to process individual binary files and then resorts to the real code linker in the resolution of cross-references among different binary files under a specific system configuration. ² CTSX is an efficient and robust source code extractor built upon Ctags and Cscope. The Ctags is invoked to locate source program entities and Cscope is used to extract references to source program entities. We now describe the four CX extractors and their pros and cons in detail. 2.1 CPPX CPPX is a general-purpose parser and fact extractor for C and C++ programs. It relies on the preprocessing, parsing, and semantic analysis of GNU g++, and produces an abstract syntax graph in accordance with the Datrix model [Bel01]. The produced fact base is in TA [Hol02], GXL [GXL02], or VCG [San95] format. Abstractly speaking, CPPX output is an E/R graph, which is essentially the abstract syntax graph of the source program being extracted. The vertices represent the program’s templates, classes, methods, compound statements, and expressions down to the lowest level of constants and variable references. The graph edges represent syntactic relationships as well as semantic facts linking identifiers to their declarations, function calls to their targets, objects to their types, and most things to their enclosing scopes. From the CPPX out- put graph it is (almost) possible to reconstruct the original program [LHM03]. CPPX is suitable for use in architectural recovery, data flow analysis, pointer analysis, program slicing, query techniques, source code visualization, object recovery, refactoring, restructuring, re-modularization, and the like. It has been used in both industrial software development environments and aca- demic software engineering research. However, CPPX has several problems. For example, CPPX may take an unusually long time (up to months) to ex- tract facts from large software systems (up to several million lines of code); and the abstract syntax graph, which is overly detailed for many downstream analysis, often needs lengthy transformations to filter out large quantities of unwanted data such as compound expressions and statements. These problems make it necessary to develop more efficient and simplified program extractors. As a result, we have developed three lightweight program extractors, each of which is detailed in the following. 3 cScopeGlb cExternSymbol cObjectFile cArchiveFile RefersToExtern RefersToExtern cFunction cObject ResolvedByExtern RefersTo RefersTo contain cNameRef Fig. 1. BFX fact schema 2.2 BFX BFX (Binary File Extractor) is built on the Binary File Description (BFD) library which is shipped with the GNU Binutils toolkit [BUM02]. Unlike CPPX which deals with source code, BFX extracts facts from binary code (machine code). It can process object modules (.o), archives (.a), dynamic libraries (.so) and executables (.exe). The output is in TA or GXL format and conforms to the fact schema shown in Figure 1. The schema in Figure 1 shows that the BFX output has three levels of granularity: the level of object modules, the level of functions and variables and the level of name references. The cObjectFile class represents files end- ing with .o, .so and .exe. The cArchiveFile represents archive files (.a ). The cFunction and cObject represent functions and variables respectively. The cExternSymbol represents unique string names. There are four rela- tions: a structural relation contain and three reference relations cRefersTo, cRefersToExtern and cResolvedByExtern. The contain relation for any binary module always forms a tree with a universal root of type cScopeGlb. The cRefersTo relation refers to resolved references to symbols defined within the same binary module. The cResolvedByExtern relation denotes references to externally defined symbols and cRefersToExtern means that a definition can be looked up globally by searching for its name 3 , i.e., cExternSymbol. In the above schema, cNameRef has an attribute called kind, which can be assigned a value of F or V. The F value means a function call and the V value means a variable use. 2.2.1 Pros BFX is built on the BFD library. Technically speaking, BFX is equivalent to the binary code dump tool objdump [BUM02]. BFX is independent of source code compilation. Therefore, it is able to extract programs written in 3 For C++ programs, an external definition has a mangled name. 4 LD .a .o .so .exe LDX LD data flow .ta/.ta.gxl LDX data flow Fig. 2. LDX as a substitute for LD programming languages other than C and C++, which include, for example, Fortran and Pascal. The BFX output is more than an order of magnitude smaller than the CPPX output since it extracts only function calls and variable uses. In terms of speed, BFX normally operates in a matter of seconds or at most minutes to extract large systems with thousands of object units. BFX’s performance will be further discussed in Section 4. 2.2.2 Cons BFX is used only after source files are compiled into object units.