In Proceedings of the ACM International Symposium on Software Testing and Analysis: Big-New-Idea Track, Portland, Maine, U.S.A., July 17–20, 2006 (to appear). Experimental Program Analysis: A New Program Analysis Paradigm Joseph R. Ruthruff, Sebastian Elbaum, and Gregg Rothermel Department of Computer Science and Engineering University of Nebraska–Lincoln Lincoln, Nebraska 68588­0115, U.S.A. fruthruff, elbaum, [email protected] ABSTRACT Recently, researchers have begun to consider a new approach to Program analysis techniques are used by software engineers to de- program analysis that harnesses principles of experimentation [21, duce and infer characteristics of software systems. Recent research 31]. Such experimental program analysis techniques might be able has suggested that a new form of program analysis technique can to draw inferences about the properties of software systems in cases be created by incorporating characteristics of experimentation into in which more traditional analyses have not succeeded. analyses. This paper reports the results of research exploring this As experimentalists, we do indeed recognize many characteris- suggestion. Building on principles and methodologies underly- tics of scientific experimentation that already are, or could poten- ing the use of experimentation in other fields, we provide descrip- tially be, utilized by program analysis techniques. These charac- tive and operational definitions of experimental program analysis, teristics include the formulation and testing of hypotheses, the it- illustrate them by example, and describe several differences be- erative process of exploring and adjusting these hypotheses in re- tween experimental program analysis and experimentation in other sponse to findings, the use of sampling to cost-effectively explore fields. We show how the use of an experimental program analy- effects relative to large populations in generalizable manners, the sis paradigm can help researchers identify limitations of analysis manipulation of independent variables to test effects on dependent techniques, improve existing experimental program analysis tech- variables while controlling other factors, the use of experiment de- niques, and create new experimental program analysis techniques. signs to facilitate the cost-effective study of interactions among multiple factors, and the use of statistical analyses to assess results. Anyone who has spent time debugging a program will recognize Categories and Subject Descriptors the relevance of several of the foregoing characteristics to that ac- D.2.5 [Software Engineering]: Testing and Debugging—debug- tivity. Debuggers routinely form hypotheses about the causes of ging aids, testing tools; F.3.2 [Logics and Meanings of Programs]: failures, conduct program runs (in which factors that might affect Semantics of Programming Languages—program analysis the run other than the effect being investigated are controlled) to confirm or reject these hypotheses, and based on the results of this “experiment”, draw conclusions or create new hypotheses about the General Terms cause of the fault. The “experimental” nature of this approach is re- Experimentation flected (in whole or in part) in existing program analysis techniques aimed at fault localization (e.g., [16, 22, 30, 32]). Keywords In this paper we argue that a class of program analysis approaches exists whose members are inherently experimental in nature. By experimental program analysis, program analysis, experimentation this, we mean that these techniques can be characterized in terms of guidelines and methodologies defined and practiced within the 1. INTRODUCTION long-established paradigm of scientific experimentation. Building Program analysis techniques analyze software systems to col- on this characterization, we present an operational definition of a lect, deduce, or infer specific information about those systems. The new paradigm for program analysis, experimental program analy- resulting information typically involves system properties such as sis, and we show how analysis techniques can be characterized in data dependencies, control dependencies, invariants, anomalous be- terms of this paradigm. havior, reliability, or conformance to specifications. This informa- We show how our formalization of the experimental program tion supports various software engineering activities such as test- analysis paradigm, together with our operational definition, could ing, fault localization, impact analysis, and program understanding. help researchers identify limitations of analysis techniques, im- prove existing program analysis techniques, and create new exper- imental program analysis techniques. We also discuss applications for experimental program analysis, and suggest how techniques fol- Permission to make digital or hard copies of all or part of this work for lowing this paradigm can approach program analysis problems in personal or classroom use is granted without fee provided that copies are manners that may not be possible by other types of techniques. As not made or distributed for profit or commercial advantage and that copies such, we believe that our formalization of the experimental pro- bear this notice and the full citation on the first page. To copy otherwise, to gram analysis paradigm will allow researchers to use it, to good republish, to post on servers or to redistribute to lists, requires prior specific effect, in various program analysis domains in which it has not pre- permission and/or a fee. ISSTA'06, July 17–20, 2006, Portland, Maine, USA. viously been considered. Copyright 2006 ACM 1­59593­263­1/06/0007 ...$5.00. The remainder of this paper proceeds as follows. Section 2 pro- (2) Choice of experiment design. Experiment design choice is vides an overview of experimentation and relevant concepts. Sec- concerned with structuring variables and data so that predictions tion 3 presents our definitions of experimental program analysis, can be appropriately evaluated with as much power and as little illustrates them by an example, and also describes several poten- cost as possible. The process begins with the investigator choos- tially exploitable differences between experimental program anal- ing, from the scales and ranges of the independent variables, spe- ysis and experimentation in other fields. Section 4 illustrates the cific levels of interest as treatments for the experiment. Next, the potential benefits of using experimental program analysis. Section investigator formalizes the predictions about the potential effects of 5 describes related work, and Section 6 concludes. the treatments on the dependent variable through a statement of hy- potheses. To reduce costs, the investigator must determine how to 2. BACKGROUND sample the population while maintaining the generalization power Empirical science is a mature field, and its methods are well- of the experiment's findings. The investigator then decides how discussed in various monographs (e.g., [4, 14, 19, 20, 23, 26, 27, to assign the selected treatments to the units in the sample to ef- 28]). Because experimental program analysis techniques draw from ficiently maximize the power of the experiment, while controlling empirical science, in this section we distill, from these monographs, the fixed variables and reducing the potential impact of the nuisance an overview of the empirical method. In addition, we highlight ma- variables, so that meaningful observations can be obtained. terial about experiments that is most relevant to the understanding (3) Performing the experiment. This activity requires the cod- of experimental program analysis. (Note that this overview is gen- ification and pursuit of specified procedures that properly gather eralized from the sources just cited. In practice, empirical science observations to test the target hypotheses. These procedures should takes on different forms in different domains. The generalization reduce the chance that the dependent variables will be affected by we present here, which we refer to as “traditional experimentation”, factors other than the independent variables. Thus, it is important suffices for our subsequent discussion.) that the investigator regularly monitor the implementation and exe- The initial step in any scientific endeavor in which a conjecture cution of the experiment procedures to reduce the chances of gen- is meant to be tested using a set of collected observations is the erating effects by such extraneous factors. recognition and statement of the problem. This activity involves (4) Analysis and interpretation of data. An initial data analysis formulating research questions that define the purpose and scope often includes the computation of measures of central tendency and of the experiment, identifying the phenomena of interest, and pos- dispersion that characterize the data, and might help investigators sibly forming conjectures regarding the outcome of the questions, identify anomalies worth revisiting. More formal analysis includes or limitations on those outcomes. As part of this step, the investiga- statistical significance assessments regarding the effect of the treat- tor also identifies the target population to be studied, and on which ments on the dependent variables. Such assessments provide mea- conclusions will be drawn. sures of confidence in the reliability and validity of the results and Depending on the outcome of this first step, as well as the con- help interpretation proceed in an objective and non-biased man- ditions under which the investigation