Improving the Testing and Testability of Software Product Lines Isis Cabral, Myra B. Cohen, and Gregg Rothermel {icabral,myra,grother}@cse.unl.edu Department of Computer Science,University of Nebraska-Lincoln Abstract. Software Product Line (SPL) engineering offers several ad- vantages in the development of families of software products. There is still a need, however, for better understanding of testability issues and for testing techniques that can operate cost-effectively on SPLs. In this paper we consider these testability issues and highlight some differences between optional versus alternative features. We then provide a graph based testing approach called the FIG Basis Path method that selects products and features for testing based on a feature dependency graph. We conduct a case study on several non-trivial SPLs and show that for these subjects, the FIG Basis Path method is as effective as testing all products, but tests no more than 24% of the products in the SPL. 1 Introduction Software product line (SPL) engineering has been shown to improve both the efficiency of the software development process and the quality of the software developed, by allowing engineers to systematically build families of products with well defined and managed sets of re-usable assets [4]. A large body of research on SPL engineering has focused on reuse of core program assets [4, 15, 17], refined feature modeling [8, 9, 23], and code generation techniques [2, 7]. There has also been research on testing software product lines [3,6, 10, 24]. Despite this prior work, there still remains a need to improve reuse during the software testing process. Kolb and Muthig [15] point out that testing has not made the same advances as other parts of the SPL lifecyle and remains a bot- tleneck in SPL development. Their work highlights issues related to testability of SPLs, where testability is viewed as the ease with which one can incorporate testing into the development cycle and increase reuse while retaining a high rate of fault detection. They comment that the primary strength of SPL develop- ment, variability, also has the greatest impact on reducing testability [15], due to the combinatorial explosion of feature combinations that occurs as variabil- ity increases [6, 17]. In other related work, Jaring et al. [13] point out that the testability of a product line can be viewed as a function of the binding time of variability, and that providing early binding can increase the ability to test prod- ucts early. McGregor [17] and Cohen et al. [6] have suggested ways to reduce the combinatorial space by sampling products for testing using combinatorial interaction testing [5]; this work does not address testability issues. While all of this work aims at the core problem of software product line testing, none of it specifically considers reuse by examining the feature model and analyzing testability at a finer grain. In this work we consider testing from this perspective. We drill down into the issue of variability and analyze different types of variability, e.g. optional features versus alternative choice features, as they relate to testability. We conjecture that while the alternative choice features have a negative impact on testability by increasing the number of products, optional choice features do not. We then propose a new black box approach for testing software product lines that attends to these issues. We hypothesize that our approach can reduce testing effort while retaining good fault detection in the presence of alternative features. Our approach, which we call the FIG Basis Path method, translates a feature model into a feature inclusion graph that is, in essence, a feature model depen- dence structure. We associate all features with sets of test cases and then walk this graph to generate a subset of independent paths (or products) that cover the graph for testing. This is analogous to the basis path approach for testing software applications [26] which finds a “possibly minimal” set of paths to cover all nodes in a program’s control flow graph. We report results of a case study on two software product lines. In both SPLs we can achieve the same fault detection results as we can testing all products. Further analysis shows that we can also use a grouped variant of the Basis Path algorithm to test subfamilies of the SPL as defined by the alternative features. 2 Background and Related Work There has been a lot of work on feature modeling of which we present only a small subset [1,2,14,19,21]. In a feature model, a product line can be represented by mandatory and variable features. The variable features may be optional or alternative choices. In their simplest form alternative choice features allow for an exclusive or relationship. We can also have cardinalities assigned that allow for 0...n or 1...n relationships where the first number is the lower bound and the second number is an upper bound. An exclusive or alternative feature is usually the default for alternative features in a model and is a 1...1 relationship. We present a small product line to help to explain these ideas and to illus- trate our techniques. The feature model for this product line (shown using the Orthogonal Variability Model (OVM)) [19] is seen in Figure 1 on the left (the right portion of this figure will be discussed later). This product line defines a family of 42 calculator programs. It has a core set of features, Exit and Clear and one optional feature, Backspace. Users can select one of three languages (English, Chinese or Spanish) used in the menus, titles and help. Finally the memory features include Memory Store and Recall. The mandatory features in our sample product line are Core and Language. Within each of these there is some variation. Store is an Or feature from Memory and it is required only if the Memory Recall feature is selected. Feature models have been used for generative programming [2,7,23], provid- ing a model based approach to the realization of product lines. Feature models Core Mandatory Features Backspace Language <1..1> Chinese English Spanish Memory Recall Store <1..2> M+ MS Fig. 1. Calculator SPL Feature Model have also been used to model the product space for instantiating products for testing [3,6,24]; for instance, the work of Uzuncaova et al. [24] transforms the fea- ture model into an Alloy specification and uses this to generate test cases, while the work of Cohen et al. [6] uses the feature model to define samples of prod- ucts that should be included in testing. Similarly, the PLUTO methodology [3] uses the feature model to develop extended use cases that contain variability which can formulate a full set of test cases for the family of products. Sch¨urr et al. [22] use a classification tree method for testing from the feature model. Other extensions of feature models have been for staged generation [8] or mod- eling constraints [9]. None of this work explicitly uses the feature model as we do, in a graph based representation, that can be used to select products (and test cases) for testing. The work of Bachmeyer et al. [1] also uses a graph based representation of a feature model, but they do not use this in the testing process. Other work on software testing product lines includes that of Denger et al. [10] who present an empirical study to evaluate the difficulty of detecting faults in the common versus variable portions of an SPL code base concluding that the types of faults found in these two portions of the code differ. They use both white and black box techniques but do not test from the feature model. 3 Leveraging Redundancy for Testing via Feature Models We begin with the conjecture that black box testing of software product lines can be made more efficient and effective by designing the product line architecture (and resulting feature model) in a manner that supports reuse of product line testing results across different products. Others have argued that variability decreases testability [15], but we believe that there should be a finer grained examination of this argument. Both optional and alternative features can be viewed as points of variability in a software product line, yet we believe they may behave differently from a black box testing approach and provide different opportunities to reduce testing effort, as we explain next. Our methodology involves the following steps: (1) transform the feature model into a feature inclusion graph; (2) associate use cases with each feature; (3) develop test cases for each use case; (4) Select basis paths on this graph; and (5) for each path (product), run all test cases for the included features. 3.1 Feature Inclusion Graph In this section we present a transformation of the feature model into a graph that we call a feature inclusion graph (FIG), which represents feature dependencies derived from the feature model. In a FIG, all features that appear on a non- branching path are included in the same product, while branches represent the variability in feature composition. We view the FIG as having a loose connection to the control flow graphs used in software testing; a control flow graph shows explicit flow of control in a program and can be used to select test cases for white box testing. Harrold [12] has suggested that regression testing techniques can be applied to different abstractions of software artifacts as long as they can be represented as a graph and tests can be associated with edges. In our scenario we do not have control flow; rather, our paths represent a combination of features and its dependencies, but we use a common method from control flow based testing to find a basis path set [26] for the graph – a set of independent paths through the program graph.