SOFTWARE TESTING, VERIFICATION AND RELIABILITY, VOL. 6, 125-252 (1996) Testing Object=Oriented Software: a Survey ROBERT V. BINDER RBSC Corporation, 3 First National Plaza, Suire 1400, Chicago, IL 60602-4205, U.S.A. SUMMARY Research and practitioner literature on testing object-oriented software published up to the end of 1994 is summarized. The contribution of each source to eight topics is presented: (1) abstract data type verification and testing as it relates to object-oriented testing; (2) testing theory-fault hypotheses for object-oriented software and adequate testing (several fault taxo- nomies are presented); (3) automatic model validation-techniques and tools for testing execut- able object-oriented representations; (4) test case design-heuristic and formal techniques to develop test cases from object-oriented representations and implementations; (5) testability- factors in controllability and observability; (6)test automation-assertions, state manipulation, comparators, object identity and built-in tests; (7) test process-strategies to organize and manage the activity of testing object-oriented implementations; and (8) experience reports. Appendices provide several cross-references. KEY WORDS abstract data types; built-in test; Ctt; CLOS; design for testability; Eiffel; fault taxonomy; formal methods; formal verification; integration testing; model validation; object-oriented analysis; object-oriented design; object-oriented testing; object-oriented programming; Objective-C; oracles, regression testing; Smalltalk; software process; test coverage; lest adequacy; test case generation; test automation; unit testing 1. INTRODUCTION This paper is a survey of research and practitioner work on testing object-oriented software published up to the end of December 1994. A concerted effort has been made to locate all available sources. It is a certainty that literature in this area will grow. The author welcomes additional citations. Readers are encouraged to study the primary sources, since many details are necessarily not reported here. Other less extensive surveys are available (Binder, 1994a and 1994~;Hayes, 1994; Overbeck, 1993 and 1994a; Turner and Robson, 1993~);however, all sources discussed in them are included here. No attempt is made to cover the extensive literature on related software testing issues. Several general texts on software testing are available, including those of Myers (1979), Howden (1987) and Beizer (1990). An early survey of verification, validation and testing techniques is that by Adrion et al. (1982). This introduction considers basic issues, the context of testing object-oriented software, and provides an overview of the survey. CCC 0960-0833/96/030 125-1 28 Received 15 June 1995 0 1996 by John Wiley & Sons, Ltd. Revised 3 June 1996 126 ROBERT V. BINDER 1.1. Perceptions of testing in object-oriented development Software testing is necessary to produce highly reliable systems, since static verification techniques cannot detect all software faults. While only a small proportion of developers practice adequate testing (Hetzel and Gelperin, 199 1), few conventional developers would contend testing is irrelevant. Current perspectives on object-oriented testing can be charac- terized as optimistic, minimalist or expansive. The optimistic view is that the technology and software processes typically associated with object-oriented development obviate or greatly reduce the need for testing. Iterative and incremental development using a ‘robust’ class library is seen as a process not unlike writing source code with an incremental compiler. Once a new class is syntactically correct and has been smoothly integrated (to the developer’s satisfaction), development is done. Subsequent changes are viewed as refinements to be conducted in the same manner. Formal testing (the development and systematic execution of a test plan and test cases) is not necessary. After recommending ‘. each class must be thoroughly tested’, Taylor (1992) speaks to the optimist. ‘To many object programmers, this degree of analysis and testing may seem like a major intrusion on what should be a quick, intuitive process of class definition.’ Many popular object-oriented methodologists do not discuss testing at all (Coad and Yourdon, 1990; Shlaer and Mellor, 1992; Embley et al., 1992; Martin and Odell, 1992; Coad and Nicola, 1993). The minimalist view is that testing remains necessary for object-oriented implemen- tations, but it will be easier and less costly. Booch (1991, p. 212) argues ‘. the use of object-oriented design doesn’t change any basic testing principles; what does change is the granularity of the units tested’. A text on advanced C++ offers a single sentence on testing which pronounces, ‘Black-box testing proceeds as for any system.’ (Coplien, 1992, p. 209). The observation of Rumbaugh et ul. (1991) characterizes this point of view: ‘Both testing and maintenance are simplified by an object-oriented approach, but the traditional methods used in these phases are not significantly altered. However, an object-oriented approach produces a clean, well-understood design that is easier to test, maintain, and extend than a non-object-oriented design because the object classes provide a natural unit of modularity.’ (Rumbaugh er nl., 1991, p. 144). A similar perspective is offered by Wirfs-Brock et al. (1990) ‘. object-oriented design means that the entities of the system can be isolated and tested one at a time. Entities can be shown to function before being plugged into the system. places where a responsibility was omitted in the design or made part of the wrong entity can be spotted and filled.’ (Wirfs-Brock et al., 1990, p. 11). The expansive point of view is that typical features of object-oriented systems require new approaches to attain adequate testing, beneficial effects of class modularity notwith- standing. New design strategies and processes for testing under a formal testing regime are needed. Siege1 (1992) notes that ‘Many people are suggesting that testing needs and costs will be lower for OOD systems. In reality, this probably will not be true.’ In the first careful analysis of object-oriented testing, Perry and Kaiser (1990) report: ‘. , . we have uncovered a flaw in the general wisdom about object-oriented languages-that “proven” (that is well-understood, well-tested, and well-used) classes can be reused as superclasses without retesting the inherited code. [Testing] may still turn out to be easier . but there are certain pitfalls that must be avoided.’ TESTING OBJECT-ORIENTED SOFTWARE 127 In addition to inheritance and polymorphism, Smith and Robson (1990) argue that other typical features of object-oriented languages are complex and therefore error-prone. With large class libraries, it may be difficult for a developer to comprehend the intended usage. Activation by message passing is argued to be significantly different from conventional activation; data flow involves objects instead of primitive types. Genericity and abstract super classes present similar problems. It may be possible to instantiate a generic class with an inappropriate type. If a generic class is changed, then all instances of it should be retested. Testing for reusability should be performed to certify reusability. While ‘object oriented software validation can be based on conventional approaches, [this] is not enough. There remains a large field to be studied.’ In particular, consistency in the abstraction hierarchy resulting from inheritance and correct behaviour of concurrently executing objects are identified as significant problems (Tamai, 1991). Object-oriented development will experience ‘different types and proportions of errors that require a different approach to testing’ compared to conventional development methodologies and languages (Firesmith, 1992 and 1993b). Encapsulation reduces but does not eliminate the hazard of global data faults (intra-class access can cause the same problems). Path testing is limited to methods and must consider exception handling and concurrency. Since methods are typically small, the path selection problem is often trivial. Boundary value analysis is ‘of limited use’ owing to the beneficial effects of strong typing and ‘proper’ data abstraction. Mesagelparameter equivalence classes are relevant instead of variable equivalence classes. Encapsulation can pose an obstacle to testing (Berard, 1993). Since the state of an object can only be reported by a method on the object, a partially tested method may have to be relied upon for reporting the state. Parametrized classes, specialization and name-clashes under multiple inheritance pose unique testing problems. 1.2. Issues in testing object-oriented software Although much of what is known about testing conventional systems applies, object- oriented development presents unique testing problems which require new solutions. While the sources surveyed here do not always agree on approach, nearly all agree that there are significant differences. Four main concerns are evident. (1) Fault hypothesis. Are some facets of object-oriented software more likely than others to contain faults? (a) What kind of faults are likely? (b) How can revealing test cases be constructed? (c) What effect does the scope of a test (method, class, cluster, subsystem, system) have on the chance of revealing a fault? (d) To what extent are fault models specific to applications, programming langu- ages and development environments? What is general and what is not? For example, are strategies