ObjectĆOriented Perspective on Software System Testing in a Distributed Environment A flexible object-oriented test system was developed to deal with the testing challenges imposed by software systems that run in distributed client/server environments. by Mark C. Campbell, David K. Hinds, Ana V. Kapetanakis, David S. Levin, Stephen J. McFarland, David J. Miller, and J. Scott Southworth In recent years software technology has evolved fromexecution, singleĆ results gathering, and report generation (see machine applications to multimachine applications (theFig. 1). Unfortunately, the test harnesses created in these realm of the client and server). Also, objectĆorientedenvironments proĆ were not easily reusable, and when the next gramming techniques have been gaining ground onproject proceĆ reached the test planning stage, the test harness had dural programming languages and practices. Recently,to test be reworked. engineers have focused on techniques for testing objects. The advent of standardized test environments such as TET However, the design and implementation of the test tools (Test Environment Toolkit)* helped to reduce this costly reĆ and code have remained largely procedural in nature. tooling by providing a standard API (application program This paper will describe the object testing framework,interface) which and tool base that test developers can adopt and is a software testing framework designed to meetuse theto testing write standardized tests. However, the difficulty is to needs of new software technologies and take advantageprovide of a standard test harness that is complete but flexible objectĆoriented techniques to maximize flexibility andenough tool to keep pace with changing software technology reuse. and remain viable for the long term. During the development and testing of the initial release of System Software Testing HP ORB Plus, which is an object request broker based on The levels of software testing include unit, integration, and the Object Management Group's CORBA specification (see system testing. Unit testing involves testing individual system page 76), we realized that distributed object technology modules by themselves, integration testing involves testing posed testing problems that were not adequately solved by the individual modules working together, and system testing any of the test harnesses currently available. We needed a involves testing the whole product in its actual or simulated flexible test environment that could handle heterogeneous operating environment. This paper focuses on software system testing. * The Test Environment Toolkit (TET) specification began in September 1989 as a joint proposal A software system test is intended to determine whetherby the Open Software the Foundation, UNIX International, and X/Open . software product is ready to ship by observing how the product performs over time while attempting to simulate its real use. System testing is composed of functional, perforĆ mance, and stress tests. It also covers operational, installaĆ Tester tion, and usability aspects of the product and may include destructive and concurrence testing. The product may supĆ port many different hardware and software configurations User Interface Command Line or GUI which all require testing. All of these aspects are combined Configuration Management, Test Harness Test Control, to assess the product's overall reliability. Software systemCreated Report Generation, etc. testing is usually done when all of the individual softwareby Test product components are completed and assembled intoDevelopers the Test Test Test Test Case Case Case Case Test Scripts, final product. 1 2 3 N Test Cases, etc. In the past, system testing environments centered around testing procedural, nondistributed software. These environĆ ments, which were also procedural and nondistributed, were System Under Test (SUT) usually developed by the test writer on an ad hoc basis along with the test code for the product. Recently, software system testing has benefited from the use of highlyFig. autoĆ1. A typical automated test environment. mated test harnesses and environments that simplify test December 1995 HewlettĆPackard Journal75 distributed systems communicating over multiple transportsWhen test clients execute, they are instructed to run for a using multithreaded clients and servers.ever, How we were specified amount of time. They report failure and status inĆ not willing to loseinvestment the we made in the testformation code back to a central location. Upon completion of the and tools developed for our earlier products. system test, clients and servers are stopped, temporary files are removed, and final summary reports are produced. Instead of abandoning the old test environment and replacĆ ing it with an entirely new system, we decided to use the The Test System objectĆoriented principles of encapsulation and polymorĆ To manage all the activities of distributed system testing, we phism to evolve our current environment base to meet our developed a test infrastructure that met our current needs needs without throwing out the old code. The ability to and could evolve with new technologies and new needs. We change or replace functional blocks of a system without afĆ followed a modular, objectĆoriented design approach to fecting the entire environment is one of the main benefits of accomplish this. objectĆoriented design (see ObjectĆOriented Programming" on page 79). ObjectĆoriented principles allowed usWe to reuse first engaged in several brainstorming sessions to proĆ existing tools. duce a list of requirements for a complete distributed testing framework. This was an attempt to pinpoint all the attributes Distributed System Testing and functionality that a perfect" test infrastructure would In a distributed object system, service providers arehave, distribĆ and it was done in the context of system testing disĆ uted across a network of systems and are calledtributed servers. objects. The needs of product developers, test deĆ Clients, who issue requests for those services, arevelopers, also disĆ and testers were considered, as well as the need to tributed across a network. A single program can bereport both metrics a to the project team. The main focus, however, client (issues requests) and a server (services requests).was CliĆon the two groups who would use the test framework ents can issue requests to local and remoteDuring servers. a the most: test developers and testers. Often these are the distributed object system test, clients are responsiblesame for people, but the distinction was made to clearly differĆ reporting any failures or status resulting from theentiate requests the needs of each group. they make. Product developers normally want quick and simple tests to The first task performed during the system testingverify of a that disĆ their code behaves correctly and at the same time tributed object software product is test setup. Clientshave and their programs work as they would for an end user. servers must be deployed across the network to targetedThey don't want to be distracted by the infrastructure. ExistĆ systems. Consideration must also be given to the facting test that APIs tend to be intrusive, requiring developers to servers may have multiple clients sending messageshave to them, knowledge of the test environment in which their tests and the distribution of clients and servers may changewill be durĆ run. Therefore, we wanted our new test framework ing a system test so that various hardware and softwareto minimize conĆ intrusiveness. This would allow developers to figurations can be tested. focus on testing the proper behavior of their code and not on the test infrastructure. Ideally, product developers should be able to write their tests with minimum restrictions, and the tests should plug and play in many different testing The Object Management Group's situations. Distributed Object Model Test developers, whose job it is to develop ways to test the product, have many of the same needs as product developĆ The Object Management Group (OMG) creates standards for the interoperability ers but are more concerned with blackĆbox testing and tryĆ and portability of distributed object-oriented applications. The OMG only produces ing to break" the product rather than verifying correct beĆ specifications, not software. Each participating vendor provides an implementation havior. To do this, test developers want to be able to plug to meet the specification. The Common Object Request Broker Architecture (CORBA) specification defines a flexible framework upon which distributed object- new tests into the test environment easily and quickly, and oriented applications can be built. This architecture represents the Object Request they want process and environment control. This would Broker (ORB) technology that was adopted by the OMG. An ORB provides the allow them to use the same tests in different scenarios to mechanisms by which distributed objects transparently make requests and receive find more product defects. Test developers are usually the responses. The ORB enables object-oriented applications on different machines to ones responsible for supporting the testing infrastructure. communicate and interoperate. Thus, more than any of the other groups, test developers The OMG has defined an Object Management Architecture object model. In this need a framework that is extensible, reusable, flexible, and model, objects provide services, and clients issue requests for those services. The controlled, and hopefully