ObjectĆOriented Perspective on Software System Testing in a Distributed Environment
A flexible object-oriented test system was developed to deal with the testing challenges imposed by software systems that run in distributed client/server environments. by Mark C. Campbell, David K. Hinds, Ana V. Kapetanakis, David S. Levin, Stephen J. McFarland, David J. Miller, and J. Scott Southworth
In recent years software technology has evolved from execution, singleĆ results gathering, and report generation (see machine applications to multimachine applications (theFig. 1). Unfortunately, the test harnesses created in these realm of the client and server). Also, objectĆoriented environments proĆ were not easily reusable, and when the next gramming techniques have been gaining ground on project proceĆ reached the test planning stage, the test harness had dural programming languages and practices. Recently, to test be reworked. engineers have focused on techniques for testing objects. The advent of standardized test environments such as TET However, the design and implementation of the test tools (Test Environment Toolkit)* helped to reduce this costly reĆ and code have remained largely procedural in nature. tooling by providing a standard API (application program This paper will describe the object testing framework, interface) which and tool base that test developers can adopt and is a software testing framework designed to meet use the to testing write standardized tests. However, the difficulty is to needs of new software technologies and take advantage provide of a standard test harness that is complete but flexible objectĆoriented techniques to maximize flexibility and enough tool to keep pace with changing software technology reuse. and remain viable for the long term. During the development and testing of the initial release of System Software Testing HP ORB Plus, which is an object request broker based on The levels of software testing include unit, integration, and the Object Management Group's CORBA specification (see system testing. Unit testing involves testing individual system page 76), we realized that distributed object technology modules by themselves, integration testing involves testing posed testing problems that were not adequately solved by the individual modules working together, and system testing any of the test harnesses currently available. We needed a involves testing the whole product in its actual or simulated flexible test environment that could handle heterogeneous operating environment. This paper focuses on software system testing. * The Test Environment Toolkit (TET) specification began in September 1989 as a joint proposal A software system test is intended to determine whether by the Open Software the Foundation, UNIX International, and X/Open . software product is ready to ship by observing how the product performs over time while attempting to simulate its real use. System testing is composed of functional, perforĆ mance, and stress tests. It also covers operational, installaĆ Tester tion, and usability aspects of the product and may include destructive and concurrence testing. The product may supĆ port many different hardware and software configurations User Interface Command Line or GUI which all require testing. All of these aspects are combined Configuration Management, Test Harness Test Control, to assess the product's overall reliability. Software systemCreated Report Generation, etc. testing is usually done when all of the individual softwareby Test product components are completed and assembled into Developers the Test Test Test Test Case Case Case Case Test Scripts, final product. 1 2 3 N Test Cases, etc. In the past, system testing environments centered around testing procedural, nondistributed software. These environĆ ments, which were also procedural and nondistributed, were System Under Test (SUT) usually developed by the test writer on an ad hoc basis along with the test code for the product. Recently, software system testing has benefited from the use of highly Fig. autoĆ1. A typical automated test environment. mated test harnesses and environments that simplify test
December 1995 HewlettĆPackard Journal 75 distributed systems communicating over multiple transportsWhen test clients execute, they are instructed to run for a using multithreaded clients and servers. ever, How we were specified amount of time. They report failure and status inĆ not willing to lose investment the we made in the test formation code back to a central location. Upon completion of the and tools developed for our earlier products. system test, clients and servers are stopped, temporary files are removed, and final summary reports are produced. Instead of abandoning the old test environment and replacĆ ing it with an entirely new system, we decided to use the The Test System objectĆoriented principles of encapsulation and polymorĆ To manage all the activities of distributed system testing, we phism to evolve our current environment base to meet our developed a test infrastructure that met our current needs needs without throwing out the old code. The ability to and could evolve with new technologies and new needs. We change or replace functional blocks of a system without afĆ followed a modular, objectĆoriented design approach to fecting the entire environment is one of the main benefits of accomplish this. objectĆoriented design (see ObjectĆOriented Programming" on page 79). ObjectĆoriented principles allowed us We to reuse first engaged in several brainstorming sessions to proĆ existing tools. duce a list of requirements for a complete distributed testing framework. This was an attempt to pinpoint all the attributes Distributed System Testing and functionality that a perfect" test infrastructure would In a distributed object system, service providers are have, distribĆ and it was done in the context of system testing disĆ uted across a network of systems and are called tributed servers. objects. The needs of product developers, test deĆ Clients, who issue requests for those services, are velopers, also disĆ and testers were considered, as well as the need to tributed across a network. A single program can be report both metrics a to the project team. The main focus, however, client (issues requests) and a server (services requests). was CliĆon the two groups who would use the test framework ents can issue requests to local and remote During servers. a the most: test developers and testers. Often these are the distributed object system test, clients are responsible same for people, but the distinction was made to clearly differĆ reporting any failures or status resulting from the entiate requests the needs of each group. they make. Product developers normally want quick and simple tests to The first task performed during the system testing verify of a that disĆ their code behaves correctly and at the same time tributed object software product is test setup. Clients have and their programs work as they would for an end user. servers must be deployed across the network to targetedThey don't want to be distracted by the infrastructure. ExistĆ systems. Consideration must also be given to the fact ing test that APIs tend to be intrusive, requiring developers to servers may have multiple clients sending messages have to them, knowledge of the test environment in which their tests and the distribution of clients and servers may change will be durĆ run. Therefore, we wanted our new test framework ing a system test so that various hardware and software to minimize conĆ intrusiveness. This would allow developers to figurations can be tested. focus on testing the proper behavior of their code and not on the test infrastructure. Ideally, product developers should be able to write their tests with minimum restrictions, and the tests should plug and play in many different testing The Object Management Group's situations. Distributed Object Model Test developers, whose job it is to develop ways to test the product, have many of the same needs as product developĆ The Object Management Group (OMG) creates standards for the interoperability ers but are more concerned with blackĆbox testing and tryĆ and portability of distributed object-oriented applications. The OMG only produces ing to break" the product rather than verifying correct beĆ specifications, not software. Each participating vendor provides an implementation havior. To do this, test developers want to be able to plug to meet the specification. The Common Object Request Broker Architecture (CORBA) specification defines a flexible framework upon which distributed object- new tests into the test environment easily and quickly, and oriented applications can be built. This architecture represents the Object Request they want process and environment control. This would Broker (ORB) technology that was adopted by the OMG. An ORB provides the allow them to use the same tests in different scenarios to mechanisms by which distributed objects transparently make requests and receive find more product defects. Test developers are usually the responses. The ORB enables object-oriented applications on different machines to ones responsible for supporting the testing infrastructure. communicate and interoperate. Thus, more than any of the other groups, test developers The OMG has defined an Object Management Architecture object model. In this need a framework that is extensible, reusable, flexible, and model, objects provide services, and clients issue requests for those services. The controlled, and hopefully has a long lifetime. If a testing ORB facilitates this model by delivering requests to objects and returning any infrastructure becomes out of date, test developers will have output values to the client. The services that the ORB provides are transparent to to repair or replace it. the client. Often test developers are the ones who perform system testĆ To request a service, a client needs the object reference for the object that is to ing, but many times this role is handed off to testers. Although provide the service. The ORB uses this object reference to identify and locate the the needs of both groups clearly overlap, testers need a testĆ object. The ORB activates the object if it is not already executing and directs the ing infrastructure that is easy to use for the installation, conĆ request to the object. figuration, and execution of tests. In many of our past projĆ ects, testing was done by temporary personnel. This freed
76 December 1995 HewlettĆPackard Journal test developers to write more tests and assist product In the develĆ following design discussion, the term object can mean opers in debugging. When the test infrastructure is class easy or to an instance of a class. It should be clear from the use, the testing role can be handed off to testers context earlier of in the discussion which is meant. the testing process. Additionally, the ability to reconfigure the test environment easily and quickly allows more OTF scenarĆ Management System ios to be tested. This increases the likelihood more of finding The OTF management system consists of the six major product defects, which leads to a better product. quality classes: user interface, OTF controller, test suite configuraĆ tion, test controller, report generator, and database controlĆ Finally, test results are usually provided to the project team ler. This system provides the user interface that the software in the form of metrics. Gathering metrics in a distributed tester interacts with. Through this interface the tester speciĆ environment can be timeĆconsuming. Data can be located on fies test configurations such as which client and server proĆ multiple systems on the network. However, when dealing grams will be running on which SUTs. The OTF manageĆ with multiple processes running in parallel on different sysĆ ment system takes the specified configurations and makes tems, results may not always occur in a consistent order. them available to each of the SUTs, ensures that the SUTs This implies the need for a centralized repository for testing run the specified tests, logs test data and results, and generĆ results. This would make the generation of metrics much ates test reports. easier and faster, while providing a central location for finding problems and debugging. The main class in the OTF management system is the OTF controller, which serves as the delegator object. It takes reĆ Design Methodology quests from the user interface object and manages the activiĆ Taking into consideration the needs of the different ties groups of the test suite configuration, test controller, and report mentioned above, we decided that the following attributesgenerator objects. The test suite configuration object is actuĆ were required for our test infrastructure. ally created by the OTF controller. For a new configuration • Extensibility. Ensure the evolution of a modular system the object that will initialize from the configuration data provided can be dealt with on a componentĆbyĆcomponent basis.by the user interface. For a previously specified configuraĆ • Reusability. Allow object and code reuse for both tion, tests the and object will initialize from the database. After this the test infrastructure. object's configuration data has been set, its primary responsiĆ • Flexibility. Provide a plugĆandĆplay environment that bility allows is to respond to configuration queries from the SUTs. for flexibility in test writing and configuration. The test controller has the overall responsibility for coordiĆ • Simulation. Provide the ability to simulate customer environĆ nating the running of tests on the SUTs. It provides the SUTs ments. with a pointer to the test suite configuration object, synchroĆ • Control. Provide centralized control of the test processes nizes the starting of tests, and passes status data and reĆ and environment. quests back to the OTF controller. It also has the capability • Nonintrusive. Hide as much of the testing infrastructure as to log status data to the database via the database controller. possible from the system under test. • Ease of use. Provide ease of use for installation, The setup, report conĆ generator, upon a request from the OTF controlĆ figuration, execution, results gathering , and test distribution.ler, queries the database controller to assemble, filter, and format test data into userĆspecified test reports. Raw test data With these attributes in mind, we set about deciding on the is put into the database by each TestEnvironment SUT's object, basic set of classes that would be needed. We used a while test process status data is put into the database by the method for objectĆoriented design called Object Modeling test controller as mentioned above. Technique (OMT)1 to develop a diagram showing class relaĆ tionships (see ObjectĆOriented Programming" on page 79). System under Test We walked through several scenarios and expanded Each and system under test (SUT) contains fifteen classes. In refined our set of classes. Once we had an initial normal design operation, we a SUT retrieves configuration data from wrote CRC (class, responsibility, and collaboration)2 cards the OTF management system, and then, based on that data, for each of the classes in our design. (CRC cards retrieves are also the specified tests from the management system. described on page 79.) This design was reviewed Since by the the SUT has the capability to build test executables product development team and their feedback was from incorpoĆ source code, it can retrieve test source code and exeĆ rated. cutables from the OTF management system. Once the test executables are in place and any specified test setup has The Object Testing Framework been completed, the SUT waits for a management system The design process produced an objectĆoriented softwarerequest to start the tests. When this happens, the SUT is reĆ testing system that we named the object testing frameworksponsible for running the tests, logging status, test data, and (OTF). Although this design is intended to test distributedresults, and cleaning up upon test completion. objectĆbased software, it can also be used to test distributed, The main object in the SUT is the host daemon, which is the procedurally based client/server software. The OTF consists SUT's delegator object. The host daemon takes requests from of the classes shown in Fig. 2. The architecture of the OTF is and forwards requests to the OTF management system and such that there is a single master test control system (OTF manages the activities of the setter upper, test executor, management system in Fig. 2) that orchestrates running tests cleaner upper, process controller, TestEnvironment and objects. on multiple systems under test. This master system can also be a system under test.
December 1995 HewlettĆPackard Journal 77 OTF Management System User Interface
OTF Controller
Test Suite Test Report Database Configuration Controller Generator Controller Data- base
System under Test Host Daemon
Test Management Subsystem Process Management Subsystem
Setter Test Cleaner Process Upper Executor Upper Controller TestEnvironment
Test TestCase Distributor
Factory Client Server Builder Checker TestCase TestCase TestCase
User Written Test MyFactory MyClient MyServer Test Case Test Case Test Case
Fig. 2. = Multiple Associations System architecture for the = Subclasses or Inheritances object testing framework.
The overall responsibility of the setter upper, test could executor, be remote copied from the management system to the and cleaner upper objects is to manage how the SUT. tests An are important design consideration was to have a single run. These three objects collaborate with the builder, repository test for tests. This makes it easy to control changes to distributor, checker, and factory TestCase objects to form thetests and is not intrusive on the SUTs. test management subsystem shown in Fig. 2. The process The checker provides the ability to customize test setup by controller and TestEnvironment objects provide the infrastrucĆ invoking a userĆwritten program that can ensure that eleĆ ture for connecting the tests to the framework. These two ments outside of the test environment are set up correctly. objects collaborate with TestCase the objects to form the For example, it could check that NFS and DCE are running, process management subsystem. that the display is set correctly, and so on. Test Management Subsystem The factory TestCase provides the setup procedures that arise This subsystem sets up and executes the tests and when then testing a CORBAĆbased distributed object system. It cleans up after the tests have completed. The setter creates upper the is CORBA objects that reside in the CORBAĆbased the object that controls test setup. It is a lowĆlevel server delegatorTestCases and stores references to these objects for use that manages the activities of the builder, test distributor,by the client TestCases. The factory TestCase class inherits from checker, and factory TestCase objects. The test distributor the is TestCase base class and the test developer writes a class responsible for retrieving test executables and sources that from inherits from the factory TestCase class. This allows the the OTF management system. When it retrieves source test code,developer to customize the TestCase factory functionality the builder is responsible for generating test executablesfor a specific test. from the code. How the tests are retrieved depends on the The test executor object starts the TestCase client s through overall system environment and resources available. A disĆ the functionality inherited from TestCase the base class. It also tributed file system, like NFS, could be used, or the tests
78 December 1995 HewlettĆPackard Journal OTF API is made available through the pointer TestEn- to the ObjectĆOriented Programming vironment class provided by the TestCase base class.
Object-oriented programming is a set of techniques for creating objects and as- Implementation Approach sembling them into a system that provides a desired functionality. An object is a Once the design was complete, an initial investigation was software model composed of data and operations. An object’s operations are used made to find an existing system that matched the characterisĆ to manipulate its data. Objects may model things such as queues, stacks, win- tics of the design. When no system was found, an analysis dows, or circuit components. When assembled into a system, the objects function was done to determine the cost of implementing the new as delegators, devices, or models. Messages are passed between the objects to infrastructure. produce the desired results. It quickly became obvious that the transition to the new The eventual goal of object-oriented programming is to reduce the cost of software development by creating reusable and maintainable code. This is achieved by infrastructure would have to be gradual since we did not using three features of object-oriented programming: encapsulation, polymor- want to impact the HP ORB Plus product release cycle. The phism, and inheritance. Encapsulation consists of data abstraction and data hid- flexibility provided by an objectĆoriented system enables ing. Data abstraction is a process of isolating attributes or qualities of something gradual migration and evolution through encapsulation, into an object model. With data hiding an object may contain its own private data inheritance, and polymorphism. Tests could be isolated from and methods, which it uses to perform its operations. By using polymorphism, an the infrastructure so that new tests could be developed and object’s operation may respond to many different types of data (e.g., graphical and evolved without modification as the infrastructure evolved. textual). Finally, using inheritance, an object can inherit attributes from other ob- This flexibility fit nicely with the realization that the time to jects. The object may then only need to add the attributes, methods, and data replace the existing infrastructure exceeded an average prodĆ structures that make it different from the object from which it inherited its basic uct life cycle. characteristics. ObjectĆoriented encapsulation provided another advantage. For the design of the object testing framework described in the accompanying article, we used an object-oriented software design methodology called object Once some basic changes were made to the existing test modeling technique (OMT). This methodology provides a collection of techniques infrastructure and tests had been converted to the new and notation for designing an object-oriented application. objectĆoriented programming model, the existing test infraĆ structure could be used to simulate some aspects of the new One important aspect of object-oriented design, or any software design, is decid- infrastructure. This allowed our system testing efforts to ing on who (i.e., module or object) is responsible for doing what. A technique provided in OMT involves using an index card to represent object classes. These benefit immediately from the features of the new test cards are called CRC (class, responsibility, and collaboration) cards. The informa- system. tion on one of these cards includes the name of the class being described, a The development of the current version of the object testing description of the problem the class is supposed to solve, and a list of other framework has taken place in two steps, which have spanned classes that provide services needed by the class being defined. three releases of the HP ORB Plus product. At each step we have continued to apply the same design principles. This work is summarized in the following sections. reports back to the host daemon the success or failure of a test start. First Step. For the first step, the goal was to consolidate the best practices of three existing test infrastructures into a The cleaner upper cleans up after the tests have single completed. infrastructure that simulated as much of the major This may include removing temporary files, removing functionality test of the OTF as possible. So as not to impact the executables, and so on. ongoing HP ORB Plus software releases, another goal was to minimize changes to existing test code. This resulted in an Process Management Subsystem infrastructure that consisted of a layer of shell scripts on top The two main objects in the process management of subsystem two existing test harness tools. This significantly reduced are the process controller TestEnvironment and objects. The the effort needed to set up, administer, and update the netĆ process controller has the overall responsibility to monitorwork of systems that were used to system test the HP ORB all testĆrelated processes on the SUT. It can register Plus or product, unĆ while the tests continued to use existing APIs. register processes, kill processes, and report process It status also confirmed that our design was indeed trying to solve back to the host daemon. the right problems. The TestEnvironment class provides the test developer with Second an Step. For this step the goal was to deploy the test application programming interface to the OTF. It providesdeveloper's API to the OTF. The result was the implementaĆ methods for aborting tests, logging test data and results,tion of the TestEnvironment C++ and TestCase classes described checking for exceptions, getting ronment envi variables, and above. so on. The test developer gets access to these methods through the base TestCase class, which has an association Additional with classes were designed to connect TestEnviron- the the TestEnvironment class. ment and TestCase classes to the existing infrastructure, but their existence is hidden from the test developer. This proĆ Creating a test involves writing a class that inherits vides from a stable API without limiting future enhancements to either the client TestCase or server TestCase base classes. Thethe infrastructure. Once the new infrastructure was deployed, initialization and setup functionality for the test would we be focused on porting existing tests to it. included in the test's constructor. The cleanup required when the test is done is included in the destructor. This Finally, framework has resulted in minimal changes to existing an implementation for the inherited run_body() method is tests and maximum increase in functionality for the tests. included, which is the test executable that runs the Most test. of The the work simply involved taking existing code and
December 1995 HewlettĆPackard Journal 79 wrapping it in the appropriate class. All of our tests 2. Fig. have 3b includes the class declaration and method definiĆ benefited from the features provided TestEnvironment by the tions of HelloWorldTest a class and a macro to register the defiĆ and TestCase classes and are insulated from changes nition to the with the TestEnvironment. The HelloWorldTest class is deĆ framework. rived from the TestCase base class. Fig. 3a source has no HelloWorldTest class. At this stage of its development the object testing framework allowed the removal of intrusive test code required 3. by The check_ev_and_ptr the macro in the Fig. 3a source is greatly old test APIs. For example, many tests included code simplified that in the Fig. 3b source, thanks TestEnvironment to the 's allowed a test to be reexecuted for a specific time print_exception period. and is_nil methods. That code was removed from the tests because the same 4. Fig. 3a has a main function, whereas in Fig. 3b, the main functionality is now provided by the framework. function is replaced by HelloWorldTest the constructor, destrucĆ In addition to supporting the design shown in Fig. tor, 2, and ourrun_body methods. This structure allows the OTF to current implementation provides the following functionality:instantiate and run the test code as needed. The constructor • Iteration. A test can be executed repetitively, either and by destructor speciĆ allow the test writer to separate out execute fying a number of iterations or the amount of time.once" code if desired. run_body The method may be executed • ContextĆsensitive execution. The object testing frameworkmore than once. behaves differently depending on how it is invoked. In the 5. Fig. 3a uses a file to store the object reference string developer's environment it is transparent and does not affect created by the server. (Use of files is potentially difficult if test behavior. In the testing environment it is bound to the the client and server are on different machines.) Fig. 3b uses test system. For example, in the developer environment, the the TestEnvironment's naming service to get the object referĆ C++ functions cerr and cout go to the terminal, but in the ence string. testing environment they go to a file and the test report jourĆ nal respectively. This encourages developers to put 6. all In existĆ Fig. argc 3b and argv are not available as input parameĆ ing tests into the framework because the test continues ters and to must be obtained from the TestEnvironment associated . work the way it did before it was ported. • Simple naming service. A naming service allows the 7. user The int to return parameter of the main function in Fig. 3a is associate a symbolic name with a particular value replaced such as by the TestEnvironment::Result of the run_body method the path name of a data file. In a distributed system, in Fig. it 3b. is The effect is the same, to return the success or necessary for multiple processes to share values that failure are of the invocation. obtained outside of the systemĊfor example, object Next Steps. The following is a list of items under considerĆ references. ation for implementing the rest of the design and adding • Automatic capture of standard output cout streams and cerr. more functionality to TestEnvironment the and TestCase classes. To simplify porting of existing tests, cout and the cerr streams• User interface class. We are investigating the possibility of are mapped to a file and the journal file respectively.encapsulating a graphical user interface that was designed • Encapsulation of functions from the product under for test. one For of the existing test infrastructures. example, the parts of CORBA used by the tests • are Test encapsuĆ controller class. Here again we are looking at encapsuĆ lated. As CORBA evolves and changes its C++ languagelating an existing test synchronization controller. bindings, only a single copy of the bindings in • the Memory frameĆ leak detection. By adding this feature Test- to the work has to change. Environment and TestCase classes, all tests will get this functionĆ • Inheritance and reuse. Inheritance allows the test case ality deĆ through inheritance. veloper to describe similar tests as a family of • test Integration cases deĆ with runĆtime debugging. This will improve rived from a common class (which in turn is derived tracing from and fault isolation in a distributed, multithreaded the TestCase class). In this case, polymorphism allows environment. test code to be reused in multiple tests, while allowing • Heterogeneous changes networks. The current object testing frameĆ to specific operations and data when needed. work handles networks of HPĆUX* systems only. We need to Example. Our experience with the current framework expand has the framework to handle other systems UNIX as shown that the time to port existing applications tests well to as the PC operating systems. new API is minimal. Fig. 3 shows an example of how test code would look before and after being ported to Summary the new test infrastructure. This example is an implementation The of object the testing framework is based on using objectĆ client for an OMG CORBA program, which simply oriented prints technology to create a test infrastructure that is hello, world." In this case, the phrase will be printed based on by thea number of small, selfĆcontained modules and say_it method provided by the server code. The followingthen developing these modules in a way that allows the testĆ descriptions point out some of the differences between ing effort the to proceed while the test infrastructure continues two source files. The numbers associated with the to descripĆ evolve. Each step of the evolution results in a usable test tions correspond to the numbers in Fig. 3. infrastructure that keeps the test effort online and provides critically needed support to product releases. 1. Fig. 3b shows portions of the test TestCase code and with TestEnvironment instrumentation. These classes are not in In the addition to our overall commitment to complete this projĆ code in Fig. 3a. ect, and a desire to see it used in other organizations in HP
80 December 1995 HewlettĆPackard Journal // Standard C++ headers // Standard C++ headers #include
CORBA::string_free(message); CORBA::string_free(message); CORBA::release(hello); CORBA::release(hello); CORBA::release(hello_objref); CORBA::release(orb); CORBA::release(orb); } }
(a) (b)
Fig. 3. (a) Test code before being ported to the object testing framework. (b) The same test after being ported. involved in distributed object technology, we will continue2. K. Beck and W. Cunningham, A Laboratory for Teaching ObjectĆ to participate in standards organizations such as OMG Oriented and Thinking," SIGPLAN Notices, Vol. 24, no. 10, October 1989. X/Open to follow the work that is being done in HP-UX the 9.* andarea 10.0 for HP of 9000 Series 700 and 800 computers are X/Open Company UNIX testing. To date, we have evaluated and provided 93 feedback branded products. to the X/Open Consortium and have a representative UNIX is moniĆ a registered trademark in the United States and other countries, licensed exclusively toring the activity at OMG. through X/Open Company Limited. X/Open is a registered trademark and the X device is a trademark of X/Open Company Lim- References ited in the UK and other countries. 1. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and Open W. Software Lorenson, Foundation is a trademark of the Open Software Foundation in the U.S. and ObjectĆOriented Modeling and Design, Prentice Hall, 1991. other countries.
December 1995 HewlettĆPackard Journal 81