Regular Paper Orthogonal Array Experiment in Systems Engineering and Architecting Thomas V. Huynh* Department of Systems Engineering, Naval Postgraduate School, 777 Dwyer Road, Monterey, CA 93943 ORTHOGONAL ARRAY EXPERIMENT IN SYSTEMS ENGINEERING AND ARCHITECTING Received 9 May 2008; Revised 23 March 2010; Accepted 4 June 2010, after one or more revisions Published online in Wiley Online Library (wileyonlinelibrary.com) DOI 10.1002/sys.20172 ABSTRACT This paper espouses the application of orthogonal array experiment to solve a class of engineering optimization problems encountered in systems engineering and architecting. It also illustrates the applicability of orthogonal array experiment in systems engineering and architecting with two examples: verification and validation of the performance of a bandwidth allocation algorithm and architecting of a system of systems to respond to small boat attacks by terrorists. The orthogonal array experiment approach does not call for linearization of nonlinear engineering optimization problems; using orthogonal arrays, it solves them directly by carrying out the smallest possible number of experiments and determin- ing their solutions from the results of the experiments. The orthogonal array experiment method has been found to be effective and efficient for these problems. The feasibility of applying the orthogonal array experiment approach to these problems suggests its potential application to other optimization problems encountered in systems engineering and architecting. © 2010 Wiley Periodicals, Inc. Syst Eng 14: Key words: orthogonal array experiment; systems engineering and architecting; engineering optimization problem; assignment problem 1. INTRODUCTION to support systems engineering management and systems engineering methodology. This paper deals with systems Three pillars of systems engineering are systems engineering engineering methods and tools. Specifically, it deals with management, systems engineering methodology, and systems systems analysis used in systems engineering and systems engineering methods and tools [Sage, 1992]. Systems engi- architecting. Systems analysis supports many areas, such as neering methodology and system engineering management requirements analysis, functional analysis, design evaluation, are the two pillars that must occur for a successful production synthesis and allocation of design criteria, determination of of a system. Systems engineering methodology involves sys- system key drivers, system performance assessment, design, tem definition, system design and development, and system development, detail design performance analysis, system per- deployment. Systems management consists of a task manage- formance analysis, trade-off studies, etc. Systems analysis ment structure, managing systems engineering tasks, and often involves solving optimization problems. decision making with respect to system development. The An optimization engineering problem can often be cast as an assignment problem (or a mathematical programming third pillar, systems engineering methods and tools, is needed problem). There are many mathematical methods to solve assignment problems of different types, such as integer pro- * E-mail: [email protected] gramming problems (linear and nonlinear), mixed integer programming problems, etc. [Minoux, 1986]. As the dimen- Systems Engineering sion of the problem (i.e., the number of assignment variables © 2010 Wiley Periodicals, Inc. and constraints) increases, the time it takes to solve the 1 2 HUYNH problem increases. Often it is impractical (because it is time- mune system which serves as a local search mechanism for consuming) to solve these assignments problems when a genetic algorithm. Holness et al. [2006] treat the personnel Monte Carlo method is involved. Often a quick solution is assignment problems from a systems view, using human needed to enable quick decision making. Often a heuristic factors methodologies and research methods such as macro- algorithm would be preferred over mathematical optimization ergonomics, human-computer interaction, the skills-rules- techniques. Often an optimal solution is needed without hav- knowledge framework, hierarchical task analysis, decision ing to search for all possible optimal solutions. And often a ladders, and abstraction decomposition spaces. problem in systems analysis calls for just an approximate A matrix experiment consists of a set of experiments in solution of an optimization problem with or without con- which the settings of the various product or process parame- straints. In these cases, the orthogonal array experiment ap- ters of interest are changed from one experiment to another proach has been proven to be effective in providing solutions and from which the data are then analyzed to determine the to the optimization problem. effects of the parameters on the response of the product or This paper espouses the application of orthogonal array process [Phadke, 1989]. An orthogonal array experiment is a experiment to solve a class of engineering optimization prob- matrix experiment using special matrices, called orthogonal lems encountered in systems engineering and architecting and arrays. Section 3.1 discusses orthogonal arrays in some detail. illustrates the mechanism of applying orthogonal array ex- Orthogonal array experiment is used heavily in quality engi- periment to the problems treated in Huynh and Gillen [2001] neering in general and robust design in particular. Quality and Huynh et al. [2007]. The problem addressed in Huynh and engineering is concerned with reducing the costs incurred Gillen [2001] is a system algorithm performance verification prior to and after the sale of a product [Taguchi, 1978, 1986, and validation problem in which heuristic algorithms are 1987; Taguchi, Wu, and Chowdhury, 2004; Taguchi and Wu, employed to determine bandwidth to be allocated on demand 1979; Taguchi and Phadke, 1984; Kackar, 1985, 1986; in a satellite communications system that maximizes satisfac- Clausing, 1988; Byrne and Taguchi, 1986; Bendell et al., tion of requests for bandwidth. The problem addressed in 1989]. Robust design is a systematic and efficient method of Huynh et al. [2007] is a systems architecting problem in which design optimization for performance, quality, and cost architectures of a system of systems to respond in a cost-ef- [Phadke, 1989]. Software testing has also benefited from fective manner to terrorists using small boats to attack mari- orthogonal array experiment [Taguchi et al., 2004; Phadke, time commerce traffic and critical shore infrastructure. This 2009]. Jeang and Chang [2002] combine the use of orthogonal problem is a subset of a larger problem of architecting a arrays, computer simulation, and statistical methods in the system of systems responding to maritime domain terrorism rapid development of new products and the planning and early [Huynh et al., 2009]. The purpose of this paper is thus to implementation of product development. In this paper, again, demonstrate the applicability of the orthogonal array experi- orthogonal array experiment is used to solve engineering ment approach to solving these engineering optimization optimization problems in systems engineering and architect- problems. ing formulated as assignment problems (or mathematical These problems are not the only assignment problems programming problems). This paper is not purported to serve encountered in systems engineering and architecting. Indeed, as an introduction of robust design. Rather, it espouses the assignment problems abound in systems engineering. Some employment of orthogonal array experiment to solve assign- of these assignment problems tackled in recent times are now ment problems. mentioned. Attagara [2006] uses heuristics to solve an NP- As discussed in Section 3.2, an orthogonal array used in hard (nondeterministic polynomial-time hard) assignment an orthogonal array experiment employed in robust design problem of allocating both the type and the number of explo- does not capture all of the experiments of a full factorial sive scanning devices at airports to different groups of passen- design. Consequently, degradation of the results otherwise gers with carry-on baggage so as to maximize the total airport obtained with the full factorial design could occur. Sound security while satisfying budget, resource, and throughput engineering judgment during the planning phase of the ex- constraints. Vidalis et al. [2005] model serial flow or produc- periment design is therefore necessary in order to properly tion lines as tandem queuing networks and formulate them as incorporate potentially significant interactions of factors into continuous-time Markov chains to minimize the average the orthogonal array [Peace, 1992]. Similarly, the solutions of work-in-process when the total service time and the total assignment problems obtained with the orthogonal array ex- number of service phases among the stations are fixed. Pettit periment as an approximation solution method could poten- and Veley [2003] discuss risk allocation in airframe systems tially deviate from the optimal solutions (i.e., degradation) engineering in general and in particular the concept of allo- obtained with the mathematical programming methods. The cating system-level risks in multidisciplinary design prob- solutions of the assignment problems that have been treated lems. Vidal [2003] solves service allocation problems in exhibit some degradation as well as agreement [Huynh, 1997; which a