SYSTEMS AND MODELING: A VISUAL ANALYTICS APPROACH

Dimitri N. Mavris*, Olivia J. Pinon*, David Fullmer Jr.* *Aerospace Laboratory (ASDL) School of Aerospace Georgia Institute of Technology, Atlanta, GA, 30332-0150, USA

Keywords: Systems Design, Visual Analytics, Surrogate Modeling, Physics-Based Modeling, Probability Theory

Abstract process. More importantly, this work illustrates that the benefits to the analyst and decision The process of designing unconventional maker of enablers such as surrogate modeling aerospace vehicle configurations is characterized or probability theory are limited if they are not by a lack of relevant , information, and integrated with the , , knowledge, which prevents the from and analytical reasoning capabilities provided by applying traditional . To address Visual Analytics. this issue, significant amount of data need to be generated, collected, and analyzed to increase 1 Introduction the designer’s knowledge about the physics of the problem and to explore the diverse spectrum Design is the process of defining and exploring a of available design options. However, the vast of possibilities that requires the build- amounts of data required to successfully pursue ing up of knowledge and familiarity with the con- unconventional can rapidly become straints and trades involved. Design is also a overwhelming, hence limiting the designer’s problem-solving activity that maps a set of re- ability to fully comprehend the problem to be quirements to a set of functions, leading to a set solved. This paper presents a multidisciplinary or series of decisions that contribute to the final perspective termed Visual Analytics which description of a solution [49, 50]. These state- can be applied to address these issues in the ments are particularly true for the design of con- context of the pre-conceptual/ ventional aircraft. The design of unconventional of unconventional configurations. The paper configurations, on the other hand, is more chal- discusses the enabling techniques essential for lenging. Indeed, while there is historical infor- the visual analytics approach and illustrates mation available on which to base traditional air- that the main requirements for a successful pre- craft designs, unconventional designs are plagued conceptual/conceptual design are to reduce the by a lack of relevant data and knowledge. Also, designer’s cognitive burden, foster his rapid un- the assumptions embedded in the historical data derstanding of the design problem, and support are very often incompatible with unconventional informed decision making. Finally this paper design concepts. Hence, significant amounts of discusses and demonstrates, through the design data need to be generated, gathered, and analyzed of a supersonic business jet, the importance and to improve the designer’s knowledge about the benefits of implementing these enablers and problem and to capture the various design options of integrating Visual Analytics into the design and perspectives involved.

1 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

The amount of data required to success- 2.1 Traditional Design of Conventional Con- fully pursue unconventional aircraft designs can figurations rapidly become overwhelming [18]. The analyst or decision maker, when faced with such a data Current legacy tools and design methods for overload problem, is limited in his ability to con- conventional configurations have been developed duct any kind of trades, test hypotheses, explore based on the information and capabilities avail- the design space, and detect unexpected trends, able at their inception. In the early 1900s, the detail or relations, as the data sets cannot be visu- Wright brothers initially developed their designs alized [39]. Consequently, he cannot fully com- based on the literature of the time. After a num- prehend the problem to be solved, or understand ber of failures, they set out to build a facility for the behavior of the system under consideration. wind tunnel testing. Numerous shapes (airfoil While unprocessed data does not hold any intrin- models) were tested and documented, resulting sic value [39], it can result in missed opportuni- in a library of information that they could rely on ties for critical actions, which may, in turn, result [30]. Since this time, aerospace engineers have in poor designs and significant loss of time and been constructing models to collect data useful money [40]. To alleviate this problem, it is nec- for design. essary to move away from static representations Expert who are well versed in these and visualizations and develop that en- information resources follow a three-phase de- able the interaction between information, and the sign process comprising conceptual, preliminary, human cognitive and perceptual systems, while and detailed phases. During this process, “de- simultaneously allowing users to integrate their sign evolves in increasing levels of detail, from background, expertise, and cognitive capabilities high- representations of the overall concept into the analytical process. The need to address to the details of each and every component” [19]. these aspects has given rise to a multidisciplinary In the conceptual phase of the design, a hand- perspective named Visual Analytics. ful of design concepts are considered [61, 65]. In the remainder of this paper we first discuss These concepts are created using a predictive set the design of conventional and unconventional of tools based on existing flight data documenting configurations, identify the technical and human both aircraft configuration and characteristics. At and challenges linked to the design of unconven- the end of the process, a full-scale aircraft is sub- tional configurations, and present techniques and jected to experimentation and flight testing, thus methods to address them. Then, we introduce Vi- generating data and knowledge leveraged to de- sual Analytics, describe its process, and briefly sign the next generation of aircraft. This library discuss the merit of implementing Visual Ana- of information, built through successive aircraft lytics in design. We later present an unconven- developments, represents the best source of infor- tional design problem to illustrate how the inte- mation on which to base derivative design con- gration of Visual Analytics into the design pro- cepts. cess provides the designer and decision maker It is often taught that design is an art or talent with the capabilities to gain the knowledge and that is obtained and built upon over time. Daniel insight needed to make informed decisions. Raymer [61] states

2 Systems Design and Modeling “If the designer is talented, there is a lot more than meets the eye on This section discusses the design of conventional the drawing. A good aircraft design and unconventional configurations with a partic- seems to miraculously glide through ular emphasis on the challenges associated with subsequent evaluations by special- unconventional configurations. ists without major changes being re- quired.”

2 Systems Design and Modeling: A Visual Analytics Approach

This talent described by Raymer is invalu- • Communication able because it indicates a familiarity with the Communicating expertise or results is es- design and an innate understanding of the con- sential to any decision making exercise sequences of early decision making. However, as such as design. Most of the challenges discussed in the following section, the character- in communicating results lie in the abil- istics and confidence that define great designers ity to present the data with an appropriate are lost when faced with the task of designing level of detail, enable the rapid extraction unconventional configurations or configurations of relevant information by all the parties involving radically new technologies. involved, independent of their background or expertise, retain the audience’s atten- 2.2 Advanced Design of Unconventional tion, and clearly articulate important in- Configurations formation to obtain the stakeholders’ buy- Design requires the exchange of data, informa- in. Based on the realization that abstract tion and knowledge [24]. However, the class knowledge without representation is hard of configurations considered for the new meth- to work with, Burkhard [15] proposes to ods presented in this paper are configurations “combine different visualization types that for which there does not exist any historical complement one another to illustrate dif- database, the designer’s expertise is limited or ferent levels of detail.” Presenting informa- nonexistent, and there is typically no feasible tion from various perspectives and formats design space unless new technologies are in- is also more likely to satisfy the different fused. The absence of these three key features thinking styles, expertise and knowledge of prevents the designer from applying traditional the stakeholders, thereby improving com- design methods. More particularly, it prevents munication between them and reducing po- him from conducting any kind of trades, un- tential misunderstandings and conflicts. derstanding sensitivities, identifying active con- • Collaboration straints and feasible design spaces, hence limit- ing the chances that these configurations be suc- The multi-disciplinary nature of design cessfully developed. calls for the involvement of various people The following sections describe in more de- with different qualifications, background tail the main human and contextual challenges as- and expertise. Stakeholders may have dif- sociated with the pre-conceptual/conceptual de- ferent interpretations of the data or may sign of unconventional configurations. These contribute at different levels of the analy- sections also briefly discuss the enablers and sis [27]. In particular, differing perceptions methods necessary to address these challenges of the design problem among the actors and provide the designer with the knowledge and may lead to difficulties in reaching consen- insight necessary to the successful development sus. Consequently, there is a need, as previ- of unconventional configurations. These meth- ously discussed, to allow users to integrate ods are further illustrated in the context of a de- their background, expertise and cognitive sign problem in Section 4. capabilities into the analytical process.

2.2.1 Human Challenges and Solutions 2.2.2 Contextual Challenges and Solutions This section provides a brief discussion on • Lack of Physical Data: human-related challenges during the design of As previously mentioned, the design of unconventional configurations. unconventional configurations is character- ized by a lack of physical and historical data. One way to alleviate this issue is

3 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

through physics-based modeling and nu- be provided with “a parametric model of merical simulation. To be of value, the the airplane to allow quick changes in the physics-based models need to be at the ap- shape and size of the vehicle.” This para- propriate fidelity level for the physics of metric formulation should also have the ap- the problem to be captured. The lack of propriate degrees of freedom to allow the data also precludes the designer from ob- decision maker to play the what-if games serving trends that have not been observed he is interested in. before. Under these circumstances, there is thus a need to create the appropriate con- • Integration of Multiple Disciplines: ditions so that trends and causality can be The design of revolutionary, unconven- studied. In other words, the data corre- tional configurations requires that the mul- sponding to a particular designer’s interest tiple disciplines describing the problem, as need to be created. This is accomplished well as their interactions, be integrated. through data farming, a concept coined by However, as mentioned by Kamdar et al. Brandstein and Horne [10], and described the “very large number of variables and by Horne and Meyer [29] as the “study and the physics behind the system are too pro- development of methods, interfaces, and found or esoteric to be fully understood” tools that make high performance comput- [33]. Additionally, the disciplinary interac- ing readily available to modelers and al- tions are so intricately coupled that a para- lows analysts to explore the vast amount of metric environment is necessary to avoid data that results from exercising models.” re-iterating the design process until all re- quirements are met [20]. This parametric • Sensitivity to Requirements: environment, depending on the level of fi- Design requirements are a direct driver of delity of the disciplinary models it is com- cost and, therefore, affordability. It is thus posed of, may run slowly, hence limiting critical to understand and capture the sig- the designer’s ability to explore the design nificance of requirements’ impact on the space. Indeed, as acknowledged by Ligetti design [4, 11], as well as the sensitivity [46] et al. “speed is critical for certain of affordability metrics to requirements. cognitive tasks.” A successful approach Also, the mission requirements and con- that addresses this challenge, while pro- straints associated with the design of a viding an all-encompassing model for ex- new concept are likely to change as the ploring a complex design space, is surro- stakeholders learn about the design prob- gate modeling [20, 33]. Surrogate mod- lem. Each change may result in the selec- eling enables virtually instantaneous anal- tion of a different design concept. Conse- yses to be run in real-time [46] by ap- quently, unless multiple scenarios are in- proximating computer-intensive functions vestigated and documented, a static ap- or models across the entire design space proach to design will rapidly show its limi- with simple mathematical/analytical mod- tations. Additionally, decision making im- els [83]. A variety of surrogate mod- plies considerations beyond the technical eling techniques exists that yield insight aspects. Consequently, the final solution into the relationships between design vari- may be different from one decision maker’s ables (x) and responses (y) and facilitate perspective to the next. There is thus a concept exploration [83, 71]. The most need to move from deterministic, serial, prevalent ones include Response Surface single-point designs to dynamic parametric Methodology (RSM) [54, 9, 8], Artificial trade environments. In particular, as advo- Neural Network (ANN) [75, 16], Krig- cated by De Baets [20], the designer should ing (KG) [70, 69], and Inductive Learn-

4 Systems Design and Modeling: A Visual Analytics Approach

ing [43]. The reader is invited to consult collected, and analyzed in order to increase [83, 73, 71, 32] for reviews and compar- the designer’s knowledge about the physics ative studies of these different techniques. of the problem. Data by itself has little One of the most popular of these surrogate value if it is not structured and visualized in modeling techniques, RSM, is discussed in a way that allows the designer to act upon more detail in the context of an example it. This challenge can be addressed by en- application (Section 4.4.1). abling the interaction between the informa- tion and the human cognitive and percep- • Uncertainty: tual systems. The design problem is plagued with uncer- 2.2.3 Preliminary Remarks tainty. Uncertainty exists at all levels and is notably present in the requirements, ve- This brief review of the challenges faced by hicle attributes, and technologies that de- the designer during the conceptual design phase fine the design concept. Consequently, the illustrates that, independently of the methods sensitivities of the outcomes to the assump- and techniques proposed, the real requirement tions need to be assessed. The need to ad- is to reduce the designer’s cognitive burden; to dress uncertainty can be handled, as further foster, in a broad sense, his rapid understanding illustrated in Section 4, through the imple- of the design problem; and to support informed mentation of probability theory and proba- decision making. bilistic design methods. Additionally, dur- ing this phase of the design process, the de- Humans deal with the world through their signer does not know which technologies senses. The human visual system, in particular, are going to be implemented in the final de- provides us with the capability to quickly identify sign. Probability theory and probabilistic patterns and structures [82] and supports the tran- design methods can support the designer sition from cognition, the processing of informa- in his selection of technologies by provid- tion, to perception, the obtaining of insight and ing him with the capability to continuously knowledge. Hence, visual representations are of- and simultaneously trade between require- ten the preferred form of support to any human ments, technologies and design concepts. cognitive task because they amplify our cogni- Finally, the use of probability theory in tive ability [68] and reduce the complex cogni- conjunction with RSM can allow the ana- tive work necessary to perform certain activities lyst and decision maker to quantify and as- [38, 40]. From the early ages, when design was sess risk and to explore huge combinatorial conducted on a piece of paper, up until today spaces. The combination of these methods, with the recent advances in Computer-Aided De- as illustrated in Section 4, can enable the sign (CAD) models, design has always been con- discovery and examination of new trends ducted and communicated through visual means. and solutions in a transparent, visual, and As explained by Wong et al. [88], “visual rep- interactive manner. resentations are essential aids to human cogni- tive tasks and are valued to the extent that they • Data Overload: provide stable and external reference points on While this may seem contrary to the lack which dynamic activities and thought processes of physical data previously discussed, it may be calibrated and on which models and the- is important to recognize that designers ories can be tested and confirmed.” However, it of unconventional configurations are fac- is important to understand that “visualization of ing a data overload problem. This prob- information alone does not provide new insights” lem mainly originates from the significant [52]. In other words, information visualization amounts of data that need to be generated, without interaction between the information and

5 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.* the human cognitive system does little to stimu- late human reasoning and enable the generation Information Analytics and synthesis of knowledge or the formulation of Interaction Geospatial Analytics

Cognitive and appropriate conclusions or actions [52]. Scientific Analytics Perceptual Science Scope of A multidisciplinary perspective termed Vi- Visual Analytics Presentation, production, and Statistical Analytics sual Analytics has originated from the need to dissemination address this issue and reasonably appears as the Data Management & Knowledge Knowledge Discovery common enabler to the many solutions enunci- Representation ated in Section 2.2. In particular, Visual Analyt- ics provides visualization and interaction capabil- Fig. 1 The Scope of Visual Analytics (Repro- ities, allowing the analyst and decision maker to duced from [40]) be presented with the appropriate level of depic- tion and detail to help them make sense of the Statistical Analysis Human Cognition Human-centered Computing Data Mining data and synthesize the knowledge necessary to Perception HUMAN Semantics-based Approaches make decisions. Data Management Visual Intelligence

MACHINE Compression & Filtering Information Visualization Decision-making Graphics and Rendering Theory

3 Visual Analytics "The best of both worlds"

3.1 Definition and Scope Fig. 2 Visual Analytics integrates Machine and Visual Analytics, as defined in the National Visu- Human Strengths (Adapted from [39]) alization and Analytical CenterTM(NVACTM) 5- year Research and Development Agenda for Vi- sualization should also be considered as an ex- sual Analytics [78], is the “science of analytical ploration means and not limited to communica- reasoning facilitated by interactive visual inter- tion or presentation purposes [82]. Along the faces.” While Visual Analytics initially had a same lines, John W. Tukey had recommended, in strong focus on homeland security, it has since 1977, a shift from confirmatory data analysis to broadened to various users [37]. It has now de- exploratory data analysis [81]. These aspects are veloped into a field of study that stems from further discussed and illustrated in the following and encapsulates diverse research areas and dis- sections. ciplines (Figure 1) grouped into three main com- ponents: interactive visualization, analytical rea- 3.2 Process soning, and computational analysis. More partic- ularly, Visual Analytics reduces the user’s cogni- The process illustrated in Figure 3 superimposes tive burden by combining and leveraging both hu- the visually enabled reasoning process, as de- man and electronic data processing strengths and fined by Meyer et al. [52] and the Visual Ana- capabilities (Figure 2), with the goal to “make lytics process, with the ultimate goal to support processing data and information transparent for informed decision making. The Visual Analytics an analytical discourse” [39]. process is based on the visualization model pro- Visualization is essential to scientific rea- posed by Van Wijk [82] and further enhanced and soning and the scientific process as a whole. formalized by Keim et al. [40] and Riveiro et al. However, in order to facilitate the generation of [64]. Both processes are discussed concurrently knowledge and the formulation of informed de- in the following paragraphs. cisions, visualization needs to be combined with 3.2.1 Data Pre-processing analytical techniques [39] and embedded in the analysis/reasoning process, as opposed to being The nature of the data sets, as described in Sec- an end-product of it [52]. In other words, vi- tion 1, requires some data pre-processing (DW )

6 Systems Design and Modeling: A Visual Analytics Approach

DATA VISUALIZATION Data Visualizations VISUAL REASONING

U V

Visualizations

Data Pre-processing V S U CV User Knowledge and Insight

DW

Data/ H V Knowledge/ Input V H Informed Decisions Information Insight

H S U CH

Inferences/ Hypotheses

UH DATA ANALYSIS Hypotheses Generation KNOWLEDGE EXTRACTION

Feedback Loop

Functions visualizing data Insight concluded from visualizations V S U CV H Functions that generate hypotheses from data Insight concluded from hypotheses S U CH H Functions that generate hypotheses from visualization Effect of user interactions on visualizations (i.e. filtering, zooming, etc.) V U V Functions visualizing hypotheses V H Effect of user interactions on hypotheses (i.e. generation of new hypotheses) UH Basic data pre-processing functionality (data transformation, cleaning, selection, integration) DW

Fig. 3 The Visual Analytics and Enabled Reasoning Processes (Adapted from [40], [52] and [64]) for any type of efficient data analysis or data vi- potheses can be generated (HS) and visualizations sualization to take place. The goal of this step, can be created (VS). as explained by Kasik et al. [35], is to pre- pare the data for visual representation by “iden- 3.2.2 Hypotheses Generation tifying higher-order characteristics in the data, Hypotheses can be formulated after applying an- such as relationships, trends, summaries, clus- alytical and statistical methods to the data (H ). ters and synopses” [35]. Pre-processing may in- S Additionally, hypotheses can drive the type of vi- clude data cleaning, selection, integration, trans- sualization to be used (V ) and, inversely, visual- formation, etc. [40, 35]. In particular, as ex- H izations can help formulate new hypotheses (H ). plained by Russell et al. [68], data transfor- V However, as explained by van Wijk [82], visu- mation “deals with transforming data into vary- alization should not be used to “verify the final ing levels of abstraction or deriving additional truth” [82], as it has been advocated in previous data that has new semantic meaning.” Data pre- publications, due to the subjective nature of what processing is usually achieved through the im- can be observed (different people may observe plementation of mathematical, statistical, and lin- different patterns, certain parameters may better guistic techniques. More detailed information re- explain by a phenomenon than others). Along the garding existing methods can be found in [35]. same lines, visualization can be misleading, be- Once the data have been pre-processed and trans- cause the data used to create a particular visual- formed into efficient data representations, hy- ization could have been subjectively manipulated

7 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.* by a user wishing to see a particular outcome. Based on his newly acquired knowledge and Consequently, it is important that the formulated exploration objectives, the user may decide to ob- hypotheses be verified afterwards [82]. tain additional insight both on the data and its visualization. He may recompute the data and 3.2.3 Data Visualization steer the analysis in a different direction (feed- Visualization is a means to extract and present back loop) or he may dynamically interact with relevant information from large volumes of gen- the visualization (UV ) through analytical means erated or compiled data [82] in a format that en- and techniques like brushing and linking (con- ables reasoning and analysis while allowing the necting of two or more views of the same data), user to navigate the overall space spanned by the and/or panning and zooming (smoothly moving data [35]. As explained by Kasik et al. [35], a camera across a scene while increasing or de- data representations obtained during the data pre- creasing the magnification of the objects in the processing phase (Section 3.2.1) need to be fur- scene) [52] to focus on a different region or di- mension of the data space. This is expressed in ther transformed (VS) in order to provide the user with intelligible and effective visual representa- the Visual Analytics Mantra proposed by Keim et tions (Figure 4). Hence, visualization methods, al.: “Analyze first, show the important, zoom, fil- defined by Lengler and Eppler [45] as “a system- ter and analyze further, details on demand” [40]. atic, rule-based, external, permanent, and graphic Exploring and investigating the data through representation that depicts information in a way a different or more focussed angle can offer a that is conducive to acquiring insights, develop- new perspective on the problem, thus helping ing an elaborate understanding, or communicat- the user refine existing hypotheses and formulate ing experiences” can be applied to data represen- new ones (UH). As claimed by Keim, “the vi- tations to help the user identify patterns, trends, sual data exploration process can be viewed as a etc. and better disseminate results and synthesize hypothesis-generation process” [38]. These new knowledge [35]. hypotheses, through the process of knowledge ex- traction, may in turn contribute to increased in-

Concise Visual sight (UCH) and a better understanding of the Raw Data Representations Representations problem. This disciplined, iterative, and interactive Fig. 4 The Two-Step Data Transformation and process [52] by means of which the user learns Representation Process about the problem through data manipulation, in- formation visualization, and hypotheses genera- The resulting visual representations depend tion and testing, eventually leads to the formula- on the type of data, the task to be accomplished, tion of informed decisions. and the time-frame allowed for the completion of the analysis [35, 38, 45]. 3.3 Preliminary Remarks 3.2.4 User Knowledge and Insight As discussed in Section 1, design is primarily a The user is a critical element of the Visual Ana- problem-solving activity. In the case of uncon- lytics process [35] but he often knows very little ventional design, characterized by a lack of his- about the data [38]. The knowledge or insight torical data and limited designer’s expertise for he may gain from visualization (UCV ) and visual this particular problem, this problem-solving ac- data exploration, in particular, is enabled through tivity requires that the data, knowledge, and in- visual reasoning and depends on the level of in- sight necessary for the formulation of informed formation and interaction provided, his level of decisions be generated throughout the design expertise and a priori knowledge, as well as his process. While significant amounts of data can cognition and perception capabilities. be created through simulation, the generation of

8 Systems Design and Modeling: A Visual Analytics Approach knowledge and insight necessitates a framework sections illustrate how the integration of Visual with which: Analytics in the design process provides the an- alyst and decision maker with the capabilities • Large and complex data sets can be ex- to gain the knowledge and insight necessary to plored make informed design decisions. • Initial concepts and hypotheses can be for- mulated and later affirmed or discarded 4.1 Design Problem and Requirements Defi- nition • Information can be synthesized and shared The definition of the design problem has been • Scientific and analytical findings can be recognized as a central issue in design [67]. In documented and communicated particular, the design of a new air vehicle is a Visual Analytics, the goal of which is to support challenging undertaking that requires a proper the analytical reasoning process [79], is thought understanding of the mission type to be per- to provide that necessary framework through the formed and the mission requirements to be met. process described above. The following sec- Mission requirements describe how the system tion illustrates, through an unconventional de- should operate for a given mission profile [4] and sign problem, how Visual Analytics and the en- represent what the user needs, the ultimate goal ablers discussed in Sections 2.2.1 and 2.2.2 are of all systems to be acquired. However, as ac- integrated into the design process to address the knowledged by Trainor and Parnell [80], what at aforementioned design challenges, and provide first appears to be the problem is rarely the real the means for the necessary generation of knowl- problem to be addressed due to changing require- edge and insight. ments or ambiguous customer needs. Hence, it is important that significant amounts of time 4 Example Application and effort be directed towards diligently captur- ing the voices and perspectives of all stakehold- The failure of the High Speed Civil Trans- ers/customers. Ultimately, the customer require- port (HSCT) program led by NASA in the mid ments need to be translated into engineering re- 90’s revealed the extreme sensitivity of com- quirements (detailed technical specifications) for mercial supersonic aircraft to both environmental the designer to be able to effectively develop and and economical parameters. Using this knowl- evaluate candidate designs. edge, and motivated by favorable market stud- A quality system developed in the ‘60s by ies, the aerospace community’s interest shifted Yoji Akao and Shigeru Mizuno [2] called Qual- from supersonic airliners to supersonic business ity Function Deployment (QFD) provides the de- jets (SBJs) [33]. Significant amount of research signer with the means to truly capture the cus- has thus been conducted on the design and de- tomer needs and translate those into design re- velopment of quiet SBJs capable of meeting the quirements. Its associated is termed stringent requirements of flying overland at su- a QFD diagram or “House of Quality”. This personic speeds with minimum sonic boom and tool also enables the of requirements, the acceptable levels of engine noise and emissions identification of synergy or conflict between the [11]. engineering requirements, and the benchmarking The following sections discuss, based on pre- of other existing products, when relevant, to help vious SBJ [11, 26, 60, 12, 59, 33, establish engineering targets. 14], the early phases of the design process (pre- Once the iterative process of mapping cus- conceptual/conceptual phases) for this unconven- tomer requirements to quantifiable engineering tional and complex vehicle, emphasizing the de- parameters is completed, the designer needs to sign challenges and enablers. In addition, these provide design alternatives to be tested against

9 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.* these requirements. dependency matrices as well as Multi-Attribute Decision Making (MADM) techniques into the 4.2 Concept Space Definition selection of product features. An IRMA, as il- lustrated by Figure 5, is populated through brain- The concept space represents all possible solu- storming and includes all possible combinations tions of a design problem. The goal of defining of the activity. The simplified ma- the concept space, as described by Kirby [41], is trices in Figure 5, for instance, include 1.8 ∗ 109 to “identify a potential class of vehicles and pro- alternatives. However, by enumerating all the vide (...) a starting point for selecting potential possible alternatives along with their dependen- solutions to satisfy the customer requirements.” cies and compatibilities, the IRMA helps scope However, there exists a multitude of combina- an intractable problem space to a manageable tions of different subsystems that may satisfy re- one. The implementation of filtering capabilities, quirements. Different methods such as brain- such as the option to eliminate alternatives with storming and affinity diagramming [85] can be respect to technology maturity, also contributes implemented to enumerate the variety of these in reducing the scope of the problem. Finally, subsystems and generate design alternative com- the integration of MADM techniques such as the binations. Morphological Analysis (MA), as dis- Technique for Order Preference by Similarity to cussed in the following section, has been par- Ideal Solution (TOPSIS), which allows decision ticularly successful in defining a concept space makers to rank alternatives according to the im- for complex system engineering problems such portance of pre-determined criteria [23], further as this one. supports experts in down-selecting the number of options to be considered. 4.2.1 Morphological Analysis The implementation of the aforementioned MA was first developed in 1966 by Fritz Zwicky, capabilities enables the exploration of a high a Swiss astronomer and astrophysicist, who used number of options, as well as the identification it to develop jet and rocket propulsion systems of potential concepts or new technological and propellants [91]. This technique has since combinations for a potential system. This been extensively used in a variety of scien- process, during which both requirements and tific disciplines [47, 62, 63, 86] and success- design configurations are considered, eventually fully implemented in complex system engineer- supports subject matter experts in determining ing problems [7, 41]. MA is particlualry at- the baseline around which a concept space that tractive for multi-dimensional, non-quantifiable, satisfies the customer requirements may exist complex problems [62] because it provides a [11]. The chosen baseline aircraft concept, structured, functional, and intelligent means to shown by the shaded boxes in Figure 5, can decompose the problem and generate alternatives be further decomposed into a set of geometric [41]. MA is implemented through the develop- and propulsive parameters that define the design ment of a matrix of alternatives, or morpholog- space to be investigated. The IRMA thus greatly ical matrix, which is formed by "identifying the helps decision makers eliminate some alterna- major functions or characteristics of a system on tives and reduce the concept space to a defined the vertical scale, and all the possible alternatives region that can be further investigated. (or system attributes) for satisfying the character- istics on the horizontal scale” [41]. Concept space exploration, design space ex- The concept of the Interactive Reconfigurable ploration, and feasibility identification, as dis- Matrix of Alternatives (IRMA), developed by cussed further in this paper, are followed by a se- Engler et al. [23], further extends the capabil- lection process in which the decision maker has ity of the matrix of alternatives by incorporating control [5, 76]. However, in the case of revo- new concepts such as filters, compatibility and lutionary, unconventional concepts, this control

10 Systems Design and Modeling: A Visual Analytics Approach

Fig. 5 Example of an Interactive Reconfigurable Matrices of Alternatives (IRMA) for a SBJ is limited by the lack of empirical data or expe- been successfully implemented [51]. In partic- rience of the decision maker. Additionally, be- ular, GA has shown to be very efficient in sup- cause the concepts are defined by several param- porting trade-offs in the early definition of sys- eters that must be allowed to vary concurrently, tem taxonomy [57, 17, 14, 13, 31]. While GA the decision maker cannot mentally analyze them can be used at different stages of the conceptual without the aid of graphical tools. The concepts design phase (concept space exploration, design are also often judged against multiple conflict- space exploration, feasibility identification, etc), ing objectives that are sensitive to the values of this paper only illustrates its applicability to con- the design parameters. The concept and design cept space exploration. spaces are thus so complex that it is exceedingly difficult for the decision maker to identify a con- 4.3 Concept Space Exploration cept with confidence [51]. It also happens that multiple different concepts result in equally fea- As previously discussed, the IRMA enables the sible design. Finally, as discussed by Mavris and reduction of the concept space to a reduced num- Jimenez [51], small variations in requirements ber of alternatives. The remaining alternatives and constraints can quickly change what con- are further decomposed into relevant geometric cept is most appropriate. To address these chal- and propulsive parameters. These parameters and lenges, Genetic (GA) [25], a search, their associated ranges define the design space optimization and machine learning technique that that will be further investigated. The ranges are became known in the 1970s through the work chosen so as to represent the highest number of of Holland [28], can be implemented. The use configurations possible. These alternatives and of Genetic (GA) for concept genera- their parameters guide the designer in deciding tion [56], exploration [72] and selection [17] has which models and codes to integrate to explore

11 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

Safety

Aerodynamics Safety Economics

Aerodynamics Economics

Geometry Structures Manufacturing SYNTHESIS & SIZING Structures Manufacturing Mission

Integrated Routines Approximating Functions Increasing Sophistication Direct Coupling of Analyses Stability & and Complexity Performance Control CONCEPTUAL DESIGN TOOLS (First-Order Methods) Stability & Performance Control Propulsion

PRELIMINARY DESIGN TOOLS (Higher-Order Methods)

Propulsion

Fig. 6 Synthesis and Sizing the concept space. The fidelity of these tools is also dictated by the parameters chosen. Many disciplinary analyses and codes are of- ten necessary to evaluate the feasibility and via- bility of the design. An efficient means to lessen the design cycle time is to study many differ- ent aspects of the design simultaneously. This is achieved through the creation of an integrated and parametric computational synthesis and siz- ing environment (Figure 6) in which conceptual and preliminary design tools and codes are linked together, eliminating the re-keying of informa- Fig. 7 Subset of SBJ Configurations tion from output files to input files. The Genetic Algorithm is then executed to help “identify the dominant designs that may off curve on which designs cannot be improved confirm subject matter expert intuition or reveal in one metric category without a tradeoff in an- unexpected trends” [51]. Through the implemen- other [58]. By selecting any point on the Pareto tation of the GA, different complete design con- front, the designer is also able to access all the cepts can be created in less than a minute, or up information regarding a particular design. to a day, depending on the level of fidelity of The development of a parametric, integrated the design tools and models (Figure 7). These modeling and simulation environment, along designs are sized for the mission requirements, with the implementation of GA, allows the which are also scalable; a change in the required concept space to be quickly explored, providing , for example, will change this matrix of de- the designer with additional insight and a deeper signs. thus provides the user knowledge about the design problem. It allows with the power to test hundreds or thousands of the designer to better understand and quantify designs, where previously, time permitted a sin- the trade-offs between metrics and concepts gle design point only. [22], hence leading him to the confirmation After some number of design generations, or discovery of trends that he could not have the dominant solutions become apparent, hence apprehended or visualized beforehand. Ad- defining the Pareto frontier and the performance ditionally, it enables the concept space to be envelope of the concept space, as illustrated in reduced from a multitude of options to one Figures 8 and 9. The Pareto frontier is a trade- family of concepts and one M&S environment.

12 Systems Design and Modeling: A Visual Analytics Approach

Low Boom Design

Best Compromise Design

Low Drag Design

Fig. 8 -CL/CD as a function of PLdB for various Design Generations

This family of concepts is then further inves- 4.4 Surrogate Modeling tigated, as discussed in the section below. In particular, higher fidelity tools may be inte- As discussed in Section 2.2.2, surrogate mod- grated into the parametric M&S environment eling techniques, by constructing approxima- described above to evaluate the sensibility and tions of analysis codes, supports the integration feasibility of the chosen concepts and assess, in of discipline-dependent and often organization- more detail, their corresponding design variables. dependent codes, and represents an efficient way to lessen the time required for an integrated para- These tools or codes often appear as “black metric environment to run. These techniques also boxes” to the user. Indeed, the equations embed- yield insight into the relationships between de- ded in them are so complex that it is impossible sign variables (inputs) and responses (outputs), for the analyst or decision maker to clearly iden- hence facilitating concept exploration [71]. Ad- tify or grasp the relationships between inputs and ditionally, by enabling virtually instantaneous outputs. This lack of understanding prevents any analyses to be computed in real-time, surrogate sensitivity analyses to be conducted and thus lim- modeling supports the use of interactive and in- its the insight and knowledge that the decision tegrative visual environments [46]. These envi- maker could gain about the design problem [83]. ronments, as described in Section 4.4.1, in turn To address this challenge, Wang and Shan [83], facilitate the designer’s understanding of the de- as well as other practitioners, advocate the use sign problem. Surrogate modeling, as stated by of surrogate modeling and visualization methods Wang and Shan [83], thus provides “a decision to support the decision maker’s understanding of support role for design engineers.” Among the the design problem. Both enablers are discussed variety of surrogate modeling techniques that fa- in the following sections. cilitate the exploration of complex design spaces [83, 71], Response Surface Methodology (RSM), first developed by Box and Wilson [9], is partic- ularly well-accepted and suitable for Aerospace

13 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

where: R: response of interest b0: intercept term bi: regressed coefficient for first-order terms bii: regressed coefficient for second-order terms bi j: regressed coefficient for cross-product terms xi,x j: design variables or factors ε: error associated with second-order approxi- mation The reader is invited to consult [54, 53, 8] for a more detailed description of RSM developments, applications, techniques and tools. The steps in the RSM process are as follows [11, 71]:

Architecture A Architecture B Architecture C 1. Select design variables and their ranges.

Fig. 9 3D Pareto Frontiers 2. Execute a 2-level Design of (DOE) and conduct a Pareto analysis of the significant metrics via an analysis of and Mechanical engineering design applications using the results form the execu- [71]. tion of the DOE: the Pareto analysis al- lows the designer to “capture and visual- 4.4.1 Response Surface Methodology (RSM) ize requirement-attribute sensitivities and RSM [54] is a mathematical modeling technique leverage trade-off analyses at any level that uses a simplified representation, called of system decomposition” [51]. In other Response Surface Equation (RSE) or surrogate words, Pareto analysis reduces the dimen- model, to approximate the behavior of a response sionality and complexity of the solution (such as a disciplinary or system level metric) as space [85] by providing, in order of prior- a function of independent design parameters or ity, the variables that contribute the most to input variables [4, 11, 85]. RSM is particularly the variability of a given response (Figure suitable when several input variables potentially 10). influence a performance measure (response), 3. Select an appropriate DOE for the number but the underlying relationship is unknown of significant factors and number of simu- [54]. As shown in Equation 1, an RSE takes lation cases. DOEs are a series of tests in the form of a polynomial approximation of which purposeful changes are made to in- the relationships across given ranges for the put variables. By only running a limited input variables and usually includes linear, number of cases, predictions can be made quadratic, and interaction terms between the regarding the influences of variables and design parameters [11, 33]. The philosophy their interactions on the responses. DOEs behind the use of a polynomial approximation is provide a maximum amount of knowledge based on the principle of Taylor series expansion. with minimal time and computational ex- penditures [54]. n n n−1 n 2 4. Run the prescribed simulation cases and R = b0 + ∑ bixi + ∑ biixi + ∑ ∑ bi jxix j + ε (1) i=1 i=1 i=1 j=i+1 collecting the appropriate response data.

14 Systems Design and Modeling: A Visual Analytics Approach

proper documentation, legacy). The use of these approximation models also enables the integrated tools to run at a fraction of the time of original models. Consequently, surrogate modeling also empowers the designer to generate and collect larger data sets, hence allowing him to capture more of the dimensionality of the problem. How- ever, the amounts of data generated can rapidly become overwhelming and prevent the designer from learning about the design problem any fur- ther. Alleviating this issue, as discussed in Sec- tion 1, requires that visualization capabilities be developed to help reduce the designer’s cogni- tive burden, foster his rapid understanding of the design problem, and support informed decision making. The importance of visualization-enabled design space exploration in general, and of visu- alization methods for multidimensional data sets . . in particular, has been widely recognized as a . means to support engineering design and deci- sion making [46]. In particular, Wong and Berg- Fig. 10 Example of Pareto Analysis for CO Re- 2 eron [87] mention that such techniques have for duction objective the synthesis of multidimensional data sets and the identification of key trends and rela- 5. Perform multivariate to tionships. build RSEs. Companies such as Chrysler, Boeing, Ford, Lockheed Martin or Raytheon, to name a few, 6. Validate the model with a confirmation test have invested significant efforts in the use of vi- and random sample case. sualization to speed and improve After validation, the RSEs obtained can be and development. The research community has used to perform rapid optimization, Monte Carlo also worked on the development and implemen- simulation, or design space exploration [11, 83], tation of diverse design space visualization en- as discussed in the following section. vironments. Past efforts to visualize multidi- mensional data include programs such as Xmd- vTool [84], Xgobi [77], VisDB[36], and Win- 4.5 Design Space Exploration Viz [44]. More recent work, such as the one by Marsaw et al. [48], for instance, discusses As previously discussed, surrogate modeling the use of an interactive visualization environ- supports the use of higher-fidelity analysis for ment to help determine the technologies and en- unconventional configurations by facilitating the gine point designs that meet specific performance integration and automation of organizationally and economic targets. In particular, this envi- disseminated tools. In particular, the creation ronment features a scatterplot that allows the de- of physics-based approximation models (surro- signer to display simultaneously both design vari- gate models) can replace the higher fidelity tools, ables and responses and to filter the discrete de- which are usually described as too slow for use in signs to determine regions of the design space the design process, cryptic in their use of inputs, that are the most promising for further and more interfaces and logic, and non-transparent (lack of detailed exploration. Additional recent interac-

15 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.* tive and multidimensional design space visual- nication and informed discussions between the ization environments include, among others, the different actors. ARL Trade Space Visualizer (ATSV) [55, 76], Baker et al. [4] and Baker and Mavris [3] Cloud Visualization [21], BrickViz [34], the Ad- have recommended partitioning the design space vanced Systems Design Suite [89, 90], the frame- model into three design spaces: a mission re- work introduced by Ross et al. [66], and the work quirement space, a vehicle space, and a tech- conducted by Simpson et al. [74]. These envi- nology space. The simultaneous assessment and ronments incorporate diverse visualization tech- rapid tradeoff between the three design spaces niques (glyphs, parallel coordinates, scatter ma- can be conducted in a Unified Trade-off Environ- trices, 3-D scatter plots, etc.) depending on the ment (UTE) to investigate the interactions among nature of the data and the end goal of the envi- input variables, as well as the impact of the vari- ronment [76]. The reader is invited to consult ability of the requirements, vehicle attributes, and [18] for a thorough review of the theory and tech- technologies on the complex design space. niques used to visualize multivariate design data. Once the major attributes have been identi- The following paragraphs and sections fied and their sensitivities assessed, the follow- discuss, in more detail, the integration of specific ing step consists of identifying if a solution exists design methods and visualization techniques in that meets the requirements and satisfies the con- support of the exploration of the design space straints. However, depicting a solution space re- and the evaluation of a feasible design. quires that the uncertainty that plagued the design problem be considered. Uncertainty in design Design space exploration is a dominant ac- is present in the requirements, vehicle attributes, tivity in conceptual design. The RSEs gener- and technologies that define the design concept. ated describe the overall design space and can The use of probability theory and probabilistic be visualized in a dynamic design space explo- methods in conjunction with surrogate models, as ration tool called a prediction profiler (Figure described in the following section, have been par- 11). A prediction profiler is constructed from ticularly successful in allowing the designers to partial derivatives and depicts the sensitivities of quantify and assess risk, and explore huge com- each response to the design parameters. In other binatorial spaces. words, a prediction profiler enables the user to evaluate how a given attribute xi varies as the 4.6 System Feasibility Evaluation value of x j changes. Hence, when the hairlines (dashed vertical lines in Figure 11) are moved to The main objective of design space exploration indicate the changing of an input variable value, is the determination of feasibility [22] and the the responses are automatically updated through creation, when necessary, of a feasible design the RSE [4]. space. A Monte Carlo Simulation (MCS) is run Such capability is important to investigate on each RSE assuming a uniform distribution on the overall design space and determine which the ranges of all the design variables as a means attributes have the greatest impact on the re- to map all the possible outcomes. This leads to sponses. By being able to explore trends and sen- the generation of Probability Density Functions sitivities in a highly visual, dynamic, interactive, (PDFs) and corresponding Cumulative Distribu- transparent, and collaborative environment, the tion Functions (CDFs). Using probability dis- designers are provided with the capability to con- tributions along with surrogate modeling enables duct and document trade-off analyses and there- thousands of designs across a user-specified dis- fore gain valuable insight and knowledge about tribution (uniform or other) to be quickly gener- the design problem. Such an environment also ated and analyzed, hence allowing a designer to brings the world of the analyst and the world of assess technical feasibility and economic viabil- the decision maker together by fostering commu- ity.

16 Systems Design and Modeling: A Visual Analytics Approach

Calculated Value

1

0.633

0.537

1 Hairlines

0.824

0.535

1

Responses 0.637

0.406

1 Prediction Trace

0.939

0.896

1

0.616

0.484 1 1 1 1 1 1 1 1 Input Value 0.8 0.899 0.670 0.870 0.965 0.903 0.4 0.696 0.918 0.945 0.8 0.333 0.742 0.806 0.833 0.889 0.931

Variable Limits Design Variables

Fig. 11 Dynamic Interactive Design Space Trade-off Environment (all values have been normalized)

By simultaneously representing specific con- able to compare any input or response charac- straints and their associated CDFs in a single plot, teristic of the point design to any other input as illustrated in Figure 12, the designer can then or response in the design space. This can be evaluate how feasible the design space is and achieved by generating and representing multi- quickly identify any “show-stoppers”, constraints ple plots of input vs. input, input vs. output, and inhibiting acceptable levels of feasibility. In ad- output vs. output combinations in a multivari- dition, he is provided with information regard- ate scatterplot. In particular, such visualization ing the magnitude and direction of the needed technique allows the designer to visualize the to- improvements to obtain an acceptable feasible tal variability of a response metric as a function space. However, a design is feasible if it meets of the collective variability of the design param- all requirements concurrently. The requirements eters. It can also represent the locus of bound- associated with two metrics of interest can be ary points of the response metrics, hence allow- evaluated simultaneously using joint probability ing the designer to quickly identify the “best” distributions. Joint distributions can be repre- and “worst” designs achievable within the design sented, along with both future target values and space for each metric [48]. Additionally, differ- Monte Carlo Simulation data, to quickly identify ent concepts or can be color-coded any points that meet the constraints (Figure 13). to make their impact on the design space immedi- Technology metric values can then be extracted ately visible, hence informing the analyst or deci- for any of the points that satisfy these constraints. sion maker about the effect of concepts or archi- Finally, these points can be further queried and tectures, and specific technologies, on the avail- investigated in other dimensions, through brush- able design space [48]. A multivariate scatter- ing and filtering, as illustrated in Figure 14. plot showing all the potential output vs. input As stated by Marsaw et al. [48], the ana- plots simultaneously can also provide essential lyst or decision-maker, in order to explore and insight regarding trends and correlations, while understand the overall design space, should be plots of outputs vs. outputs provide

17 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

Fig. 12 Design Feasibility Assessment Enabled by Surrogate Modeling of the relevant responses. Using a multivariate scatterplot along with filtered Monte Carlo anal- ysis, a technique for performing “inverse design” Fig. 13 Selecting Potential Solutions to Meet Fu- that combines Monte Carlo simulation with sur- ture Goals rogate models, the designer can also quickly in- teractively filter in all dimensions simultaneously trated in Figure 15. When no feasible design and identify the solutions that concurrently meet space exits, as it is often the case for unconven- all of the constraints placed on the output param- tional designs, or if the extents of the feasible de- eters [6]. Hence, filtered Monte Carlo, a method sign space is unacceptable, the designer has the first introduced by Kuhne et al. [42], can help options to: find all the design combinations that meet the constraints and thus present the decision maker • Open design variable ranges: the ranges and analyst with more than one feasible option. considered being wide, as discussed in In particular, the interactive and visual capabil- Section 4.3, this does not represent a viable ities illustrated in Figure 14 allows the analyst, option. through brushing and filtering, to query a com- plex, multidimensional design space graphically, • Relax constraints: this may or may not be reduce the number of solutions to a handful of an option, depending if the constraints are points by excluding undesirable ones, and iden- rigid or not. tify feasible concepts quickly. A feasible design space, if it exists, can also • Select a different concept space: this is be identified by plotting both constraint contours already investigated with the requirements and design points on a bivariate graph, as illus- aspects of the method.

18 Systems Design and Modeling: A Visual Analytics Approach

Fig. 14 Scatterplot Matrix of Outputs

• Infuse new technologies: this is discussed generated from the design space exploration can in more details in Section 4.7. also be shown as three dimensional contours to which constraints can be applied (Figure 16). Being able to identify the constraints that Using the sliders, the surface can be reshaped in overlap with the design space is thus paramount real time, hence enabling trade studies to be per- as it helps the designer gain valuable insight formed and visualized instantaneously. In fact, regarding the constraints to relax or the type of this capability enables the designer and decision technology to infuse. A parametric, dynamic, maker to uncover trends or solutions that have and interactive environment such as the one never been examined in a transparent, visual, depicted in Figure 15, hence allows the designer and interactive manner. This interaction between to rapidly explore hundreds or thousands of the designer and the exploration environment potential design points for multiple criteria, allows the designer to constantly reformulate while giving him the freedom to change the constraints and variables as he gains new insight space by moving both the design point and the about the design problem, thereby facilitating the constraints. The designer is thus able to visualize formulation of informed decision [68, 1]. the active constraints and identify the ones that most prevent him from obtaining the largest As previously discussed, an unconventional feasible space possible and, consequently, from design will likely, at first, not have a feasible gaining the full benefits of the design concept. design space. The creation of a feasible de- Additionally, the response surface equations

19 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

Fan Eff slidebar is moved to the right

Slidebars control variable values

Constraints are set here

White area indicates White area indicates a available design space smaller available design space

Filled regions indicate areas which violate set constraints

Fig. 15 Design Space Exploration

4.7 Refinement of the Design Space

Refining and opening the feasible space entail pushing the state-of-the-art and identifying, eval- uating, and selecting technologies that are under development or close to maturity. These steps are part of methodologies such as the Technology Identification, Evaluation, and Selection (TIES) methodology [41], which provides a process by which requirements can be met and a feasible de- sign space identified. These steps are briefly de- Fig. 16 Surface Plot for Percentage of CO2 scribed in the following sections. sign space is thus the main objective in the early 4.7.1 Identify Technologies phases of design. The techniques and visualiza- tion described above allow the designer to iden- Technologies are identified according to their tify the most limiting constraints, hence guiding compatibility with respect to one another, as well him in the choice of concepts or technologies to as their effects on the system. In TIES, technol- be further investigated or infused. ogy impacts are captured by k-factors and mod- eled as vectors of k-factors. k factors are, in essence, scale factors added within a M&S en-

20 Systems Design and Modeling: A Visual Analytics Approach vironment to model changes introduced by new and economics, can be utilized to compare technologies on particular metrics. Vectors of k- various technology alternatives and may factors, or technology vectors, are then defined be constructed from a user defined utility for each technology whose elements consist of function for which maximization is desired. the benefits and degradations associated with the • Resource Allocation: Resource allocation technology allows the analyst and decision maker to 4.7.2 Evaluate Technologies quickly and efficiently identify the tech- nologies that have the strongest impact on RSM is then used to create surrogate models of the baseline metrics. the responses as a function of k-factors. How- ever, because the technologies considered are of- ten under development, there exists some uncer- tainty regarding their impact on the system. In TIES, uncertainty is modeled by treating each k- factor as a with a specified PDF. Monte Carlo simulation is then run for each set of k-factors sampled.

4.7.3 Technology Selection A technology space exploration is then con- ducted, analogous to the design space explo- ration described in Section 4.5. A prediction profiler illustrating the impact of independent ef- fect of each technology on the responses can be Fig. 17 Example of Technology Frontier built. Similar filtering and visualization capabili- ties can be implemented to determine which tech- The selection of the final concept alternative nology enables a baseline that meets the require- is highly dependent on the people present in the ments. However, for any multi-attribute, multi- room when a decision has to be made. To mini- constraint, multi-objective problem, the selection mize this effect, one of the purposes of the tech- of the “best” alternative is inherently subjective. niques and capabilities described throughout this The techniques proposed by the TIES methodol- paper is to allow the analyst to run cases and ogy are aimed at providing the decision maker conduct some analysis ahead of time to be able with the necessary knowledge and justification to provide the decision maker with the justified for selecting the appropriate set of technologies. means of making an informed decision. In par- These techniques include: ticular, providing some degrees of freedom on the variables with which the decision maker is famil- • MADM Selection Techniques: MADM iar with will eventually help him gain valuable techniques, such as TOPSIS, are used to insight and optimally direct program resources. identify the best mix of technologies for the stated evaluation criteria. 5 Concluding Remarks

• Technology Frontiers: Technology fron- Design is a problem-solving process during tiers allow the analyst and decision maker which data, information, and knowledge are ex- to visualize the limiting threshold effective- changed to foster the formulation of appropriate ness parameters attainable from any com- conclusions and actions. In the case of unconven- bination of technologies (Figure 17). Effec- tional design, historical data are nonexistent, ex- tiveness parameters, such as performance pertise is limited, and new technologies must be

21 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

Fig. 18 The Visual Analytics Process applied to an unconventional Design Problem considered in order to satisfy the constraints and making. In particular, it allows the analyst and create a feasible design space. A brief review of decision maker to the challenges faced by the designer during the conceptual design phase has highlighted the fact, • Rapidly explore huge combinatorial that, independent of the methods and techniques spaces, proposed, the real need is to reduce the designer’s • Identify potentially feasible concepts or cognitive burden, and to support the analytical technology combinations, reasoning process that eventually leads to a better understanding of the design problem. • Formulate and test hypotheses, Figure 18 illustrates, in the context of the Vi- sual Analytics and enabled reasoning processes, • Steer the analysis by requesting additional how the tools and methods described in Section 4 data as needed (data farming), are implemented and integrated to foster knowl- • Integrate their background, expertise and edge and the formulation of informed decisions. cognitive capabilities into the analytical This integration of Visual Analytics in the de- process, sign process offers a valuable answer to the needs discussed above, while strongly supporting both • Understand and quantify trade-offs be- qualitative exploration and quantitative decision tween metrics and concepts,

22 Systems Design and Modeling: A Visual Analytics Approach

• Study correlations, explore trends and sen- confirm that they give permission, or have obtained sitivities, permission from the copyright holder of this paper, for the publication and distribution of this paper as part of • Provide interactive feedback to the visual- the ICAS2010 proceedings or as individual off-prints ization environment, from the proceedings. • Synthesize and share information, References • Investigate the design space in a highly vi- sual, dynamic, interactive, transparent and [1] Ahlberg C and Schneiderman B. Visual in- collaborative environment, and formation seeking: Tight coupling of dynamic query filters with starfield displays. Proc of the • Document and communicate findings and Human Factors in Computing Systems, pp 313– decisions. 317, New-York, NY, USA, 1994. Association However, it is important to acknowledge that for Computing Machinery. although enablers such as response surface [2] Akao Y. Quality Function Deployment: In- methodology or probability theory are essen- tegrating Customer Requirements into Product Design. Productivity Press, 1990. tial to the investigation of unconventional design problems, their benefits to the analyst and deci- [3] Baker A. P and Mavris D. N. Assessing the simultaneous impacts of requirements, vehicle sion maker are limited if they are not integrated characteristics, and technologies during aircraft with the visualization, interaction, and analytical design. Proc of the 39th Aerospace Sciences reasoning capabilities provided by Visual Analyt- Meeting and Exhibit, No AIAA 01-0533, Reno, ics. NV, 8-11 January 2001. Finally, while the conceptual design phase is [4] Baker A. P, Mavris D. N, and Schrage D. P. The the most important phase of design in terms of the role of requirements in conceptual aircraft de- number of concepts to be examined, technologies sign and their interactions with technologies and to be considered, and mappings that need to oc- vehicle attributes. Proc of the AIAA 1st Aviation cur between requirements and configurations, we Technology, Integration and Operations (ATIO) believe that the design methods and solutions de- Forum, No AIAA 2001-5225, Los Angeles, CA, scribed in this paper could, in principle, apply to 16-18 October 2001. all the phases of the design process. [5] Balling R. Design by shopping: A new paradigm. Proc of the Third Wordl Congress 6 Acknowledgment of Structural and Multidisciplinary Optimiza- tion (WMCMSO-3), pp 295–297, Buffalo, NY, This paper has benefited from the very helpful in- USA, 1999. sights and comments offered by Dr Brian Ger- [6] Biltgen P. T. Systems of Systems Engineer- man and Mr Matthew Daskilewicz of the School ing: Principles and Applications, chapter Chap- of Aerospace Engineering, Georgia Institute of ter Five: Technology Evaluation for System of Technology, Georgia. Systems, p 480. Taylor & Francis Group, LLC, November 2008. 7 Copyright Statement [7] Biltgen P. T. A Methodology for Capability- Based Technology Evaluation for Systems-of- The authors confirm that they, and/or their company or Systems. PhD thesis, Georgia Institute of Tech- organization, hold copyright on all of the original ma- nology, Atlanta, GA 30332, May 2007. terial included in this paper. The authors also confirm [8] Box G. E. P, Hunter W. G, and Hunter J. Statis- that they have obtained permission, from the copy- tics for Experimenters. John Wiley & Sons, Inc, right holder of any third party material included in this NY, 1978. paper, to publish it as part of their paper. The authors [9] Box G. E. P and Wilson K. B. On the ex-

23 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

perimental attainment of optimum conditions. of theory and techniques. Submitted to the AIAA Journal of the Royal Statistical Society, Vol. 13, Journal of Aircraft, 2010. No Series B, pp 1–45, 1951. [19] D’Avino G, Dondo P, lo Storto C, and Zezza V. [10] Brandstein A. G and Horne G. E. Data farming: Reducing ambiguity and uncertainty during new A meta-technique for research in the 21st cen- product development: a system dynamics based tury. Technical report, Marine Corps Combat approach. Technology Management: A Unifying Development Command Publication, Quantico, Discipline for Melting the Boundaries, pp 538– VA, USA, 1998. 549, September 2005. [11] Briceno S, Buonanno M, Fernandez I, and [20] De Baets P. W. G and Mavris D. N. Methdol- Mavris D. N. A parametric exploration of su- ogy for the parametric structural conceptual de- personic business jet concepts utilizing reponse sign of hypersonic vehicles. Proc of the 2000 surfaces. Proc of the AIAA 2nd Aviation Tech- World Aviation Conference, No 2000-01-5618, nology, Integration and Operations (ATIO) Fo- San Diego, CA, 10-12 October 2000. rum, No AIAA 2002-5828, Los Angeles, CA, [21] Eddy J and Lewis K. Visualization of multi- 1-3 October 2002. dimensional desgn and optimization data using [12] Bucovsky A, Eyer J, and Mavris D. N. De- cloud visualization. Proc of the ASME Design sign space exploration for boom mitigation on Engineering Technical Conferences - Design a quiet supersonic business jet. Proc of the Automation Conference, No DETC02/DAC- 3rd Aviation Technology, Integration and Op- 02006, Montreal, Quebec, Canada, 2002. erations (ATIO) Forum, No AIAA 2003-6802, [22] Ender T. R, McClure E. K, and Mavris D. N. Denver, Colorado, 17-19 November 2003. Development of an integrated parametric envi- [13] Buonanno M. A. A Method for Aircraft Concept ronment for conceptual hypersonic missile siz- Exploration using Multicriteria Interactive Ge- ing. Proc of the AIAA 2nd Aviation Technol- netic Algorithms. PhD thesis, Georgia Institute ogy, Integration and Operations (ATIO) Forum, of Technology, 2005. No AIAA 2002-5856, Los Angeles, CA, 1-3 [14] Buonanno M. A and Mavris D. N. Small super- October 2002. sonic transport concept evaluation using interac- [23] Engler W. O, Biltgen P. T, and Mavris D. N. tive evolutionary algorithms. Proc of the AIAA Concept selection using an interactive reconfig- 4th Aviation Technology, Integration and Op- urable matrix of alternatives (IRMA). Proc Pro- erations (ATIO) Forum, No AIAA 2004-6301, ceedings of the 45th AIAA Aerospace Sciences Chicago, IL, 20-22 Septembert 2004. Meeting and Exhibit, No AIAA Paper 2007- [15] Burkhard R. A. Learning from architects: The 1194, Reno, NV, January 2007. difference between knowledge visualization and [24] Fujimoto T and Clark K. Product Development information visualization. Proc of the 8th In- Performance. Harvard Business School Press, ternational Conference On Information Visual- 1995. ization (IV’04). London, England, 14-16 July [25] Goldberg D. E. Genetic Algorithms in Search, 2004. Optimization, and Machine Learning. Addison- [16] Cheng B and Titterington D. M. Neural net- Wesley, Reading, MA, USA, 1989. works: A review from a statistical perspective. [26] Hamel L, Folk T, Jimenez H, and Mavris D. Statistical Science, Vol. 9, No 1, pp 2–54, 1994. Conceptual design of an n+2 supersonic airliner. [17] Crossley W. A, Martin E. T, and Fanjoy D. W. Proc of the 9th AIAA Aviation Technology, In- A multiobjective investigation of 50-seat com- tegration, and Operations Conference (ATIO), muter aircraft using a genetic algorithm. Proc No AIAA 2009-7075, Hilton Head, South Car- of the 1st Aircraft, Technology Integration, and olina, 21-23 September 2009. Operations Forum, No AIAA-2001-5247, Los [27] Heer J and Agrawala M. Design considerations Angeles, CA, USA, October 16-18 2001. for collaborative visual analytics. Informtion Vi- [18] Daskilewicz M. J, German B. J, and Mavris D. sualization, Vol. 7, pp 49–62, 2008. Visualizing multivariate design data: A survey [28] Holland J. H. Adaptation in Natural and Ar-

24 Systems Design and Modeling: A Visual Analytics Approach

tificial Systems. MIT Press, Cambridge, MA, pp 38–44, 2001. USA, 1992. [39] Keim D. A, Andrienko G, Fekete J.-D, Görg [29] Horne G. E and Meyer T. E. Data farming: Dis- C, Kohlhammer J, and Melançon G. Informa- covering surprise. Proc of the 36th Conference tion Visualization, Vol. 4950 of Lectures Notes on Winter Simulation, pp 807–813, Washington, in Computer Science, chapter Visual Analytics: D.C., USA, 2004. Definition, Process, and Challenges, pp 154– [30] Jakab P. L and Crouch T. D. The Wright Broth- 175. Springer Berlin / Heidelberg, 2008. ers and the Invention of the Aerial Age. National [40] Keim D. A, Mansmann F, Schneidewind J, Geographic, 2003. Thomas J, and Ziegler H. Visual Data Min- [31] Jimenez H and Mavris D. N. Conceptual de- ing, chapter Visual Analytics: Scope and Chal- sign of current technology and advanced con- lenges, pp 76–90. Lecture Notes in Computer cepts for an efficient multi-mach aircraft. SAE Science. Springer Berlin / Heidelberg, 2008. Transactions, Vol. 114 Part I, No 2005-01-3399, [41] Kirby M. R. A Method for Technology Identi- pp 1343–1353, 2005. fication, Evaluation, and Selection in Concep- [32] Jin R, Chen W, and Simpson T. W. Comparative tual and Preliminary Aircraft Design. PhD the- studies of metamodeling techniques under mul- sis, School of Aerospace Engineering, Georgia tiple modeling criteria. Structural and Multidis- Institute of Technology, Atlanta, GA, U.S.A., ciplinary Optimization, Vol. 23, No 1, pp 1–13, March 2001. December 2001. [42] Kuhne C, Wiggs G, Beeson D, Madelone J, and [33] Kamdar N, Smith M, Thomas R, Wikler J, and Gardne M. Using monte carlo simulation for Mavris D. N. Response surface utilization in probabilistic design. Proc of the 2005 Crystal the exploration of a supersonic business jet con- Ball User Conference, 2005. cept with application of emerging technologies. [43] Langley P and Simon H. A. Applications of ma- Proc of the World Aviation Congress & Exposi- chine learning and rule induction. Communi- tion, No 2003-01-0359, Montreal, QC, Canada, cations of the ACM, Vol. 38, No 11, pp 55–64, September 2003. 1995. [34] Kanukolanu D. L. K. E and Winer E. H. A [44] Lee H.-Y, leng Ong H, whatt Toh E, and Chan multidimensional visualization interface to aid S.-K. A multi-dimensional data visualiza- in tradeoff decisions during the solution of cou- tion tool for knowledge discovery in databases. pled subsystems under uncertainty. ASME Jour- Proc of the 19th Annual International Computer nal of Computing and Information Science in Software & Applications Conference, pp 7–11, Engineering, Vol. 6, No 3, pp 288–299, 2006. 1995. [35] Kasik D. J, Ebert D, Lebanon G, Park H, and [45] Lengler R and Eppler M. Towards a periodic Pottenger W. M. Data transformations and rep- table of visualization methods for management. resentations for computation and visualization. Proc of the Conference on Graphics and Visual- Information Visualization: Special Issue on the ization in Engineering (GVE 2007), Clearwater, Foundations and Frontiers of Visual Analytics, FL, U.S.A, 2007. Vol. 8, No 4, pp 275–285, Winter 2009. [46] Ligetti C, Simpson T. W, Frecker M, Barton [36] Keim D and Kriegel H.-P. Visdb: Database R. R, and Stump G. Assessing the impact of exploration using multidimensional visualiza- graphical design interfaces on design efficiency tion. IEEE Computer Graphics and Applica- and effectiveness. Journal of Computing and In- tions, Vol. 14, No 5, pp 40–49, 1994. formation Science in Engineering, Vol. 3, No 2, [37] Keim D. A, Mansmann F, and Thomas J. Vi- pp 144–154, June 2003. sual analytics: How much visualization and how [47] Mackinlay J, Card S. K, and Robertson G. G. A much analytics. SigKDD Explorations Journal, semantic analysis of the design space of input December 2009. devices. Human-Computer Interaction, Vol. 5, [38] Keim D. A. Visual exploration of large data sets. No 2, pp 145–190, 1990. Communications of the ACM, Vol. 44, No 8, [48] Marsaw A, German B, Hollingsworth P, and

25 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

Mavris D. An interactive visualization en- 132, No 5, pp 1–11, 2010. vironment for decision making in aircraft en- [57] Perez R. E, Chung J, and Behdinan K. Aircraft gine preliminary design. Proc of the 45th conceptual design using genetic algorithms. AIAA Aerospace Sciences Meeting and Exhibit, Proc of the 8th AIAA/USAF/NASA/ISSMO Sym- No AIAA-2007-1333, Reno, NV, USA, 8-11 posium on Multidisciplinary Analysis and Op- January 2007. timization, No AIAA-200-4938, Long Beach, [49] Mavris D. N and DeLaurentis D. A. A CA, USA, September 6-8 2000. stochastic design approach for aircraft afford- [58] Perloff J. M. Microeconomics. Addison-Wesley, ability. Proc of the 21st Congress of the In- Reading, MA, USA, 1999. ternational Council on the Aeronautical Sci- [59] Phan L. L, Yamaoka Y, and Mavris D. N. Im- ences (ICAS), No ICAS-98-623, Melbourne, plementation and benefits of variable geometry Australia, September 1998. wings for a supersonic business jet. Proc of [50] Mavris D. N and Jimenez H. Architecture and the AIAA 3rd Aviation Technology, Integration Principles of , chapter Sys- and Operations (ATIO) Forum, No AIAA 2003- tems Design, pp 301–322. Complex and Enter- 6812, Denver, CO, 17-19 November 2003. prise Systems Engineering Series. CRC Press, [60] Rallabhandi S. K and Mavris D. N. Design 2009. and analysis of supersonic business jet concepts. [51] Mavris D. N and Jimenez H. Architecture and Proc of the 6th Aviation Technology, Integration Principles of Systems Engineering, chapter Ad- and Operations (ATIO) Forum, No AIAA 2006- vanced Design Methods, pp 323–352. Com- 7702, Wichita, Kansas, 25-27 September 2006. plex and Enterprise Systems Engineering Se- [61] Raymer D. P. Aircraft Design: A Conceptual ries. CRC Press, 2009. Approach. 3rd edition edition, Education Se- [52] Meyer J, Thomas J, Diehl S, Fisher B, Keim D, ries, American Institute of Aeronautics and As- Laidlaw D, Miksch S, Mueller K, Ribarsky W, tronautics, Inc, Reston, Virginia, 1999. Preim B, and Ynnerman A. From visualization [62] Ritchey T. General morphological analysis - to visually enabled reasoning. Technical report, a general method for non-quantified modelling. Dagstuhl Seminar 07291 on “Scientific Visual- Proc of the 16th EURO Conference on Opera- ization”, July 2007. tional Analysis, Brussels, Belgium, 1998. [53] Myers R. H, Khuri A. I, and Carter W. H. Re- [63] Ritchey T. Problem structuring using computer- sponse surface methdology: 1966-1988. Tech- aided morphological analysis. Journal of the nometrics, Vol. 31, No 12, pp 137–157, 1989. Operational Research Society, Vol. 57, pp 792– [54] Myers R. H and Montgomery D. C. Response 801, 2006. Surface Methodology: Process and Product Op- [64] Riveiro M, Falkman G, and Ziemke T. Visual timization Using Designed . 2nd analytics for the detection of anomalous mar- edition edition, Wiley Series in Probability and itime behavior. Proc of the 12th IEEE Inter- , John Wiley & Sons, Inc, 2002. national Conference on Information Visualiza- [55] O’Hara J. J, Stump G. M, Yukish M. A, tion (IV’08), pp 273–279, London, UK, July 9- Harris E. N, Hanowski G. J, and Carty 11 2008. IEEE. A. Advanced visualization techniques for [65] Roskam J. Airplane Design I. Darcoporation, trade space exploration. Proc of the 48th 1985. AIAA/ASME/ASCE/AHS/ASC Structures, Struc- [66] Ross A. M, ans J. M. Warmkessel D. E. H, tural Dynamics, and Material Conference, and Diller N. P. Multi-attribute tradespace ex- No AIAA-2007-1878, Honolulu, HI, USA, 23- ploration as front end for effective space sys- 26 April 2007. tem desgn. Jourmal of Spacecraft and Rockets, [56] Patel J and Campbell M. I. An approach to auto- Vol. 41, No 1, pp 20–28, 2004. mate and optimize concept generation of sheet [67] Rouse W. B. On the value of information in metal parts by topological and parametric de- system design: A framework for understanding coupling. Journal of Mechanical Design, Vol. and aiding designer. Information Processing &

26 Systems Design and Modeling: A Visual Analytics Approach

Management, Vol. 22, No 2, pp 217–228, 1986. 1998. [68] Russell A. D, Chiu C.-Y, and Korde T. Vi- [78] Thomas J and Cook K. A, editors. Illuminat- sual representation of construction management ing the Path: The Research and Development data. Automation in Construction, Vol. 18, Agenda for Visual Analytics. IEEE CS Press, pp 1045–1062, 2009. 2005. [69] Sacks J, Schiller S. B, and a W. J. W. Design for [79] Thomas J and Cook K. A, editors. Illuminat- computer experiments. Technometrics, Vol. 31, ing the Path: The Research and Development No 1, pp 41–47, 1989. Agenda for Visual Analytics, chapter 2: The Sci- [70] Sacks J, Welch W. J, Mitchell T. J, and Wynn ence of Analytical Reasoning, pp 33–68. IEEE H. P. Design and analysis of computer experi- CS Press, 2005. ments. Statistical Science, Vol. 4, No 4, pp 409– [80] Trainor T and Parnell G. S. Decision Making in 435, 1989. Systems Engineering and Management, chapter [71] Simpson T. W, Peplinski J. D, Koch P. N, and Problem Definition, pp 263–316. Wiley Series Allen J. K. Metamodels for computer-based in Systems Engineering and Management. John engineering design: Survey and recommenda- Wiley & Sons, Inc, 2008. tions. Engineering with Computers, Vol. 17, [81] Tukey J. W. Exploratory Data Analysis. pp 129–150, 2001. Addison-Wesley, Reading MA, 1977. [72] Simpson T. W. A Concept Exploration Method [82] van Wijk J. The value of visualization. Proc of for Product Family Design. PhD thesis, School IEEE Visualization, pp 79–86, November 2005. of Mechanical Engineering, Georgia Institute [83] Wang G. G and Shan S. Review of metamod- of Technology, Atlanta, GA, USA, September eling techniques in support of the engineering 1998. . Journal of Mechanical De- [73] Simpson T. W, Peplinski J. D, Koch P. N, and sign, Vol. 129, No 4, pp 370–380, April 2007. Allen J. K. On the use of statistics in design and [84] Ward M. Xmdvtool: Integrating multiple meth- the implications for deterministic computer ex- ods for visualizing multivariate data. Proc of periments. Proc of the 1997 ASME Design En- Visualization, pp 326–333, Wahsington, D.C., gineering Technical Conferences (DETC’97), USA, 1994. Sacramento, CA, USA, September 14-17 1997. [85] West P. D. Decision Making in Systems Engi- [74] Simpson T. W, Spencer D. B, Yukish M. A, and neering and Management, chapter Solution De- Stump G. Visual sterring commands and test sign, pp 317–356. Wiley Series in Systems problems to support research in trade space ex- Engineering and Management. John Wiley & ploration. Proc of the 12 AIAA/ISSMO Mul- Sons, Inc, 2008. tidisciplinary Analysis and Optimization Con- [86] Wheelwright S. M. The design of downstream ference, No AIAA-2008-6085, Victoria, British processes for large-scale protein purification. Columbia, Canada, 10-12 September 2008. Journal of Biotechnology, Vol. 11, No 2-3, [75] Smith M. Neural Networks for Statistical Mod- pp 89–102, August 1989. eling. Von Nostrand Reinhold, NY, 1993. [87] Wong P. C and Bergeron R. D. 30 years of mul- [76] Stump G, Simpson T. W, Yukish M, and Bennett tidimensional multivariate visualization. Proc of L. Multidimensional visualization and its appli- the Workshop on Scientific Visualization, 1995. cation to a design by shopping paradigm. Proc [88] Wong P. C, Rose S. J, Jr G. C, Frincke D. A, of the 9th AIAA/ISSMO Symposium on Multidis- May R, Posse C, Sanfilippo A, and Thomas J. ciplinary Analysis and Optimization, No AIAA- Walking the path: A new journey to explore and 2002-5622, Atlanta, GA, USA, 4-6 September discover through visual analytics. Information 2002. Visualization, Vol. 5, pp 237–249, 2006. [77] Swayne D. F, Cook D, and Buja A. Xgobi: [89] Zhang R, Noon C, Oliver J, Winer E, Gilmore Interactive dynamic data visualization in the B, and Duncan J. Development of a soft- x window system. Journal of Computational ware framework for conceptual design of com- Graphical Statistics, Vol. 7, No 1, pp 113–130, plex systems. Proc of the 3rd AIAA Multidis-

27 DIMITRI N. MAVRIS*, OLIVIA J. PINON*, DAVID FULLMER JR.*

ciplinary Design Optimization Specialists Con- ference, No AIAA-2007-1931, Honolulu, HI, USA, 2007. [90] Zhang R, Noon C, Oliver J, Winer E, Gilmore B, and Duncan J. Immersive product configurator for conceptual design. Proc of the ASME De- sign Engineering Technical Conferences - De- sign Automation Conference, No DETC2007- 35390, Las Vegas, NV, USA, 2007. [91] Zwicky F. Discovery, Invention, Research - Through the Morphological Approach. The Macmillan Company, 1969.

28