A Thesis Presented to the Faculty of the Graduate School of Cornell University In Partial Fulfillment of the Requirements for the Degree of Doctorate of Philosophy, Mechanical and Aerospace Engineering By Nathaniel Steven Knerr May 2020 2020 Nathaniel Steven Knerr LOST IN DESIGN SPACE: INTERPRETING RELATIONS/STRUCTURE BETWEEN DECISIONS AND OBJECTIVES IN ENGINEERING DESIGN Nathaniel Steven Knerr, Ph. D. Cornell University 2020 In the early phase design of engineering systems, it has become increasingly popular to use a system model and an optimizer to generate a Pareto Frontier of designs that satisfy certain objectives or criteria (e.g. cost, performance, risk). A decision maker then considers this design space and considers which design appears to best satisfy a stakeholder’s requirements. While the generation of alternatives is relatively straightforward, the stakeholder must still make sense of the potentially thousands of alternatives. Additionally, stakeholders often want to consider the possibility of changing priorities, market segmentation via design families and decision sensitivity for further investigation of system priorities. In practice, stakeholders also have many criteria (5 or more), which can be difficult to visualize or represent simultaneously. As such, we aim to create new tools and techniques to extract or aid in finding this information from large multi-criteria decision problems. We approach this with three techniques. The first, Cityplot, uses a virtual reality representation of the design space by use of dimension reduction to represent both the engineering decisions that create a design and the criteria that a stakeholder would care about. A human study is performed to show the validity of the approach by having human subjects perform basic tests and evaluations of real and synthetic design problems. We find the Cityplot allows the intuitive visualization of decisions and a large number of objectives. The second, MOMS-CFCs, provides an interpretation of a single linkage tree and matching procedure to find rules of thumb regarding system dependencies and decision sensitivities. While promising, a few key issues prevent the approach from being applicable in most practical applications. The third, MAPSA, models the shape of the Pareto frontier with a mean plane which parameterizes the space and a model of the residuals which characterizes the overall shape of the frontier. This enables characterization of the tradeoffs and summarization of key features of the frontier. We show some simple mathematical results and create a rough model of how such an approach can be useful for decision making. BIOGRAPHICAL SKETCH Nathan Knerr is a Mechanical Engineering PhD Student at Cornell University under Daniel Selva. His research interests include machine learning, statistical analysis and visualization in service to engineering design and trade-space evaluation. Nathan graduated with high honors from the University of Texas at Austin with a BS in Aerospace Engineering. He is a member of the ASME and AIAA. iii ACKNOWLEDGEMENTS I would like to thank my advisor, Daniel Selva and my thesis committee, Guy Hoffman, Peter Frasier, and Karthik Sridharan for all of their support. I would also like to thank Marcia Sawyer for her great assistance in administrative support during my PhD at Cornell. Finally, I would like to thank Brian Kirby for being open to talk in difficult times. Financially, I would like to thank Daniel Selva for finding funding, primarily via startup resources; that is, I should thank the students and donors of Cornell University and the taxpayers of the state Texas. v TABLE OF CONTENTS BIOGRAPHICAL SKETCH ......................................................................................................... iii DEDICATION ............................................................................................................................... iv ACKNOWLEDGEMENTS .............................................................................................................v TABLE OF CONTENTS ............................................................................................................... vi LIST OF FIGURES ...................................................................................................................... vii LIST OF ILLUSTRATIONS ....................................................................................................... viii LIST OF TABLES ......................................................................................................................... ix BIOGRAPHICAL SKETCH .......................................................................................................... 5 ACKNOWLEDGEMENTS ............................................................................................................ 6 TABLE OF CONTENTS ................................................................................................................ 7 LIST OF FIGURES ...................................................................................................................... 12 LIST OF TABLES ........................................................................................................................ 18 1 INTRODUCTION ............................................................................................................ 19 1.1 Overview and Motivation ................................................................................................19 1.2 Previous Approaches ......................................................................................................22 1.2.1 Systems Engineering and Design Space Exploration .......................................22 1.2.2 Visualization .........................................................................................................24 1.2.2.1 Traditional Approaches .....................................................................................24 1.2.2.2 Novel Engineering approaches .........................................................................27 1.2.2.3 Dimension Reduction Approaches ...................................................................28 1.2.2.4 Virtual Reality for Engineering ........................................................................29 1.2.2.5 Virtual Reality for General Data Analysis ........................................................30 1.2.3 Evaluation of Design Visualization.....................................................................31 1.2.3.1 Evaluation Metrics ............................................................................................31 1.2.3.2 Experiments in Visualization ............................................................................31 1.2.4 Statistical Analysis for Design Space Applications ...........................................32 1.3 Objectives of Research ....................................................................................................34 1.3.1 Formal Notation and Problem Definitions ........................................................34 1.3.2 Aim to Accomplish ...............................................................................................37 1.3.3 Roadmap to a Solution and this Document. ......................................................45 2 CITYPLOT: A VIRTUAL REALITY VISUALIZATION OF THE DESIGN SPACE .. 47 vi 2.1 On Distance Functions ...................................................................................................48 2.2 Nomenclature/Setting .....................................................................................................48 2.3 Problem Types and Distance Functions ........................................................................49 2.3.1 Down-Selection .....................................................................................................50 2.3.2 Assignment............................................................................................................50 2.3.3 Partitioning ...........................................................................................................51 2.3.4 Decision-Option ....................................................................................................51 2.3.5 Continuous Design Spaces ...................................................................................52 2.3.6 Distance Themes and Summary .........................................................................52 2.4 Cityplot Drawing/Technique ..........................................................................................53 2.4.1 Skyscraper Normalization and Visibility...........................................................54 2.4.2 City Placement with Multidimensional Scaling ................................................55 2.5 Application Methodology And Considerations ..............................................................56 2.6 Features To Extract From Cityplot ................................................................................58 2.6.1 Smoothness, Design Families and Tradeoffs .....................................................58 2.6.2 Interactions of Distance and Set Choices ...........................................................59 2.7 Examples .........................................................................................................................60 2.7.1 Toy Example 1: Effect of distance function choice ...........................................60 2.7.2 Toy Example 2: Continuous