Crowdsourcing for Engineering Design: Objective Evaluations and Subjective Preferences by Alexander Burnap A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Design Science) in The University of Michigan 2016 Doctoral Committee: Professor Panayiotis Y. Papalambros, Co-Chair Professor Richard D. Gonzalez, Co-Chair Assistant Professor Matthew K. Johnson-Roberson Assistant Professor Honglak Lee c Alexander Burnap 2016 All Rights Reserved For the objective evaluations that unite our values, and the subjective preferences that keep them interesting. ii ACKNOWLEDGEMENTS Thank you to all the following people, I am very grateful for the roles you have played in my life, and hopefully I will continue to play a role in yours: Dato Kighuradze, for mentoring me early on that, \if you understand limits, you understand everything." The grown-up version of that is not limited to math. Will Marotta, the owner of a small mechanic shop that played classic music in engine bays and repaired cars based on need. You were the first to show me how to mix art and technology. Jeff Hartley, the first design scientist I have met at a company. Thank you for years of the long-winded stream-of-consciousness meanderings of the mind and mentoring me without judgment. Diann Brei, for helping my leadership and the writing algorithm; Colleen, for giving me the unabridged read; Zoran Filipi, Jeff Stein, Dawn Tilbury, and Richard Gerth for helping me figure out what was important to work on with vehicles early on in graduate school. Honglak Lee, for opening up the world of changing repre- sentations. Matt Johnson-Roberson, for crowdsourcing but also ethics and work-life balance. To my DESCI family and ODE family: Yanxin Pan and Ye Liu, you are honestly the best possible team to balance our strengths; Max Yi Ren, your constructive criticism was so important, I am very lucky to have began maturing research-wise with you; Namwoo, you always brought compassion; Emrah you helped keep scope of the world ethics and politics; Bill you brought leg day; Yuqing you brought laughs. Steven, Soodeh, and Anna, good times. Thank you also to Giannis, Carlie, Charlie, iii Christina, and Alex Jr. for helping the process. Panos Papalambros and Rich Gonzalez, my advisors and deepest intellectual men- tors: Panos, you have taught me so much, but perhaps \focus" internally and \re- spect" externally. Thank you for the mentorship that evolved to friendship and academic kinship. Rich, paradoxically, it is the opposite; focus \externally" on other fields as nothing is that new, and \respect" internally. Thank you for taking me in and allowing me to develop, I can truly not have asked for better advisors. To my meditation friends, Li Nong and Josh Damron, who share respect for Aurelius, Kabat-Zinn, and Batchelder, for continuing to shape my worldview on the human condition and experience the present. To my life friends, thank you from the depth of my heart. I will continue to give you nice wooden cutting boards at your weddings: Nir, John, Joao, Max, Hannah, Lucija, Zheng, David Dai, Oto, Ananda, Hani, Esteban, Clover, Luis, Efren, and Brian. Lastly, the most important in my life, my family: Mo and Do, you two individually then collectively overcame the most adverse conditions and raised a family with as much integrity and opportunity as possible. I love you very much and hope to one day carry your incredible torch. To my best friend, and brother, Andrew. We have gotten so much closer and I look up to you more than you know. I am not sure I can keep up the 600 mile rule, but I will be a good uncle{dog pound out. iv TABLE OF CONTENTS DEDICATION :::::::::::::::::::::::::::::::::: ii ACKNOWLEDGEMENTS :::::::::::::::::::::::::: iii LIST OF FIGURES ::::::::::::::::::::::::::::::: viii LIST OF TABLES :::::::::::::::::::::::::::::::: xvi ABSTRACT ::::::::::::::::::::::::::::::::::: xviii CHAPTER I. Introduction ..............................1 1.1 Introduction . .1 1.2 Crowdsourcing in Engineering Design . .4 1.2.1 What is Crowdsourcing? . .4 1.2.2 The Design Process and Decision-Based Design . .9 1.2.3 The Promise of Crowdsourcing for Making \Good" and Catching \Bad" Decisions . 12 1.3 Quantitative Crowd Aggregation Models . 15 1.3.1 Single Evaluator Models . 16 1.3.2 Crowd Aggregation Models . 20 1.4 Research Gap and Dissertation Contributions . 27 1.5 Dissertation Overview . 30 II. Why does Crowdsourcing Fail for Objective Evaluations? .. 32 2.1 Context: Do current crowdsourcing aggregation models work for engineering design? . 32 2.2 Related Work . 35 2.3 A Bayesian Network Model for Crowd Aggregation . 37 2.4 Simulated Crowds Experiment . 42 2.5 Human Crowd Experiment . 47 v 2.6 Results . 48 2.7 Additional experiments to assess what went wrong? . 51 2.7.1 Human crowd augmented with simulated experts . 51 2.7.2 Human crowd with \consistently wrong" evaluators removed . 52 2.7.3 Simulated crowd with \consistently wrong" evaluators 53 2.8 Summary . 54 III. Finding Experts in the Crowd using \Expertise Heuristics" 57 3.1 Context: How do we find the experts in the crowd? . 57 3.2 Related Work . 61 3.3 Problem Formulation: Models of Expertise Prediction . 63 3.4 Hypotheses and Experiments . 66 3.4.1 Pilot Study: Calibrating Design Difficulty . 66 3.4.2 Demographics, Task Behavior, Mechanical Reason- ing, and Using an Easy Task to Predict Expertise on the Actual Hard Task . 69 3.5 Results . 75 3.5.1 Practical Usage: Identifying Experts to Improve Crowd Aggregation . 78 3.6 Summary . 80 IV. Do we need to Filter Experts for Subjective Preferences? .. 82 4.1 Context: Balancing Design Freedom and Brand Recognition . 82 4.2 Related Work . 88 4.3 Problem Formulation . 94 4.3.1 L1 Multinomial Logit for Brand Recognition . 95 4.3.2 Design Freedom Distance Metric . 96 4.3.3 Crowdsourced Markov Chain for Design Attributes . 98 4.4 Experiment . 102 4.5 Results . 108 4.6 Summary . 112 V. A Representation to Assess Evaluations and Preferences .. 113 5.1 Context: A \perfect" product form design tool? . 113 5.2 Related Work . 116 5.3 Problem Formulation . 121 5.3.1 Deep Generative Model . 123 5.3.2 Model Architecture . 125 5.4 Experiment: Generating the Last Decade of Automobiles . 127 5.5 Results and Discussion . 130 5.6 Summary . 135 vi VI. Why does Crowdsourcing Fail for Subjective Preferences? . 136 6.1 Context: Which passenger vehicle would you purchase? . 136 6.2 Related Work . 139 6.3 Preference Prediction as Binary Classification . 142 6.4 Feature Learning Models for Preference Prediction . 146 6.4.1 Principal Component Analysis . 146 6.4.2 Low-Rank + Sparse Matrix Decomposition . 148 6.4.3 Restricted Boltzmann machine . 150 6.5 Experiment . 155 6.6 Results . 159 6.7 Using Features for Design . 160 6.7.1 Feature Interpretation of Design Preferences . 161 6.7.2 Features Visualization of Design Preferences . 162 6.8 Summary . 165 VII. Conclusion ............................... 167 7.1 Summary of Dissertation . 167 7.2 Contribution to Design Science . 169 7.2.1 Limitations . 170 7.2.2 Future Work . 173 BIBLIOGRAPHY :::::::::::::::::::::::::::::::: 175 vii LIST OF FIGURES Figure 1.1 Enablers of crowdsourcing as we define in this dissertation; while crowdsourcing as human input aggregation is not new, what is new is the reach and scale we now have to access evaluators and customers who may have potentially valuable input during the early-stage design process. .6 1.2 Taxonomy of crowdsourcing processes as identified for this disserta- tion; most identified properties are not considered within this disser- tation. Grey shaded boxes show properties that were varied within this dissertation. The red shaded box shows the property that was varied and explicitly studied throughout this dissertation, namely, crowd expertise or crowd preferences. .7 1.3 Depiction of the design process from the enterprise standpoint laid out in chronological order. Note that while general for many complex engineering products, this design process was recorded from inter- views with practicing design executives at a major automotive man- ufacturer (Manoogian II, 2013; Hartley, 1996a), and may not gener- alize to all product or service designs. In particular, the partitioning or even existence of various design process steps may be different, as well as the number of major and minor design concepts. Further, note that while technically all major and minor design concepts are unique within the design space, we make the distinction between competing design concepts that are very far apart (major) and those that are small perturbations around a baseline design concept (minor). This distinction assumes some notion of distance within the design space. 10 1.4 Depiction of design process augmented with crowdsourcing system to help designers make better decisions, particularly during design stage-gates. 13 viii 1.5 Selected subset of enterprises spanning industry, governmental agen- cies, and academia, engaged in crowdsourcing. This selection was made to cover enterprises that have had recurring academic and me- dia coverage, as well as a diversity of enterprises exhibiting both successes and failures of crowdsourcing. 15 1.6 Spectrum of heterogeneity for a given objective or subjective design decision. The left-hand end is the case of only a very sparse minority of the crowd having enough expertise for the given design task. In be- tween both extremes are various levels of expertise needed for a given design task. The right-hand end is the case in which by definition no expertise is needed due to individual-level preferences. 30 2.1 Graphical representation of the Bayesian network crowd consensus model. This model describes a crowd of evaluators making evalua- tions rpd that have error from the true score Φd. Each evaluator has an expertise ap and each design has an difficulty dd.