(Pfx): a Prototyping Framework To

Home , Prototype

The Pennsylvania State University

The Graduate School

College of Engineering

PROTOTYPE FOR X (PFX): A PROTOTYPING FRAMEWORK

TO SUPPORT PRODUCT DESIGN

A Dissertation in

Mechanical Engineering

Jessica Dolores Menold

 2017 Jessica Dolores Menold

Submitted in Partial Fulfillment of the Requirements for the Degree of

Doctor of Philosophy

May 2017

The dissertation of Jessica Menold was reviewed and approved* by the following:

Kathryn Jablokow Associate Professor of Engineering Design & Mechanical Engineering Co-Advisor of Dissertation and Co-Chair of Committee

Timothy W. Simpson Paul Morrow Professor of Engineering Design & Manufacturing Co-Advisor of Dissertation and Co-Chair of Committee

Meg Small Director of Social Innovation, Bennet Pierce Prevention Resource Center

Scarlett Miller Assistant Professor of Engineering Design & Industrial Engineering

Aaron Knochel Assistant Professor of Art Education

Karen A. Thole Distinguished Professor of Mechanical Engineering Head of the Department of Mechanical and Nuclear Engineering

*Signatures are on file in the Graduate School

iii

ABSTRACT

Of the estimated 140 billion US dollars spent in new product development by large companies each year, around 40% is wasted on failed products. The largest sunk cost in new product development occurs during prototyping activities. We know prototyping activities are critical to the design process as they translate often fuzzy ideas into physical artefacts, support communication, enhance design development, and aid in decision-making. Engineering design research has failed to provide designers and engineers—practitioners as well as educators—with formal methods or approaches for prototyping to help reduce these losses and increase the likelihood of product success. Instead, designers and engineers must rely on experience, tacit knowledge, and individual judgment to navigate prototyping activities, often resulting in the inefficient use of resources and time. An extensive literature review of prototyping research and a study of novice designers’ perceptions of prototyping are used in this work to develop and validate a set of specifications for a holistic and structured prototyping framework. A novel framework to help structure prototyping, Prototype for X (PFX), is proposed as an alternative to traditional prototyping approaches in engineering design. The PFX framework is composed of three main phases: (1) Frame, (2) Build, and (3) Test. The phases of PFX help designers optimize resources to build prototypes that test core assumptions and inform the design and development new products.

Similar to the “illities” in Design for X, PFX uses lenses to structure and scaffold the prototyping process to make improvements in specific areas. In order to validate the PFX framework, in this work we study the effects of three lenses, namely, Prototype for Desirability, Prototype for Feasibility, and

Prototype for Viability. These lenses are based on Human-Centered Design and Design Thinking frameworks for innovation and innovative products.

In order to evaluate the effectiveness of PFX at improving technical quality, manufacturability, and user satisfaction of end designs, we assess functional prototypes developed in a junior-level mechanical engineering design course. Results from a between-subjects analysis indicate that using

PFX can help increase the desirability, feasibility, and viability of functional prototypes when those

iv lenses are applied; specifically, student teams introduced to PFX produced prototypes that outperformed those from control teams with no formal prototyping methods on user satisfaction, perceived value, and manufacturability metrics. This study confirms the impact that structured prototyping methods like PFX can have on the prototyping process and final designs.

In order to understand the effect of structured and holistic prototyping models on designers themselves, we evaluate the impact PFX has on designers’ prototyping awareness. The prototyping literature has typically evaluated the few prototyping methods, tools, and frameworks using design- based metrics, such as binary evaluations of completion of a design task. Based on a detailed literature review, we hypothesize that structured prototyping methods, specifically PFX, can increase novice designers’ and engineering students’ self-efficacy, leading to an increase in feelings of control throughout the prototyping process, which may lead to an increase in creative output, higher levels of motivation, and an increase in the quality of final designs. As an initial step in the measurement of these outcomes, we sought to understand if PFX influences designers’ prototyping awareness. In order to measure prototyping awareness, a new measurement tool is proposed and validated, the Prototyping

AWareness Scale (PAWS). Results from this study partially support the notion that structured prototyping frameworks influence the prototyping awareness of novice engineering designers.

Finally, the work concludes with a study exploring the effect that PFX might have on the prototyping process itself. We sought to understand how the order or sequence of PFX lenses might affect the feasibility, viability, and desirability of an end design and the prototyping awareness of engineering designers. The results of our findings indicate that the sequence of PFX lenses has some effect on product outcomes and prototyping awareness. The contributions from this research lie primarily in the field of engineering design, specifically in the area of prototyping during the new product design process. They include: 1) establishment of four specifications for a structured and holistic prototyping framework, 2) development of the Prototype for X framework, 3) creation of alternative metrics to evaluate prototypes, 4) creation of a scale to measure prototyping awareness and

5) validation of the effect of a structured prototyping framework on the many facets of a product.

Future work will explore in more detail the full effect of lens sequencing on design and designer outcomes, but this initial work highlights the potential of PFX to be used in new product development to positively influence products and designers. Future work will focus on validating the

PFX framework in industry settings and studying the effects of PFX on the designers’ understanding of prototyping, the decisions made during prototyping, and the artefacts produced during prototyping.

TABLE OF CONTENTS

List of Figures ...... ix

List of Tables ...... xii

Acknowledgements ...... xiii

Chapter 1 Introduction ...... 1

1.1 Motivation ...... 1 1.2 Review of the Literature: Specifications of Prototyping Frameworks ...... 5 1.3 Prototyping as Learning Catalysts in Functional System Design ...... 7 1.4 Prototypes as Milestones in the Optimization of Product Planning ...... 10 1.5 Prototypes as Enablers of User and Designer Engagement ...... 12 1.6 Summary of Specifications and Critical Gaps ...... 16 1.7 Validating the Need for a Holistic and Structured Prototype Framework Using Students’ Perceptions of Prototypes ...... 18 1.7.1 Study Participants and Context ...... 19 1.7.2 Data Collection ...... 19 1.7.3 Qualitative Coding of Open-Ended Responses ...... 20 1.7.4 Summary of Findings and implications for Future Work ...... 22 1.8 Dissertation Outline...... 24

Chapter 2 The Prototype for X Framework ...... 26

2.1 Introduction ...... 26 2.2 Prototype for X: Frame, Build, Test ...... 27 2.2.1 The Frame Phase of PFX ...... 28 2.2.2 The Build Phase of PFX ...... 30 2.2.3 The Test Phase of PFX ...... 32 2.3 Targeted Outcomes of PFX: Drawing from Design for X and Human Centered Design ...... 34 2.3.1 Human Centered Design ...... 34 2.3.2 Design for X ...... 35 2.3.3 Design Thinking ...... 38 2.3.4 Initial Lenses of PFX: Desirability, Feasibility, and Viability ...... 39 2.4 Exploring the Implications of Prototype for X ...... 41 2.4.1 How are end designs affected by a structure and holistic prototyping framework? ...... 42 2.4.2 How is designer’s prototyping awareness affected by a structured and holistic prototyping framework? ...... 43 2.4.3 How does the sequence of PFX lenses affect end designs and prototyping awareness? ...... 44 2.4.4 Summary and Looking Ahead ...... 44

Chapter 3 Assessing the Impact of PFX on End Designs ...... 45

3.1 Introduction ...... 45 3.2 Background in Ideation Metrics ...... 46 3.3 The Translation of Idea Characteristics throughout the Design Process ...... 48

vii

3.4 Selecting Appropriate Metrics for Prototypes: Study Design ...... 50 3.4.1 Participants ...... 51 3.4.2 Procedure ...... 52 3.4.3 Application of Existing Metrics ...... 53 3.4.4 Research Questions, Data Analysis, and Results ...... 55 3.4.5 Discussion and Implications for Evaluations of Prototypes ...... 65 3.4.6 The Case for Alternative Metrics to Evaluate Prototypes...... 67 3.4.7 Desirability Metrics: User Satisfaction and User Perceived Value ...... 68 3.4.8 Feasibility Metric: Technical Quality ...... 69 3.4.9 Viability Metric: Manufacturability ...... 70 3.4.10 Assessment Summary ...... 71 3.5 Research Hypotheses...... 71 3.6 Experimental Protocol ...... 72 3.6.1 Participants ...... 75 3.7 Results ...... 75 3.7.1 User Satisfaction and User Perceived Value ...... 76 3.7.2 Technical Quality Results ...... 77 3.8 Implications ...... 80 3.9 Limitations ...... 81 3.10 A Look Back and a Look Ahead ...... 82

Chapter 4 Assessing the Impact of PFX on Students’ Prototyping Awareness ...... 83

4.1 Introduction ...... 83 4.2 Self-Efficacy and Awareness in Prototyping ...... 83 4.3 Measuring Prototyping Awareness with the Prototyping Awareness Scale (PAWS) 85 4.3.1 PAWS: Desirability Sub-Scale ...... 86 4.3.2 PAWS: Feasibility Sub-Scale ...... 88 4.3.3 PAWS: Viability Sub-Scale ...... 89 4.3.4 Summary of the PAWS ...... 90 4.4 Research Hypotheses...... 91 4.5 Experimental Protocol ...... 91 4.5.1 Participants ...... 93 4.6 Results ...... 93 4.6.1 Within Subjects Experiment ...... 93 4.6.2 Between Subjects Experiment ...... 98 4.7 Implications ...... 99 4.8 Limitations ...... 101 4.9 A Look Back and a Look Ahead ...... 102

Chapter 5 Assessing the Impact of PFX Sequence on Designs and Designers...... 103

5.1 Introduction ...... 103 5.2 Research Hypotheses...... 103 5.3 Experimental Protocol ...... 104 5.3.1 Participants ...... 107 5.3.2 Data Collection ...... 108 5.4 Results: PFX Sequence and Product Outcomes ...... 109 5.4.1 User Satisfaction and Perceived Value ...... 110 5.4.2 Effectiveness ...... 113 5.4.3 Manufacturability Rating ...... 116

viii

5.4.4 Discussion of Results and Limitations ...... 117 5.5 Results: PFX Sequence and Prototyping Awareness ...... 119 5.5.1 Between Subjects Results ...... 120 5.5.2 Within Subjects Analysis ...... 122 5.5.3 Results from Qualitative Data Set...... 130 5.5.4 Discussion of Results and Limitations ...... 143 5.6 Implications ...... 146 5.7 A Look Back and a Look Ahead ...... 148

Chapter 6 Contributions and Future Work ...... 149

6.1 Summary and Conclusions ...... 149 6.2 Research Contributions ...... 151 6.3 Limitations and Shortcomings ...... 154 6.4 Areas for Future Work ...... 155 6.4.1 Prototype for X: Translation to Industry and Beyond...... 155 6.4.2 Prototype for X: Development of Appropriate and Rigorous Metrics for Assessing Prototyping ...... 157

References...... 158

Appendix A Qualitative Coding Tags from Perceptions of Prototyping Study ...... 178

Appendix B Build for X Handouts ...... 179

Appendix C Test for X Handouts ...... 182

Appendix D Description of Project Distributed to Students at the Start of the Semester 185

Appendix E Prototyping AWareness Scale ...... 187

LIST OF FIGURES

Figure 1-1. Dyson’s (a) early prototypes, to (b) the full line of final products...... 2

Figure 1-2. Design tools are enabling designers to more quickly go from (a) prototypes [14], to (b) final products [15]...... 3

Figure 1-3. Examples of successful free standing towers from Neeley, et al [44]...... 8

Figure 1-4. Example designs from a prototyping task [32]...... 9

Figure 1-5. Houde and Hill’s model for prototypes and design artefacts [99]...... 13

Figure 1-6. Nielson’s prototype space and the three categories of low fidelity prototypes. . 14

Figure 1-7. Nielson’s prototype space and the three categories of low fidelity prototypes. . 22

Figure 2-3. PFX structure and integration into engineering design process [79]...... 28

Figure 2-4. Example of the Frame Phase of PFX from Case Study One ...... 30

Figure 2-3. The model for innovative product development proposed in Human-Centered Design; solutions that emerge should hit the overlap of these three lenses, making them desirable, feasible, and viable...... 35

Figure 2-4. An example of a Design for Manufacture framework is highlighted showing the decisions and considerations taken while using DFM methods [123]...... 36

Figure 3-1. Overview of the study conducted over eight weeks during a junior-level mechanical engineering design course...... 51

Figure 3-2. The distribution of originality scores at Times 1, 2, and 3...... 59

Figure 3-3. The distribution of effectiveness scores at Times 1, 2, and 3...... 62

Figure 3-4. A selection of student work highlighting the evolution of one concept from Time 1 to the Final Prototype...... 66

Figure 3-5. Overview of Experiment Flow ...... 72

Figure 3-6. Example Prototypes from the (A) Control and (B) Experimental Classes ...... 73

Figure 3-7. Example of Prototype for Feasibility Process ...... 74

Figure 3-8. Distributions of User Satisfaction Scores in Control and Experimental Groups 76

Figure 3-9. Distribution of Perceived Value Scores in Experimental and Control Groups .. 77

Figure 3-10. Distribution of Amount of Rice Collected by Final Designs in Lbs...... 78

Figure 3-11. Distribution of Critical Part Count Ratio for Final Designs ...... 80

Figure 4-1. Overview of Experiment Flow ...... 92

Figure 4-2. Mean Desirability Subscale Score at T1, T2, T3, and T4 ...... 95

Figure 4-3. Mean Feasibility Subscale Score at T1, T2, T3, and T4 ...... 95

Figure 4-4. Mean Viability Subscale Score at T1, T2, T3, and T4 ...... 96

Figure 5-1. Timeline and data collected in each sample ...... 106

Figure 5-2. Distributions of User Satisfaction Scores in Samples A, B, and C ...... 111

Figure 5-3. Distribution of Perceived Value Scores in Samples A, B, and C ...... 112

Figure 5-4. Distribution of Effectiveness Ratings in Samples A, B, and C ...... 114

Figure 5-5. An Example of Components that Should not have been 3D Printed ...... 115

Figure 5-6. Distribution of Critical Print Ratio in Samples A, B, and C ...... 115

Figure 5-7. Distribution of Critical Part Count Ratio in Samples A, B, and C ...... 116

Figure 5-8. Distribution of Desirability Prototyping Awareness Scores for Sample A ...... 124

Figure 5-9. Distribution of Feasibility Prototyping Awareness Scores for Sample A ...... 124

Figure 5-10. Distribution of Viability Prototyping Awareness Scores for Sample A...... 125

Figure 5-11. Distribution of Desirability Prototyping Awareness Scores for Sample B ...... 126

Figure 5-12. Distribution of Feasibility Prototyping Awareness Scores for Sample B ...... 126

Figure 5-13. Distribution of Viability Prototyping Awareness Scores for Sample B ...... 127

Figure 5-14. Distribution of Desirability Prototyping Awareness Scores for Sample C ...... 128

Figure 5-15. Distribution of Feasibility Prototyping Awareness Scores for Sample ...... 129

Figure 5-16. Distribution of Viability Prototyping Awareness Scores for Sample C ...... 129

Figure 5-17. Distribution of Audience Tags at Time 2 ...... 133

Figure 5-18. Distribution of Audience Tags at Time 3 ...... 133

Figure 5-19. Distribution of Purpose Tags at Time 2 ...... 136

Figure 5-20. Distribution of Purpose Tags at Time 3 ...... 136

Figure 5-21. Distribution of Method Tags at Time 2 ...... 139

Figure 5-22. Distribution of Method Tags at Time 3 ...... 139

Figure 5-23. Distribution of Evolution Tags at Time 2 ...... 142

Figure 5-24. Distribution of Evolution Tags at Time 3 ...... 142

xii

LIST OF TABLES

Table 1-1. Sources Included in Literature Review ...... 6

Table 1-2. Four Specifications for a Holistic and Structured Prototyping Framework...... 18

Table 1-3. Categories, Their Definitions, Associated Tags, and Example Responses...... 22

Table 3-1. Correlation Matrix for Originality Scores...... 59

Table 3-2. Correlation Matrix for Originality Scores...... 62

Table 3-5. Correlation Matrix for Originality Scores...... 71

Table 4-1. Summary of Items for Desirability Sub-Scale ...... 88

Table 4-2. Summary of Items for Feasibility Sub-Scale ...... 89

Table 4-3. Summary of Items for Viability Sub-Scale ...... 90

Table 5-1. Data Collected throughout Experiment ...... 107

Table 5-2. Significant difference observed across items of PERVAL Scale (bold and italics indicate significant difference observed) ...... 112

Table 5-3. Category and tags with Descriptions and Example from Qualitative Findings ... 131

xiii

ACKNOWLEDGEMENTS

I would first like to thank my dissertation advisers, Dr. Simpson and Dr. Jablokow. They have both been incredible sources of support and knowledge and have pushed me to pursue new areas both in and out of research. I can honestly say that without them I would not have started two companies, fallen in love with teaching, or found the passion to pursue an academic career. Their insights throughout my graduate career have helped me to push through many problems that I have encountered in my work. I am also very grateful for the patience that each has shown when reviewing and editing my writing, specifically when it comes to grammar. Although I will never fully understand what a split infinitive is, I am grateful for their unwavering persistence to teach me the fundamentals of third grade grammar.

I am grateful for an incredible committee, many of whom have provided me with insights and new perspectives about my research. Dr. Small, Dr. Miller, and Dr. Knochel, thank you for serving on my committee and being willing to read my work. Dr. Miller I want to thank you for the many fascinating courses you have taught, they have been wonderful sources of inspiration. Dr. Small I will never be able to thank you enough for creating the HUDDIL lab, a place where engineers and social scientists can work to translate social interventions to products, in order to create a bigger impact. I also want to thank you for your unwavering faith in CurioSpace, without you I think our little startup would have sunk by now, but your positive attitude and creativity have helped me to keep chugging along.

I am extremely grateful for the fantastic network of friends and family that have supported me over the years. I have found an incredible home here in State College thanks to places like New Leaf

Initiative, the Launch Box, and Innoblue. Lee Erickson and Liz Kisenwether have been magnificent mentors and have helped me develop business plans, products, and research ideas. I also want to thank

Carlye Lauff who has been the best partner in crime, co-founder, and friend I could have asked for. My family has been the biggest and best source of support throughout my time at Penn State. I am so happy my father decided to teach me CAD at age ten and always encouraged me to take things apart (even

xiv though, most of the time, I could not put them back together). My sister and best friend, Elizabeth, has inspired me with her passion and intelligence. Lastly, I want to thank Shane Haydt, who has been an incredible roommate and best friend throughout my graduate school experience. I do not know where my research would be without the countless Ru Paul breaks and exciting games of slap cup.

Chapter 1

Introduction

1.1 Motivation

Each year, $141.8 billion is invested by large companies in product research and design activities [1]. Studies indicate that anywhere from 40-50% of that money is spent on cancelled products or those that yield inadequate results. Prototyping represents the largest sunk cost of the product development process; however, it remains the least researched or understood [2]. As Wall states,

“prototyping is one of the most critical activities in new product development” [3]. Prototypes ranging from low fidelity (simple physical models) to high fidelity (fully functioning devices or systems) are used throughout the design process to communicate ideas [4], gather user feedback [5,6], explore parallel design concepts [7,8], and make decisions [9,10]. Dyson created approximately 5,127 prototypes over the course of five years (see Figure 1-1) , culminating in the creation of the Dyson cyclone arguably one of the most successful vacuums on the market [11].

Ulrich and Eppinger define a prototype as “an approximation of the product along one or more dimension of interest” [12]. Houde and Hill define prototype more broadly as a tool that should be catered to your audience in order to gather deeper insights about that audience. Gerber defines a low fidelity prototype “as minimally detailed physical manifestation of an idea” [13]. In this work, we use the broadest and most applicable definition of a prototype from Yang et. al [14]: a prototype is an early, physical embodiment of the design. In this work we also limit the scope of our research to physical prototypes of physical end products.

(a) (b)

Figure 1-1. Dyson’s (a) early prototypes, to (b) the full line of final products.

Until recently, the creation of so many prototypes would seem infeasible to most designers, startups, or established companies, but with the increase in availability of rapid prototyping tools, such as Little Bits [15], 3D printers, and a host of other gadgets, consumers are able to more quickly and cheaply create physical models of their concepts. Ringly, a startup backed by some of the top VC firms in Silicon Valley, sells high fashion jewelry that syncs with your phone through a blue tooth connection, discreetly alerting its user when a call, text, or other notification comes through. The founder used Little

Bits and 3D printed components to create the first prototype, which led to a $1 million seed round in fundraising; the company has recently raised over $6.1 million and is selling its products around the world (see Figure 1-2) [15]. Because the founder was able to rapidly prototype and iterate on her idea, the concept was more easily communicated to customers, stake holders, and design team members.

Prototypes are clearly critical artefacts in the design process, helping designers and design teams progress towards a finalized product, but as Camburn summarizes, “prototyping may be simultaneously one of the most important and least formally explored areas of design” [16] .

(a) (b)

Figure 1-2. Design tools are enabling designers to more quickly go from (a) prototypes [17], to (b) final products [18].

Of particular interest in prototyping, research is the creation and validation of a structured prototyping framework that incorporates previous research findings and best practices from industry into a cohesive strategy. We know from previous work that structured prototyping methods can have enormous benefits for individual designers [19,20], design teams [21,22], and end products [16,23,24].

Two prototyping frameworks were recently proposed by Christie et al. [19] and Camburn et al. [16] based on literature reviews from engineering and management science, respectively. Camburn et al. documented prototyping strategies specifically related to engineering during product development, while Christie et al. explored prototyping strategies related to business and engineering actions, where prototyping strategies were defined as “the set of decisions that dictate what actions will be taken to accomplish the development of the prototype(s)” [19]. Specifically, Christie et al. [19] described nine factors and suggested thirteen questions to consider while selecting prototyping strategies and making prototyping decisions; these factors and questions focus on taking a design from concept to reality. Five of the nine factors address resource management, and the remaining four address the prototype’s functionality; in essence, the factors focus on the viability and feasibility of the prototype, respectively.

The purpose of Christie et al.’s thirteen prototyping questions, then, is to determine which factors are the most relevant to the prototype in question. Christie et al. reported on early results from the implementation of this prototyping framework in a senior-level engineering design course, stating that

4 teams who used the framework appeared more confident during the prototyping process and had a clearer idea of design tasks at each step.

Camburn et al. [16] divided the determination of a prototyping strategy into four distinct phases; they suggested that after working through these four phases, design teams will have a set strategy to guide the prototyping process. The effectiveness of these measures to increase the efficiency of the prototyping process was tested by Dunlap and Camburn [23] through an experimental study of a basic design challenge. The challenge required each design team to move a small object (a coin) to a target without using human energy over a period of three hours. The researchers evaluated the feasibility (e.g., Did the object reach the target?) and viability (e.g., Did the prototype utilize the allocated resources?) of each end design; they found increased success among prototypes developed by teams who used their four-phase approach to building a prototyping strategy.

Camburn et al.’s and Christie et al.’s contributions highlight the potential for prototyping frameworks to decrease uncertainty and increase effectiveness during the design process, but they are not complete. For instance, these two frameworks focus prototyping efforts almost exclusively on either technical development or resource management, while ignoring factors related to user satisfaction and perceived value—important considerations in the product design process, as noted in the previous sections. Without the incorporation of user-centered design practices, these frameworks will be difficult to adapt to other necessary aspects of new product development, such as testing the usability of a new product or gauging market interest. A more comprehensive and holistic framework is still needed that can be generalized to real-world design challenges. More than technical quality and effective resource management determines the success of a product in the market, and factors such as user satisfaction, user-perceived value, and manufacturability are considered.

To address this issue, we introduce a novel, holistic framework for prototyping called

Prototype for X (PFX) that helps bridge the gap between research and practice by providing designers with a structured set of methods for prototyping activities (see Chapter 2). While previous prototyping frameworks exist, we identify two critical gaps in these frameworks, namely: (1) failure to incorporate

5 user-centered design practices, and (2) failure to evaluate their impact in realistic design settings. This dissertation describes the overarching specifications that guided the development of PFX to address these gaps. While we believe that PFX could improve outcomes for designers in industry, initial testing of the framework was done using novice designers in an educational setting. The results of our studies do not generalize to professional settings, and future work will focus on implementing PFX in professional settings to evaluate the framework’s effectiveness. Early results from the implementation of PFX in this educational setting indicate that this framework can positively impact both product quality [25] and prototyping awareness [26]. Our work has the potential to reduce the number of failed products, increase the effectiveness and efficiency of design teams by reducing the amount of wasted time, effort, and resources spent during prototyping activities, and increase the overall desirability of products.

In the remainder of this chapter 2 a literature review of the existing prototyping research is undertaken as background for the dissertation, after which specifications for a prototyping framework are derived. Following this, we present the results from an exploratory pilot study in which we evaluated students’ perceptions of prototypes. We conclude this chapter by reviewing research objectives and outlining the remainder of the dissertation.

1.2 Review of the Literature: Specifications of Prototyping Frameworks

In the first stage of our work, we conducted a systematic review of prototyping research in the management science, engineering management, industrial engineering, engineering design, engineering education, and human computer interaction literature was conducted, with an eye toward the state-of-the-art in terms of prototyping frameworks. We examined 80 articles from relevant engineering, design, and management conferences, journals, and texts, such as Management Science,

Design Studies, Journal of Mechanical Design, and the proceedings of ASME’s International

Conference on Design Theory and Methodology, among others. To locate appropriate articles for

6 consideration, we searched for terms such as “prototyping”, “physical models”, “prototyping strategies”, and “prototyping frameworks” under the topics of engineering design, engineering management, and innovation management. Table 1 provides a summary breakdown of articles and sources and highlights the top ten most frequently cited sources in our literature review. The 22 articles categorized as “other” came from less frequently cited journals, texts, or conferences, while still reflecting the four overarching categories previously mentioned.

Table 1-1. Sources Included in Literature Review

# Abbrev. Journal/Conference/Text # of Articles Citations 1 MS Management Science 8 [10,27–34] 2 DS Design Studies 8 [14,35–41]

International Conference ASME on Design Theory and 3 IDETC Methodology 8 [16,24,42–47] Conference on Human Factors in Computing 4 CHI Systems 7 [5,48–53] American Society of Engineering Education 5 ASEE Conference 7 [54–60]

Journal of Mechanical 6 JMD Design 6 [2,61–65] 7 DTR Design Thinking Research 5 [66–70] 8 JM Journal of Marketing 3 [71–73] Research in Engineering 9 RED Design 3 [3,74,75] Journal of Creative 10 JCB Behavior 2 [76,77] Other [10 journals, 2 books, 11 conference proceedings] 23 [4,12,20–22,78–96] TOTAL 80

Based on our review, the extant prototyping literature is predominantly focused on technical function and resource management, with a secondary focus on user interactions and prototype development. As a result of our review, we identified three key functions of prototypes, which led to the further identification of four specifications that should be met by any prototyping framework (see

Section 1.6). In this context, we use the definition of design specifications from Ulrich and Eppinger

[12], who define specifications as “the requirements a product must fulfill that enable the designer to define and understand the problem he/she is trying to solve”. In this respect, we have used the specifications identified here to understand more thoroughly the need for a holistic prototyping framework.

The three major functions of prototypes and prototyping frameworks we identified from our literature review are: 1) prototypes catalyze learning during functional subsystem design, 2) prototypes act as decision variables in the optimization of product planning and development, and 3) prototypes enable richer and deeper engagements between designers and end users.

1.3 Prototyping as Learning Catalysts in Functional System Design

Prototypes aid designers by catalyzing their learning during the technical subsystem development of new products. Jang and Schunn [64] found that capstone design teams who created and used physical models or prototypes on a more consistent and frequent basis significantly outperformed

8 their peers with respect to technical function in a senior-level mechanical engineering course. Team success was measured in terms of meeting client requirements; the authors demonstrated that prototypes enabled richer team discussions of design concepts and gave teams a more realistic understanding of technical and functional constraints [64]. Similarly, Neeley, et al. [46] found that the functionality of end designs was significantly improved when designers created more prototypes. In Neeley et al.’s experiment participants were tasked with building a freestanding cardboard structure using only an

8.5”x11” sheet of corrugated cardboard (see Figure 1-3). In their experimental design, the experimental group had to build four more prototypes than their control group; although designer satisfaction levels were lower in the experimental group, their final designs outperformed the control group [46].

Figure 1-3. Examples of successful free standing towers from Neeley, et al [46].

Similarly, Elsen et al. [44] found that teams of graduate and professional engineers produced end designs of significantly better technical quality when they produced prototypes earlier in the design process. The teams that prototyped sooner were able to identify problem areas in their designs earlier and had more time to iterate and build better systems to achieve core functions [44]. Likewise, Yang

[14] observed that earlier, simpler prototypes with fewer parts outperformed more complex prototypes; she hypothesized that by testing the simplest core function, teams were able to increase the technical quality of their designs. In a similar study, Lemons et al. [35] asked engineering students to prototype a one-handed jar opener using the materials provided. In the experiment, the researchers did not

9 specifically instruct students to build a prototype (See Figure 1-4), in order to evaluate if students would be naturally inclined to build. The researchers found that all students built models and confirmed

Yang’s previous hypothesis, stating that “physical construction of a model during an open-ended design task helped students generate and evaluate ideas, better visualize their ideas, and uncover differences between real behavior and the conceptual model used to predict that behavior” [35]. Students in the study felt that the development of their prototypes were critical in the demonstration of the ideas viability and the physical models helped them uncover flaws in their sketches and ideas.

Figure 1-4. Example designs from a prototyping task [35].

A common theme in these studies is prototypes serving as learning opportunities for designers to understand and advance the design and understanding of core critical functions. The research shows that the development of more prototypes often leads to more technically sound final designs, as designers are able to quickly identify problems and pivot their concepts accordingly. This finding aligns with Schrage’s work [93], which shows that breakthroughs made by engineering designers are dependent upon their ability to experiment and test concepts. From this portion of our literature review, we identify the first specification for prototyping frameworks, including PFX:

Specification #1: A prototyping framework should encourage iterative prototype development early and often in the design process to increase the overall quality of the final design.

1.4 Prototypes as Milestones in the Optimization of Product Planning

Second, prototypes also serve as milestones during new product development, around which companies and design teams can effectively plan product development based on time and budget constraints. Smith and Eppinger [10] present a quantitative interpretation of the Design Structure Matrix

(DSM) that optimizes the sequence of design tasks to reduce development time. Smith and Eppinger’s model uses task time and probability of iterations dependent on task outcomes to structure a design process that reduces development time. Although Smith and Eppinger created this modified DSM for the entirety of the design process, it is particularly pertinent to prototyping, because it emphasizes the importance of iterative testing and feedback loops within the design process and lays the groundwork for more specific research in prototype planning and path optimization [10].

Thomke [29,30] built on Smith and Eppinger’s work and focused exclusively on experimenting with “modes” of prototypes, i.e., prototypes developed via computer simulation or rapid prototyping methods. Thomke showed that switching among a variety of prototyping modes at optimal times could reduce overall product development costs and time [30], leading to higher experimental yields (i.e., errors identified in the design and fixed during redesign). Thomke’s use of prototypes as milestones for testing and analysis frames the decision to build and test prototypes as the result of comparing experimental yield, engineering effort, and time needed to complete the proposed prototype [29].

Similar to Thomke, Wall, et al. [3] developed a systematic method of evaluating prototyping processes in order to determine the best process for a given product development context. Specifically, they characterized prototyping processes along three dimensions: (1) part performance, (2) unit cost, and (3) lead time. Part performance is modeled, as a vector that evaluates the fidelity of the prototyped parts with respect to the properties those parts would have in final production processes; unit cost and lead-time are modeled as scalars that represent the estimates of cost and time, respectively, to create prototype parts. Wall et al.’s resulting model selects the best prototype process for a given lot size; however, their work is exclusive to electromechanical design and is also limited to four prototyping

11 processes: (1) computer aided design, (2) stereo lithography, (3) computer numerically controlled machining, and (4) rubber molding [3].

McCurdy et al. [51] explored the prototyping process within the field of human computer interaction from a similar perspective, i.e., how to optimize prototype output based on time and budget constraints. They identified five dimensions of a prototype: (1) level of visual refinement, (2) breadth of functionality, (3) depth of functionality, (4) richness of interactivity, and (5) richness of data model.

They argued that a prototype could have a variable level of fidelity ranging from low to high on each dimension, and that optimal prototypes would have mixed levels of fidelity, with lower fidelity in less critical dimensions and higher fidelity in more critical dimensions. While this work adds value to the literature dedicated to planning and optimizing prototyping practices, it needs to be generalized to other domains as it is currently only applicable to digital prototypes [51].

The research studies described above and others [27,28,97], all describe models for decision- making and planning during the prototyping stage of new product development. From this research, we conclude that prototypes act as milestones that set the objective or goal of the design team. Existing work emphasizes the use of prototypes to set the objective of the design team: to minimize time, cost, or effort or maximize quality, value, or innovativeness. It is important to note that the aforementioned works reviewed are predominantly concerned with decision-making during prototyping activities, and offer little guidance for the actual building and development of physical prototypes. In addition, while these models are useful, they are often intensive optimization problems; we propose distilling the main theme in each model into more general heuristics that could be implemented more rapidly with less effort. Each model uses prototypes as an objective around which to manipulate the remaining constraints and variables of the design process; we hypothesize that a prototyping method that encourages designers to structure an effective prototyping path and build prototypes that test end production processes could help design teams plan more effectively for future prototypes. Drawing from this line of research, we identify a second specification for prototyping frameworks, including

PFX:

Specification #2: A prototyping framework should enable design teams to quickly select a prototyping focus tailored around specific design goals, to support effective decision-making in the design process.

1.5 Prototypes as Enablers of User and Designer Engagement

Our review of the literature also shows that prototypes enable richer communication between users, designers and other stakeholders, providing designers with tools to gather important insights and feedback about a product [5,45,98]. A variety of articles within the Human Computer Interaction (HCI), interaction design, and participatory design literature have explored the impact of prototype fidelity on the usability and perceived usefulness of the product, as well as the quality of interactions that occur between the designer and users when gathering feedback [71,72,99]. Houde and Hill proposed an entirely new perspective on prototypes in What do Prototypes Prototype [100], suggesting that practitioners begin prototyping by asking three fundamental questions about the interactive system being designed: 1) what role will the artefact play in the user’s life, 2) how should it look and feel, and

3) how should it be implemented. They proposed a model that describes a prototype in terms of the artefact being designed (see Figure 1-5) and use four categories to describe any prototype and three of these categories are characterized by interactions with the user: role, look and feel, and integration.

Role prototypes are “those which are built primarily to investigate questions of what a prototype could do for a user”. Look and feel prototypes are “built primarily to explore and demonstrate options for the concrete experience of an artifact”. Integration prototypes are “built to represent the complete user experience of an artefact”. Houde and Hill go on to provide practitioners with recommendations for practice, such as knowing your audience and preparing a prototype specifically for this audience.

Figure 1-5. Houde and Hill’s model for prototypes and design artefacts [100].

Nielson [101] argued that prototype fidelity is a critical factor when evaluating prototypes with users, stating “prototype fidelity is a measure of how authentic or realistic a prototype appears to the user when compared to the actual service”. He makes the case that low fidelity prototypes are better suited for the early stages of the design process as they are able to provide designers with rich user feedback while minimizing the burden of prototype creation. Nielson goes on to characterize low fidelity prototypes as either horizontal prototypes, vertical prototypes, or scenarios; horizontal prototypes “represent all of the features but at a lower level of functionality”, and vertical prototypes

“reduce the number of features but fully implement those chosen so that only a part of the system is available”. Scenarios are a combination of both vertical and horizontal prototypes, reducing both the number of features and the level of functionality so that only a planned path of use is simulated (see

Figure 1-6).

Figure 1-6. Nielson’s prototype space and the three categories of low fidelity prototypes.

Other researchers in the field of HCI have explored the relationship between fidelity and users more closely. For example, Sonderegger [102] found that users’ emotions were positively affected by the operation of higher fidelity digital prototypes; however, task completion time was more accurately recorded and represented when users interacted with lower fidelity paper prototypes. Westbrook [72] also noted that users gave richer, higher quality feedback about the navigation of a design when interacting with a lower fidelity paper prototype, but these same users gave richer feedback about the overall structure and aesthetics of the design when interacting with the higher fidelity digital prototype.

Similarly, Tohidi, et al. [52] found a significant interaction between prototype format and user response across user groups. Specifically, they found that differences in the media type of the prototype

(physical story-boards or videos) led to differences in the interactions between users and prototypes.

Users in the story-board group were more likely to use the prototype in their responses as a tool for communicating thoughts and insights about the overall design, leading to higher quality feedback.

Other researchers within the field of HCI have praised low fidelity prototypes (as compared to higher fidelity prototypes) for their ability to provide design teams with more critical feedback from users, helping designers identify problems throughout the design process more quickly [50,82,103].

Researchers have also evaluated the effects that prototypes have on interactions among design team members. For example, Dow, et al. [7] found that sharing multiple prototypes within a design team improved design outcomes, group communication, and group rapport. They also found that simply creating multiple prototypes was not enough to impact design outcomes; getting feedback on prototypes was necessary to improve the quality of the final designs. These researchers also found that teams that developed and shared multiple prototypes had a more equitable distribution of speaking turns and integrated partner’s concepts into their own designs more often. Gerber [20] found that creating low fidelity prototypes allowed design teams to reframe failures as opportunity and encouraged a sense of forward progress among team members. Gerber studied 35 professional designers in practice using

15 ethnographic research methods and found that designers who engaged in low fidelity prototyping activities allowed designers to “quickly produce visible results that were validated by others leading to a sense of perceived control”. As a result, Gerber argues that designers felt an increase in design self- efficacy and experienced higher levels of motivation to act.

Empathic experience design (i.e., prototyping the entire user experiences) has been shown to positively affect design teams’ mental models about final products [104]. In empathic experience design, design team members “become” lead users who immerse themselves in product prototypes to observe and interact with the product from the perspective of the end user [45]. Often the empathic experiences are simulated by allowing designers to interact with a prototype of the product and creating an environment that replicates the experience of a lead user, for example, dimming the lights to mimic the experience of a visually impaired user [45]. This specific type of prototyping has been shown to improve design team members’ discovery and understanding of latent user needs, leading to improved designs and final products [104].

In combination, these works demonstrate that prototypes act as boundary objects for design teams and help ensure that team members have similar conceptions about the design itself, thereby serving as a communication tool to enhance the quality of interactions between end users and designers.

Henderson [105] describes boundary objects as “agents that socially organize cognition”. The majority of the work we reviewed that evaluated user interaction was exclusive to the development of digital products; however; a full exploration of how prototypes of physical products might enhance user feedback and communication between design teams and end users remains a gap in the current literature. We know that the development of prototypes improves a design team’s understanding about the design itself [106], improves group rapport [7], and in some cases, increases creativity levels and divergence in ideation [20,62]. From the foregoing research, we identify two more specifications for prototyping frameworks:

Specification #3: A prototyping framework should enable the development of prototypes that engage users and the design team in order to maximize insights about the product or design.

Specification #4: A prototyping framework should be flexible concerning prototype constraints (i.e., material, fidelity, and mode), encouraging a prototype appropriate for the level of interaction or engagement desired by the design team.

1.6 Summary of Specifications and Critical Gaps

Our literature review revealed critical gaps in existing prototyping research. First, we noticed that existing prototyping frameworks fail to incorporate user-centered design practices. We saw this trend extend to prototyping research, which predominately focused on increasing the technical quality of prototypes but failed to evaluate the desirability, usability, or general appeal of prototypes. We propose using Human-Centered Design’s (HCD’s) three-lens model of innovation in conjunction with a structured prototyping framework to address this gap. In HCD’s model, innovation occurs when a product is desirable, feasible, and viable. The desirability lens asks questions about the product such as “will users want this product”, the feasibility lens asks questions about the product such as “is this product technically and organizationally possible”, and the viability lens asks questions about the product such as “is the creation and distribution of this product economically viable” [121]. These lenses are redefined in the context of prototyping and a new prototyping framework introduced in

Chapter 2.

We also noticed a tendency among design researchers to oversimplify the design tasks used in their prototyping studies. Oftentimes, the complexity of the design tasks used in prototyping research does not mimic real-world design problems and challenges, making it difficult to generalize findings beyond the specific study conditions. For example, Camburn et al. [16], Yang et al. [14], and Neely et al. [46] all evaluated the technical success of prototypes using a simple binary metric for completion of

17 the design task (e.g., the successful retrieval and placement of a flag, the ability of a simple mechanism to hit a target, or building a cardboard structure that was as tall as possible). In each case, the design task was simplified to reduce the burden of analysis and comparison between designs across control and experimental groups. This simplification is understandable and convenient from a research standpoint, but it makes it difficult to generalize results to address the more complex design challenges often encountered in the real world.

Finally, we realized that a structured and holistic prototyping framework does not currently exist that integrates all four specifications. Table 1-2 summarizes the four specifications for a prototyping framework derived from our literature review. In closing this section, we note once again how these specifications are being addressed in part (although not always explicitly) across the domain of prototyping research. From a theoretical point of view, a holistic and structured framework for prototyping that incorporates all of these specifications and addresses the aforementioned gaps mentioned above is needed to ensure that prototyping is taught effectively in academic settings and practiced effectively in professional settings.

Table 1-2. Four Specifications for a Holistic and Structured Prototyping Framework.

Specifications Evidence from Literature See For Example 1. A prototyping framework should encourage -Prototypes provide design teams [14,19,107] iterative prototype development early and often with learning opportunities in the design process to increase the overall -More prototypes lead to an quality of the final design. increase in technical quality -Prototyping earlier leads to higher quality designs 2. A prototyping framework should enable design -Prototypes provide critical [27,28,97] teams to quickly select a prototyping focus feedback during decision-making tailored around specific design goals, to support in engineering design effective decision-making in the design process. -Existing prototyping optimization is far too time intensive for realistic implementation into industry 3. A prototyping framework should enable the -User feedback can lead to critical [7,13] development of prototypes that engage users and improvements in design the design team in order to maximize insights -Prototypes can enhance about the product or design. communication amongst design team members 4. A prototyping framework should be flexible with -Prototype fidelity is dependent [29,78,83] regards to prototype constraints (i.e. material, upon the level of interaction fidelity, mode), encouraging a prototype needed appropriate for the level of interaction or -Prototype fidelity, material, and engagement desired by the design team. mode directly impact user feedback

1.7 Validating the Need for a Holistic and Structured Prototype Framework Using Students’ Perceptions of Prototypes

To identify themes potentially not represented by the literature and to get a “pulse” of students’

prototyping perceptions, we distributed a simple open-ended question in a pilot study using juniors in

mechanical engineering. The goal of this pilot study was to identify themes or gaps in novice designers’

understanding that a follow on study could dive deeper into. Specifically, we were concerned with

answering the following research question: what general perceptions about prototyping do novice

designers hold? In this work we present surface level findings from our initial data set, that indicate the

need for more in depth research exploring the many layers of prototyping perceptions held by students.

Our initial study resulted in the creation of a coding scheme to be used in future research and

highlighted several content areas to develop survey questions around so that a deeper understanding

19 of prototyping perceptions can be gained. In the following sections, we describe the study participants, data collection, data analysis, and implications of our findings.

1.7.1 Study Participants and Context

The study participants were comprised of 194 students in a junior-level mechanical engineering design course at the Pennsylvania State University. In terms of demographics, 11% of the students were female, and 89% were male; ages ranged from 19 to 21, with a mean age of 20. The course introduces engineering design to mechanical engineering students and focuses on one design project for the entire semester. During the semester in question, the course project required students to create a new product in the field of engineering education utilizing 3D printing technology and following the design process outlined in [12]. Students were taught the fundamentals of 3D printing through a variety of lectures and hands-on activities. Students worked in teams of three or four, which were assigned via CATME, or the Comprehensive Assessment of Team-Member Effectiveness, a widely used online teaming tool

[108].

1.7.2 Data Collection

Because this was an informatory pilot study intended to gather general insights from novice designers regarding prototyping perceptions, a single open-ended question was posed to students during the third week of the semester: How would you define prototyping in your own words? This question was asked in order to get the “pulse” of students’ perceptions towards prototyping. We argue that the perceptions about prototyping most prominent in student’s minds would stand out in their definitions.

Because there has been no prior work exploring the prototyping perceptions of novice designers we used this question as a generative exercise to create more specific research questions, identify themes not present in the literature, and develop a more rigorous scale for future studies. We would like to note

20 that this is a pilot study with the intention of gathering general insights into the prototyping perceptions of students.

Students had no lectures or readings on prototyping theory within the course prior to distribution of the scale. All students had previously taken a freshmen-level design course, but that course provides only a general overview of design and does not cover prototyping in depth. One screening question (How many prototypes have you created prior to this course?) was also posed to gauge students’ previous prototyping experience. In response to this question, 20% of the students indicated they had never created a prototype, 45% indicated they had created fewer than 2 prototypes,

25% indicated they had created fewer than 4 prototypes, and the remaining 10% indicated they had created more than 5 prototypes. We allowed students to freely interpret the definition of “a prototype” in the screening question. This free interpretation can be seen as a limitation of our study since students might interpret “prototype” in many different ways (e.g., a physical prototype, a throw away prototype, a software model, an alpha prototype, etc.), and these diverse views will affect the students’ responses.

Nevertheless, the nature of our open-ended question had students clarify their views and definitions of prototyping, which we believe helped mitigate these differences to some extent. Future studies will explore the relationship between self-assessed prototyping experience and the diversity of prototype definitions, as well as the impact of setting bounds and context on prototype definitions (i.e. physical vs. digital, created within a university course vs. outside). For this pilot study, however, we were mainly concerned with generating an initial coding scheme, and therefore, we evaluated the set of responses as one sample, regardless of previous prototyping experience.

1.7.3 Qualitative Coding of Open-Ended Responses

Qualitative content analysis [109,110] was used to analyze the open-ended responses from the survey. The response set (194 total responses) was read carefully by two independent coders, and segments of text that described distinct purposes and functions for prototypes were highlighted;

21 appropriate tags were assigned to these segments. In the open coding of all students’ responses, 21 tags were identified (e.g., “size”, “cost”, “early”); a full list of the tags is provided in Appendix A. Each student response was then tagged, using as many tags as necessary to describe the response. Once all responses had been tagged, the results were examined for recurring combinations of two or more tags.

These combinations were used to generate a smaller set of five categories based on the study participants’ understanding of prototyping; inter-rater reliability was found to be .823. For example, the tag “ideas” appears in several categories, but this tag alone was not sufficient to sort a response into this category; instead, sorting was done based on the combination of all tags assigned to the response.

Each category corresponded to a function of prototyping in terms of modeling: (1) Model to

Link, (2) Model to Test, (3) Model to Communicate, (4) Model to Decide, and (5) Model to Interact.

Each of these categories map to the four specifications derived from literature in some way; this mapping is discussed in section 1.7.4. Definitions and an example from each category are presented in

Table 3. Once the five categories were finalized, category definitions and two examples from each category were given to two independent raters, who analyzed the response set independently and assigned a single code to each response that corresponded to the category that best represented the meaning, word choice, and tone of the response. Inter-rater reliability was found to be acceptable at

0.81. The frequency of each category code in the response set is illustrated in Figure 1-7.

Table 1-3. Categories, Their Definitions, Associated Tags, and Example Responses.

Category Category Definition Tags Example Response Model To Link The act of physically modelling or Visualize, Ideas, [A prototype is] A link visualizing an idea(s) that links the Link, Final Product between initial ideas and the idea(s) to the final product (turns final product concepts into concrete models) Model to Test The first model of the concept that tests Model, Ideas, Early, [A prototype is] An initial basic assumptions about the concept or Test, Concept, model to test the concept and idea ideas you have come up with Model to An early, simple model or sample of Communicate, Basic, [Prototyping is the process of] Communicate the eventual final product that helps to Ideas, Concept, Creating a model that gets communicate the product Function across the basic ideas and functions of a concept Model to Decide A model that provides insights that help Model, Decide, [Prototyping is a method of] the design team make decisions Rough Making a crude model of the project(s) to further the decision-making process Model to Interact A model that tests or enables in some Model, Test, [Prototyping is the process of] way an interaction between the concept Feedback, Testing models for getting and the end users Customers, Final feedback from customers Product before the real product

Figure 1-7. Nielson’s prototype space and the three categories of low fidelity prototypes.

1.7.4 Summary of Findings and implications for Future Work

Our findings provide important insight into the baseline understanding of prototyping activities held by novices and help inform future studies and exploration of this topic. First, as the most common category found in the response set, over half of the 194 total responses (98) were coded as Model to

Link. It is important to note that no responses were tagged with more than one category. These responses

23 described prototyping generally as a link between ideas and final products or as a stage gate in the design process. Often, these responses assigned no other purpose to prototypes other than to physically model ideas or to be the first physical representation of the final product; they were the least specific in terms of the purpose of prototypes or prototyping activities.

Model to Test was the second most common category found in the response set, with 61 responses coded in this category. These responses described the main purpose of prototyping as testing the features or qualities of a design in some way in order to make changes to the overall concept. Model to Test responses typically did not describe specific tests or experiments, but referenced only general evaluations of models that would lead to another iteration of the design. So, while many of the study participants recognized the value of prototypes for testing, their understanding was incomplete in terms of how to make that testing effective.

Model to Communicate was the third of the five categories in terms of response frequency, with

22 responses coded in this category. Model to Communicate responses described the main purpose of prototyping as communicating a design concept to users or fellow team members. These responses often referenced prototypes as tools to help others understand more clearly what a concept is and what the design would eventually be. For a design team to be effective, good communication is key [80], and the HCI literature has confirmed the necessity of prototypes when communicating design ideas to end users [50,98], as previously discussed.

Model to Interact was the fourth category in decreasing order of responses; only 10 of 194 responses were coded as Model to Interact. This category might have been viewed as a sub-category of

Model to Test, but the specificity of the Model to Interact responses was greater and warranted its own category. These responses described the main purpose of prototyping as gathering feedback from end users; however, few of the responses included how this feedback might be used to improve the overall design. The literature shows that interactions with end users are critical for the development of successful products [79,104,111]; we also know from the literature that lower fidelity prototypes developed rapidly can lead to richer feedback from end users [48,51,90].

Finally, Model to Decide had the fewest responses of any category, with only 3 responses coded as Model to Decide. Nevertheless, the distinct theme presented in all three responses merited a category of its own. Each response specifically highlighted the main purpose of prototypes as decision-making tools for the design team. The infrequency of this category highlights the need for a holistic and structured prototyping framework that highlights more clearly the potential for prototypes to serve as an effective decision-making tool for design teams, as discussed in the previous section.

From our qualitative analysis, we hypothesize that novice designers hold imprecise and incomplete perceptions about the purpose and value of prototypes and prototyping activities within the design process; however, further work is needed to validate this claim. Our findings imply that while novice designers are aware that building prototypes is a necessary step in the design of new products, they are unaware of specific goals for prototyping activities, the importance of the context in building a prototype, the need that determines the building of a prototype, and most importantly, how prototypes can contribute to the refinement and improvement of concepts, ideas, and products. From these findings, we hypothesize that a holistic and structured prototyping framework should clearly define a need, context, and goal for each prototyping activity.

1.8 Dissertation Outline

Based on the discussion in Sections 1.1-1.6, it is clear that while methods for effective prototyping exist across domains, there is a need for a structured and holistic framework to address the concomitant gaps found in the prototyping research, specifically regarding previous development and implementation of prototyping frameworks. Based on our findings, we assert that a more holistic, structured prototyping framework can unify critical insights in prototyping research and their implementation in prototyping practice. We suggest that implementing this framework within the context of more realistic design challenges will enable research findings to be generalized more broadly

25 across the field of engineering design. The goal in the current research is thus to develop, test, and validate a prototyping framework, we call Prototype for X, in a moderately complex design task.

In Chapter 2, the Prototype for X (PFX) framework is presented and justified. We review the core elements of PFX, namely, Frame for X, Build for X, and Test for X. We also review the first lenses chosen for PFX, i.e., desirability, feasibility, and viability. Similar to the “illitites” in the field of Design for X (DFX), we propose using lenses to help designers structure and scaffold the phases of prototyping activities.

In Chapter 3, an application of the PFX framework is presented. Data on the desirability, feasibility, and viability of end designs is presented. We review experimental protocol, findings, and implications of this work for future product development.

In Chapter 4, the same application from Chapter 3 is used to evaluate the effect of PFX on the designers themselves. We start by reviewing the development of a new psychometric scale, PAWS or the Prototyping AWareness Scale. We then review how this scale was used to gather data on students’ prototyping awareness during various prototyping activities. Finally, we discuss experimental protocol, findings, and implications for engineering educators.

In Chapter 5, we explore whether the sequence of PFX lenses has any impact or effect on end designs and prototyping awareness. A new application of PFX is presented and tested. Two sequences of PFX are used to investigate the differences that the sequence of lenses can have on product and people outcomes.

In Chapter 6, conclusions are discussed, and research contributions are summarized. Finally, topics for future work are presented.

Chapter 2

The Prototype for X Framework

2.1 Introduction

In Chapter 1, we identified three major functions of prototypes and four specifications for prototyping frameworks from a review of the existing literature. We also proposed the development of a new prototyping framework to address the concomitant gaps found in prototyping research, specifically regarding previous development and implementation of prototyping frameworks. We suggest that implementing this framework within the context of more realistic design challenges will enable research findings to be generalized more broadly across field of engineering design.

Specifically, we propose Prototype for X (PFX) as a novel framework to guide product design during prototyping activities throughout the design process and to address the issues previously stated. We also claim that structured prototyping methods focused on the three lenses of human- centered design (HCD) can positively influence final design outcomes, i.e., the desirability, feasibility, and viability of the end product. Details on the key elements of PFX and its three lenses follow.

In this chapter the development of the Prototype for X framework is presented. The first step in this development is to review the theoretical underpinnings upon which PFX draws. We then review the elements at the core of the PFX framework. We conclude by reviewing the initial lenses used to evaluate the impact of the PFX method on both design and designer outcomes.

2.2 Prototype for X: Frame, Build, Test

We introduce Prototype for X (PFX) as a novel framework to guide prototyping during product design activities and to address the gaps in the literature reviewed in Chapter 1. We also claim that structured prototyping methods focused on the three lenses of HCD can positively influence final design outcomes, i.e., the desirability, feasibility, and viability of the end product.

The Prototype for X (PFX) framework is composed of three key phases: (1) Frame, (2)

Build, and (3) Test. Through these phases, PFX helps designers focus their resources and efforts on building prototypes that test core assumptions and lead to deeper and richer insights about specific aspects of the design at the time of testing. As shown in Figure 2-1, PFX is informed by and informs the entirety of the design process proposed by Ulrich and Eppinger [12]. While some argue that prototyping may be a distinct phase in the product development process, we assert that prototyping is a useful activity throughout design and introduce PFX as an overarching framework that can guide prototyping activities throughout the entire design process. In Figure 2-1, “analyze” refers to a design team’s decision-making step at the conclusion of testing and evaluating a prototype; if they choose to iterate on the design, then the Frame, Build, Test phases are repeated.

These results are analyzed within the context of the specific “lens” for PFX, i.e., the “X” of PFX, which is determined at the start of the process. As previously stated, the lenses of PFX are the foci designers can use to structure and scaffold the phases of prototyping activities. We propose that there are many lenses at a variety of abstraction levels, and in this work we use three lenses for the initial implementation of PFX (see Section 2.3). At the conclusion of the PFX process, the prototype must be evaluated against a contextually relevant metric or set of metrics.

Figure 2-1. PFX structure and integration into engineering design process [12].

2.2.1 The Frame Phase of PFX

Studies have shown that the framing of a problem can have a significant impact on the results that designers produce. Specifically, in over twenty studies from the creativity and problem- solving literature, problem framing and explicit instructions (e.g., to “be creative”) were shown to have some facilitative effect on the end results [76]. This framing effect, or “the finding that subjects often respond differently to different descriptions of the same problem” [112], has been used to show the dependence of an individual’s preferences and actions on the formulation or framing of the problem or task [56,112,113]. The Frame element of PFX combines the framing effect with

Specifications 1– 4 derived from the literature to create a prototyping frame. From the literature review in Chapter 1, pilot study, and previous work in framing psychology we identified that a complete prototyping framework would identify the following elements to frame prototyping activities: 1) a context to orient the design team, 2) a need to constrain design efforts, and 3) goals to test and evaluate prototypes. This structure is paralleled in ideation research through the Design

Problem Framework (DPF) of Silk et al. [114], which has been shown to be an effective scaffold to successfully shift students’ ideas and diversify ideation outcomes as needed for the design problem. The DPF is grounded in research on cognitive styles and problem framing; it was

29 originally used to understand how the framing of a design challenge affects ideation and solution generation [114]. The DPF divides design challenges into the same three elements, namely: (1) context, (2) need, and (3) goal. In ideation research, context refers to who needs a solution and what purpose the solution serves; need refers to functional requirements and constraints of the design challenge; and goal refers to the instructions used to generate ideas and the metrics used to evaluate those ideas.

We adapted the Design Problem Framework [114] based on the four specifications for prototyping frameworks derived in Chapter 1, to create an appropriate framing element within PFX.

The importance of a prototyping context within the prototyping frame is apparent after reviewing

Specification 4 (a prototyping framework should be flexible with regards to prototype constraints

(i.e. material, fidelity, mode), encouraging a prototype appropriate for the level of interaction or engagement desired by the design team). In this sense, context refers to the constraints of prototype development and more specifically, the purpose the prototype will serve, such as gathering user feedback, testing subsystems, or evaluating manufacturability. Next, Specification 2 highlights the importance of need within a prototyping frame (a prototyping framework should enable design teams to quickly select a prototyping focus tailored around specific design goals, to support the effective decision-making in the design process). In the context of prototype development, need refers to the most critical information for the design team to gather about the desired product at that point in time in order to facilitate effective decision-making. The need helps constrain prototyping efforts to only the most critical aspects of a design. Finally, goals are important aspects of a prototyping frame, as evidenced by the focus of Specifications 1 and 3 on the goals of prototype development – either overall prototype quality (Specification 1) or user feedback (Specification 3).

In PFX, goals refer to the outcomes or metrics by which prototype success will be measured (e.g., increases in technical quality or system functionality).

An example of the frame phase of PFX is shown in Figure 2-4. For the initial implementation of PFX, framing was done by the instructors of a junior-level mechanical engineering design course during a lab section of that course. The example shown is from the first pilot study of PFX in which students were instructed to build a hand-held vacuum. The context

(ACME Tool Company) remains the same across the three frames; however, the needs and goals are adapted dependent upon the lens being applied (i.e., desirability, feasibility, or viability).

Figure 2-2. Example of the Frame Phase of PFX from Case Study One

2.2.2 The Build Phase of PFX

At the core of the PFX framework is the act of building physical prototypes. As reviewed earlier, developing physical prototypes is key to the development of new products. Andreason and

Hein [115] and Bucciarelli [116] showed that building physical models in the early stages of the design process can help visualize problems and highlight incorrect design assumptions. Brereton

[85] found that engineering students often seek out physical “props” or design small-scale models when struggling to communicate design ideas. PFX uses the prototyping frame as the prompt to help guide building, so that designers and engineers can allocate their efforts and resources towards

31 specific prototyping outcomes more effectively. Fidelity, materials, and tooling used to build prototypes are then decided upon by the design team as a function of the critical assumption they are testing and the prototyping frame in which they are working. Specification 4 involves critical aspects of material selection, scaling, or time management and highlights the importance of a structured building step within a prototyping framework such as PFX.

The Build phase of the PFX framework also draws on Lean Startup methods [117], which advocate the use of a minimum viable product (MVP). An MVP is the version of the product that allows the design team to collect the maximum amount of validated learning about customers with the least amount of effort (i.e., a prototype). We expand this MVP concept to apply more broadly to engineering contexts, in which it may be more important to test the functionality, manufacturability, or systems architecture of a prototype or design, in addition to validating user feedback. Fidelity, materials, and tooling are then a function of the assumption being tested, which requires design teams to think critically about the simplest and fastest path to developing a prototype to test this assumption using the least amount of prototyping effort.

As an example of how one might implement this phase, we have included the handout that was provided to student designers during our second case study in Appendix B. This handout instructs students to build a prototype that best validates or invalidates a core assumption about their design. For example, the Build for Desirability handout instructs students to “Build a prototype that your team feels best answers or addresses an important user need. You will be testing your prototype with at least five users and taking notes of feedback and comments. You should not focus on building the best prototype you possibly can, but building a prototype that will spark the richest or most useful feedback from users”. During this junior-level mechanical engineering design, course students were provided with handouts for in-class activities during each lab section, and the goal of the overall course project was to utilize 3D printing to create an innovative, hands- on, educational kit. For the PFX lab sections, students first received a lecture on the PFX lens being

32 implemented, then instructors used one of the frames shown in Figure 2-4 to contextualize the student’s prototype, and finally students were given one of the sheets from Appendix B and instructed to begin building their prototype in class. For example, during the Prototype for

Desirability build activity, students were instructed to reflect upon a critical assumption that they have made about their user, and write down the features of their concept a user would value most.

They were then instructed to build in the simplest manner possible a prototype that would enable them to test this assumption with a user. They were given example prototypes that test the usability, ergonomics, or appeal of a concept from industry and were provided with an abundance of materials ranging from low fidelity media (such as cardboard) to higher fidelity (such as 3D printed materials). The Build for X sheets can be found in Appendix B.

2.2.3 The Test Phase of PFX

The final phase of the PFX framework is testing core assumptions using the prototypes developed in the Build phase. Häggman and Yang [42] found that teams who more frequently brought prototypes to users or performed analysis on core functions through rapid experimentation had higher quality final designs. As discussed in Sections 1.3, prototypes that test user feedback can help catalyze the design process through rapid identification of errors and design flaws. Testing is a critical part of the prototyping process, and PFX integrates focused testing into the overarching framework. “Focused testing” means that engineering designers and students are not building a prototype just because it is the next stage in design; instead, they are building prototypes to gather specific feedback or to analyze core functions critical to the improvement of the design. The importance of focused testing as an element of PFX is reflected in Specifications 1, 2, and 3, in which the critical insights design teams can gather from specific testing related to technical quality,

33 cost, time, and users. Specific examples of this focused testing are provided next, as we detail the targeted design outcomes of PFX and review implementation of the PFX framework.

As an example of how one might implement this phase of PFX, we have included the handout that was provided to student designers during our second case study in Appendix C. In this handout, students were instructed to evaluate their prototype against a set of relevant metrics. For example, in the Test for Feasibility handout students were instructed to devise a method to evaluate the functionality of their design and record all findings in the students’ journals. They were also told to use root-cause analysis if the product failed and determine which feature, sub-system, or component failed and why. During this junior-level mechanical engineering course students were provided with handouts for in class activities during each lab section, and the goal of the overall course project was to utilize 3D printing to create an innovative, hands on, educational kit. For the

PFX lab sections, following the build activity reviewed in section 2.2.2 students were presented with one of the handouts from Appendix C and asked to complete a reflection once the testing activity was complete. For example, during the Prototype for Desirability test activity students were instructed to test with at least five users from their target audience. They were then instructed to record any insightful feedback from users and ways in which the user interacted with their prototype. A graded journal activity was paired with the test phase and required students to reflect upon the test and not only write down their findings but discuss as a team how they might iterate on their prototype based on the feedback from users. The Test for X sheets can be found in

Appendix C.

2.3 Targeted Outcomes of PFX: Drawing from Design for X and Human Centered Design

In the field of Design for X (DFX), designers use the “illities” (e.g., design for manufacturability, design for sustainability) as the lenses through which they structure and scaffold the design process to make improvements in specific areas. Similar to the “illities” of DFX, the

PFX framework uses lenses to help designers structure and scaffold the phases of prototyping activities. From Specifications 1 and 3, we know that a prototyping framework should improve outcomes in terms of technical quality and the usability of the product. DFX methods have successfully helped reduce costs of manufacture and assembly, as well as reduce overall assembly complexity and improve technical quality, at firms like Honda [118] and Ingersoll Rand [119].

HCD uses a comparable but somewhat different set of lenses than DFX to structure design efforts, namely, the feasibility, viability, and desirability of the product. We propose that these three lenses (feasibility, viability, and desirability), taken together, will also help close key gaps in prototyping research. Meanwhile, the PFX framework is defined independently of these lenses to allow designers to use other “ilities” during the prototyping process as needed.

2.3.1 Human Centered Design

“Human centered design advocates that a more promising and enduring approach is to model a user’s natural behavior to begin with, including any constraints on their ability to attend, learn, and perform so that interfaces can be designed that are more intuitive, easier to learn, and freer of performance errors” [120]. Although this statement from Oviatt refers specifically to user interface design within human computer interactions, it can be generalized to the broader field of design.

Human Centered Design, HCD, views innovation through three lenses: 1) desirability, 2) feasibility, and 3) viability as pictured in Figure 2-3. The desirability lens asks questions like how will the user engage with the product, will the user find the product compelling, and how desirable is the product. The feasibility lens ask questions about what is technically and organizationally feasible, and the viability lens asks questions about what is financially and economically viable for the company [121]. The HCD model for innovation is used by Stanford’s d.school and IDEO [121], and countless other organizations and institutions [22,106,122], and it is adopted in this work.

2.3.2 Design for X

Design for Assembly (DFA) and Design for Manufacture (DFM), the first research area within Design for X methodology, was pioneered by Boothroyd and Dewhurst as an alternative design process to reduce the final cost of a product [123]. Boothroyd and Dewhurst define design

36 as “... an activity that starts with sketches of parts and assemblies, and progresses to the drawing board or CAD workstation, where assembly drawings and detailed part drawings are produced”

[119]. In DFM, estimates for assembly time are used to predict the cost of design concepts. This knowledge is then used to adjust design variables such as number of parts, number of assembly moves, etc., ultimately resulting in new concepts that minimize assembly costs. An example of a

Design for Manufacture (DFM) framework is shown in Figure 2-4.

Figure 2-4. An example of a Design for Manufacture framework is highlighted showing the decisions and considerations taken while using DFM methods [123].

DFA and DFM quickly became popular at large manufacturing companies such as Honda and Ingersoll Rand, who incorporated the method as early as the 1980’s as an attempt to decrease product costs and time to market [123]. Designers and engineers were now able to make informed decisions about concepts by evaluating their overall manufacturability and assembly costs early in the design process. Researchers began adjusting and adapting DFA and DFM, tailoring the overall structure to fit new needs such as eco-friendly designs [118], and the field of Design for X, or DFX, was born.

Kuo et al. [118] note that “the practice [of Design for X] will lead to more optimal designs when the entire lifecycle of the product from conception to disposal is considered”. Kuo et al. highlight the underlying theory of DFX by considering the lifecycle constraints of a product as early as possible in the design process, the end design will require less time and capital. The underlying theory of DFX is that by actively designing for x (manufacturability, etc.) the amount of redesigns, delays, and wasted resources further along in the design process is reduced

[118,119,123]. Although DFA has expanded into a variety of other areas, now termed DFX, the main focus of DFX remains as cost, and function. We categorize this as viability and feasibility of design respectively.

Kuo et al. [118] also called for more attention to Human-centered Design (HCD) as part of

DFX. They stated that the future of DFX lies in the design of products that incorporate user feedback and related findings earlier in the design stage. Design for X has historically focused on the viability and feasibility of products and concepts by focusing on the financial and functional aspects of design. Design for assembly, manufacture, and countless others are used in order to reduce the overall cost of the product itself, thus increasing the product’s financial viability. Design for quality and disassembly focus on the technical feasibility of making a higher quality product, or a product that is easily disassembled.

None of the DFX methods currently address user desirability. User satisfaction was found to be directly related to return on investment in a study from Anderson et al. [71], and this same study showed that a one point increase in user satisfaction would result in a 11.4% increase in return on investment. User satisfaction and perceived value (both indications of a product’s overall desirability) are key to product and firm success. In the next section, we review Design Thinking

(DT) and the potential synergies with DFX methods.

2.3.3 Design Thinking

Design Thinking is the Human-Centered Design method created by the product design firm

IDEO, and used by countless universities and companies such as Stanford, MIT, SAP, Google, GE, and Procter and Gamble [22,124,125]. Design Thinking is described as “an iterative process that moves from generating insights about end users, to idea generation and testing, to implementation”

[124]. In a review of Design Thinking and designerly thinking literature, Johansson et al. [126] cite

IDEO as one of the origins of the design thinking “way of working”. The article also highlights that the Design Thinking methodology put forward by IDEO is not grounded in any form of a theoretical framework. Instead, IDEO’s method is based on years of experience designing innovative products like the first Apple mouse [125]. Design Thinking has a bias towards action and has helped IDEO to be consistently ranked among the top ten most innovative companies by organizations such as Fast Company and the Boston Consulting Group.

In the Design Thinking process, designers use these three lenses to generate ideas and more importantly build prototypes. Prototypes can be used to gather feedback on the feasibility, viability, and desirability of each concept. As Gabrysiak et al. note “design thinking is at its best if tangible prototypes can be used to capture and validate end user needs and envision new products and services” [19]. While design thinking methods have become popular within industry, specifically on product development teams [125], there is little research exploring the inner workings of the method. As Kimbell notes “just what design thinking is supposed to be is not well understood, either by the public or those who claim to practice it” [124]. Preliminary research within design thinking has focused on the cognitive processes used in [127] and mood levels throughout the process [122]. The most important activity within Design Thinking, namely, prototyping, is the least clear. David Kelley, founder of IDEO, is quoted as saying “never go to a meeting without a

39 prototype” [128]; however, Design Thinking does not specify the type or fidelity of prototype, nor is a clear prototyping process presented.

2.3.4 Initial Lenses of PFX: Desirability, Feasibility, and Viability

As we found in our systematic literature review, prototyping research within management science and engineering management has led to several models that aid in the decision-making process in order to optimize prototype planning in terms of cost or time. However, these works often fail to deliver actual methods for the development of the prototypes themselves; in reference to the frame-build-test phases of PFX, previous work has stopped at the frame phase. Applying the viability and feasibility lenses to prototyping will close this gap by providing methods for the Build and Test Phases of prototyping activities, in addition to the frame phase. Another gap in previous research on prototyping frameworks exists in the failure to incorporate user-centered design practices. Applying the desirability lens to prototyping closes this gap by providing the structure and guidance typical of a prototyping framework, but encouraging the implementation of Human

Centered Design methods.

Informed by these observations, we propose combining the structure of DFX with the focal points of HCD and incorporating DT methods to create the initial three lenses for the PFX framework that are explored in this work: Prototyping for Feasibility (PFF), Prototyping for

Viability (PFV), and Prototyping for Desirability (PFD). These three lenses or goals (i.e., the X’s of PFX) are designed to integrate user-centered design with resource, time, and function-focused design, while providing designers with a clear road map for prototyping activities. Definitions for the three PFX lenses were generated from the literature (i.e., engineering management, management science, engineering design, design thinking, and human computer interaction literature) as follows:

Prototyping for Viability (PFV): We define Prototyping for Viability as the practice of creating prototypes that test the design’s likelihood of fitting into time and budget constraints. This definition was created by synthesizing literature within the management sciences [10,27,29,32,129] and engineering management [3,91,130].

Prototyping for Feasibility (PFF): We define Prototyping for Feasibility as the practice of creating prototypes that test the technical functionality of the design. This definition was created by synthesizing literature within engineering design [20,23,42,44,98], engineering design education [14,35,46,55,62,90], and human computer interaction [51,90,95,100,131].

Prototyping for Desirability (PFD): We define Prototyping for Desirability as the practice of creating prototypes that test the purchase-ability and consumer value of a product or solution. This definition was created by synthesizing literature within human computer interaction

[71,97,118] and interaction design [45,104].

These definitions provide a foundation for the methods used to implement PFX. As discussed previously, the overarching structure of PFX is Frame, Build, and Test. Accordingly, each lens has corresponding Frame, Build, and Test phases that are specifically related to that lens; the lens also impacts the specific methods applied in those phases. For example, PFF (Prototyping for Feasibility) uses design methods such as functional decomposition [63,132] and black box diagrams [28,133] to anchor the design team's efforts and focus them on testing key assumptions about the design’s feasibility by iteratively testing the core subsystems (achieving Specification 1 and 2). Although functional decomposition and black box diagrams are often taught in engineering design courses, the key difference in PFF is the use of these methods to help participants’ focus

41 their prototyping efforts as opposed to their conceptual ideation activities, where they are typically used [12]. PFF pairs these methods with engineering decision-making tools, such as the Analytical

Hierarchy Process [134], to determine which subsystem should be built and how a prototype can be made in the fastest and most efficient way possible to test critical assumptions about the design’s functionality. During the Test Phase of PFF, the prototype is evaluated based on its overall technical quality, and the design team decides whether they can move on to another sub-function or whether re-design of the current sub-function is needed. In this way, Prototype for Feasibility (PFF) helps designers work through the technical functionality of their designs and provides a formalized roadmap for prototyping for technical function.

The other two lenses of PFX defined here follow similar patterns and structures, relying on methods such as critical part count ratio from DFMA [119] and usability testing from HCI [90] and

HCD [135] to frame prototyping efforts. PFV aims to increase the manufacturability of end designs and optimize the ease of assembly, thus reducing the overall cost and resource use of the product, which reflects Specification 2. Finally, PFD focuses on increasing user satisfaction and user- perceived value of a final product by encouraging designers to iteratively and quickly test with end users, mapping to Specification 3. Overall, PFX helps design teams “focus on what matters” by constraining prototyping activities to only test the relevant critical assumptions, whether they are assumptions about the user, function, cost, or manufacture of the design, thus reducing wasted effort and increasing the overall efficiency of the design process.

2.4 Exploring the Implications of Prototype for X

As a formalized framework to guide product design during prototyping, PFX is composed of three key phases: (1) Frame, (2) Build, and (3) Test. Through these phases, PFX helps designers focus their resources and efforts on building prototypes that test core assumptions and lead to

42 deeper and richer learning about specific aspects of the design at the time of testing. PFX draws from Human-Centered Design (HCD), Design Thinking (DT), and Design for X (DFX) methods to positively impact final design outcomes using the three lenses of HCD, namely, the desirability, feasibility, and viability of the end product. Initial implementation of the framework was done in undergraduate engineering design courses. Future work will focus on implementing and testing the impact of PFX in professional settings; however, the focus in this work is on evaluating the impact of PFX in educational settings with novice designers in engineering. Although we cannot generalize the results of this experiment to an industrial environment, we argue that improving the educational practice surrounding prototypes and improving the prototyping practice of novice designers in engineering will likely translate to improved prototyping practices in professional settings as these students graduate and take jobs in industry.

Using the PFX framework introduced in this chapter, we implemented the Prototype for

Desirability, Prototype for Feasibility, and Prototype for Viability methods in two junior-level mechanical engineering design courses in order to answer three overarching research questions:

(1) How are end designs affected by a structured and holistic prototyping framework, such as PFX? (2) How are designers’ prototyping awareness affected by a structured and holistic prototyping framework, such as PFX? (3) How does the sequence of PFX lenses affect end designs and prototyping awareness?

In the sections 2.4.1-2.4.3 we review the hypotheses and rationale that go along with each of these three research questions.

2.4.1 How are end designs affected by a structure and holistic prototyping framework?

We believe that the final design’s perceived value and user satisfaction will increase as a result of the Prototype for Desirability method compared to unstructured prototyping practices.

This hypothesis is based on previous research that indicates positive relationships between user

43 feedback in early stage prototype development and user satisfaction [4, 17-22]. We also hypothesize that using PFX will increase the technical quality of the final design based on previous research that indicates positive relationships between prototyping strategies and technical quality

[8, 9]. Finally, we hypothesize that the viability of the final design will be improved through PFX methods; specifically the manufacturability of final designs will be improved. This hypothesis is based on previous research that indicates positive relationships between DFMA methods and final manufacturability of components and designs [46].

2.4.2 How is designer’s prototyping awareness affected by a structured and holistic prototyping framework?

The prototyping literature has typically evaluated the few prototyping methods, tools, and frameworks using design-based metrics, such as binary evaluations of completion of a design task

[6,14,136] as reviewed in Chapter 1. In other words, there are few prototyping studies that evaluate how the designers themselves are being affected by prototyping methods.

We hypothesize that structured prototyping methods, specifically PFX, could increase novice designers’ and engineering students’ self-efficacy, leading to an increase in feelings of control throughout the prototyping process, which may lead to an increase in creative output, higher levels of motivation, and an increase in the quality of final designs. As an initial step in the measurement of these outcomes, we sought to understand if PFX influences designers’ prototyping awareness. We believe that awareness in engineering design is the first step towards the development of self-efficacy beliefs. We define prototyping awareness as the ability to identify and remain open to new prototyping behaviors, frames, and perspectives.

2.4.3 How does the sequence of PFX lenses affect end designs and prototyping awareness?

In addition to evaluating the effect of a structured and holistic prototyping framework on end designs and designers themselves, we are interested in studying the effect that PFX might have on the prototyping process. Specifically, we wanted to understand how the order or sequence of

PFX lenses might affect the feasibility, viability, and desirability of an end design and the prototyping awareness of designers. The Human Centered Design framework states that designers should begin with the desirability lens when designing new products [121], while traditional engineering design methods start by approaching design challenges from a functional perspective

[12], i.e., a feasibility lens. This focus on feasibility has been confirmed in previous studies of engineering students, which found that students typically focus on the technical quality of designs when moving through the new product development cycle [137]. We hypothesize that the sequencing of these lenses during prototyping activities will impact the final designs in addition to the prototyping awareness of the designers.

2.4.4 Summary and Looking Ahead

The Prototype for X framework was proposed and the main elements of the framework were detailed. We mapped specifications from Chapter One onto the elements of PFX as well as the initial lenses chosen for implementation of PFX (desirability, feasibility, and viability). We justified the use of the Human-Centered Design framework, Design for X methods, and Design

Thinking methods, as the guiding structures for the initial Prototype for X lenses. Lastly, we introduced our three over-arching research questions that have guided the remained of this dissertation.

Chapter 3

Assessing the Impact of PFX on End Designs

3.1 Introduction

Previous research has highlighted the impact that formal prototyping methods could have on the qualities of end designs. Much of the existing work has only evaluated the overall “binary” success or failure of prototypes [6,14,16,19,23]. In contrast, we see prototyping as a multi-faceted activity that requires multiple unique lenses for evaluation [138]. Therefore, although previous research shows the influence of prototyping methods on product outcomes, it is unclear which aspects of the final product are affected by prototyping methods. The success or failure of a product in the market is dependent upon a variety of factors and product characteristics, including its usability, technical quality, novelty, perceived value, manufacturability, and more.

We begin this chapter by investigating the appropriateness of using classic ideation metrics to evaluate functional prototypes using a within subjects experiment. We first review the ideation literature and discuss the translation of concept qualities, typically associated with innovation

(novelty, quality, and creativity), through the product development process. We summarize our findings from this study and make a case for the creation of new metrics that are more relevant and applicable to prototypes.

Following this investigation into prototyping metrics, we select and review the metrics used to evaluate the effectiveness of PFX on end designs in a between-subjects’ analysis. Results from this second study indicate that using PFX can help increase the desirability, feasibility, and viability of functional prototypes; specifically, student teams introduced to PFX produced end designs that outperformed end designs from control teams across user satisfaction, perceived value, and manufacturability metrics. This study confirms the impact that PFX could have on the

46 prototyping process and end designs, potentially reducing the amount of wasted resources during new product development.

3.2 Background in Ideation Metrics

Systematic methods to generate novel and feasible ideas have existed and been in use for many years. The effectiveness of these methods has traditionally been evaluated with ideation metrics, such as those from Shah, et al. [139], Dean et al. [140], or Amabile [141]. Typically, ideation studies have evaluated the novelty, and quality of ideas [88,139,141,142]. For this work, we briefly review the works of Shah, Vargaz-Hernandez, and Smith’s (SVS), Dean et al., and Amabile. We use Dean et al.’s metrics [140] in this work because our team has found that their metrics are more appropriate for large data sets unlike the SVS ideation metrics, which are affected by sample size (less effective for larger samples), or Amabile’s method, which is extremely time-intensive.

The SVS metrics have been widely used in the engineering design community to evaluate ideation methods [143], understand fixation effects [144,145], and explore decision-making processes [146]. Shah et al. [139] described four objective measures of ideation effectiveness, 1) novelty, 2) variety, 3) quality, and 4) quantity. To calculate an idea’s score on these measures, each design is decomposed into representative features, and weights are assigned to each feature level.

While this method has been used in multiple design studies, the SVS method only provides insights into the relative creativity of a set of ideas as all of the metrics are baselined against a single sample.

Amabile’s [141] Consensual Assessment Technique uses expert raters with a background relative to the design artefacts domain to independently rate ideas generated using a 6-point scale

(very uncreative to very creative). Creativity is often defined by the raters as an idea that is both novel and useful [147]. Unlike the SVS method, Amabile’s Consensual Assessment Technique rates the global creativity of the ideas, but these ratings are completely dependent on the experts.

For example, if two experts using this method do not share the same definition of what a creative idea is, than this method is useless. Studies have also shown that while raters consider both novelty and usefulness when rating overall creativity, they often place a larger emphasis on the idea’s novelty [148,149].

Dean et al. [140] evaluated the utility of ideation metrics by reviewing 90 studies; they found that while the novelty of ideas was always measured in a study, there was inconsistency across studies in the evaluation of an idea’s quality and creativity. Dean et al. [140] found that the attributes measured across all 90 studies could be mapped to one of MacCrimmon and Wagner’s

[150] four dimensions of an idea: (1) novelty, (2) workability, (3) relevance, and (4) specificity.

Dean et al. [140] extended MacCrimmon and Wagner’s [150] dimensions and developed ordinal rating scales to evaluate ideas on these sub-dimensions. Previous ideation studies have used these scales as an effective and efficient technique to evaluate ideation methods [56,151,152]. While

Dean et al. proposed eight metrics (two measurable sub-dimensions for each of the four dimensions), in this study, we utilize only four of these metrics as they were most appropriate for our design challenge and engineering context.

The purpose of all of these ideation metrics is to aid in concept selection, namely, choosing ideas or concepts that are more innovative, have higher quality, and are more likely to advance in the product development process. These metrics also provide practitioners with useful quantitative data regarding the characteristics of an idea and idea set (i.e., what do more innovative ideas look like, what are the most creative ideas in the set, etc.). The metrics reviewed in this section have only been applied to ideas and, as of yet, no researchers have attempted to utilize ideation metrics to evaluate the design artifacts that come after the idea generation phase. It is not enough to generate novel, feasible, and creative ideas; these novel concepts must also be selected and developed into prototypes and eventually products.

3.3 The Translation of Idea Characteristics throughout the Design Process

Recently studies have begun to evaluate the translation of creative concepts through the entirety of the innovation process. Research has found that while idea generation methods help design teams generate a large and rich solution space rapidly, creative ideas are often filtered out during the concept selection process [137,153]. Starkey et al. [137] found that student design teams often focus on the technical feasibility of concepts during concept selection processes. Rietzschel et al. [153] and Ford et al. [154] found that individuals often filter out creative ideas due to an inherent preference for conventional or previously successful concepts.

In order to combat these biases, several concept selection tools have been proposed.

Traditional concept selection tools such as the Pugh chart [155] and Analytical Hierarchy Process

[134,156] use engineering specifications to score concepts. This results in the technical feasibility of concepts being emphasized and little attention is paid to the creativity and novelty of concepts.

Starkey et al. [157] propose the Tool for Assessing Semantic Creativity, or TASC, as an alternative concept selection tool to reduce a design team’s bias against more radical types of creativity. They found that TASC successfully measures similar concepts of creativity as informal team discussions and expert evaluations.

While concept selection has been described as the “gate keeper” of creative ideas [146], we know that multiple stage gates exist throughout the new product development process. While the aforementioned works provide tremendous insight into the concept selection process, little research has explored what comes next. Ideas do not stop developing once they have been selected.

During the prototyping process, critical gaps and shortcomings of designs are often identified, and ideas are refined and evolved into final products. No one has explored how the novelty and creativity of an idea changes throughout this process or whether such metrics are even appropriate for assessing an idea as it develops through prototypes and ultimately into a product.

Engineering design, engineering design education, and human computer interaction researchers have all explored the effects of fidelity and frequency of prototyping on the technical feasibility of the end design. When technical feasibility is evaluated in reported studies, it is typically measured with a binary metric (e.g., did the object hit the target [16,23], did the device move the flag [14], did the egg survive the drop [67]) as a measure of technical quality. The success or failure of a product in the market is dependent upon a variety of factors and product characteristics, including its technical quality, novelty, creativity, and more.

Saunders et al. [158] studied the characteristics of over 197 innovative products on the market in order to understand what separates successful products from failed products. They found that innovative products exhibited enhanced user interactions, most displayed innovative product architectures, and over a third of these innovative products had additional features when compared to less innovative competitors. The researchers define an innovative product as “a product that changes or has the potential to change the nature of the marketplace by satisfying a new (or latent) customer need or by satisfying customer needs in a significantly new way” [158]. We clearly have rigorous metrics and methods to evaluate products in the market as well as ideas; however, there is not a standard method or set of metrics to evaluate prototypes.

From Saunders’ definition of innovative products, the differentiating factor for successful products in the market is the novelty, creativity, and desirability of the product or its features.

Prototyping research has only focused on the impact of prototyping on the final design’s feasibility or technical quality using binary metrics (i.e., success or failure in meeting an objective). Although previous research shows the influence of prototyping methods on product outcomes, it is unclear how a concept’s creativity is refined and evolves from the ideation process through the prototyping process into a final product.

3.4 Selecting Appropriate Metrics for Prototypes: Study Design

In order to understand if traditional ideation methods reliably and validly measure idea and prototype characteristics as an idea evolves over time, five existing ideation metrics were used in a within subject study. A small pilot study was conducted over a semester timeframe (i.e., 8 weeks) in a junior-level mechanical engineering design course with over 228 undergraduates and approximately 70 design teams. At the start of the semester, students were tasked to develop educational kits that leverage 3D printing to create novel and engaging hands-on classroom experiences. Students were instructed that the kit should be easily packaged and distributed and should cost less than $50. Students were given the entirety of the semester to design, develop, fabricate, and test the educational kit and were provided access to ample 3D printers. Each team was also given a budget of $50 to develop and create the educational kit. The full description of the project, distributed to students at the start of the semester, is included in Appendix D.

In order to gather data about the transformation of concepts from ideas to prototypes, the study took place over the final eight weeks of the semester. During the eight weeks, data were gathered during three-ideation generation sessions and at the close of the semester when final prototypes were submitted. Figure 3-1 highlights the timeline of the study and data collected.

Figure 3-1. Overview of the study conducted over eight weeks during a junior-level mechanical engineering design course.

Collecting data over the course of this time period allowed us to gather insights into the evolution of ideas once concepts had been selected, as at the start of the eighth week, each team was required to select a concept to advance to prototyping. During ideation activities, teams were instructed to brainstorm new features or ideas for their educational kits and sketch the current concepts their teams were developing on ideation sheets. These sheets were collected and analyzed as a part of the study. During the final week of the course, each team presented their final prototypes; these prototypes were photographed in a standard class “photo booth”, using the same camera and set-up for every team. Teams were also instructed to submit a one-page summary of their final prototype describing the concept and the features. These photos and summaries were collected and analyzed as a part of the study.

3.4.1 Participants

Participants were recruited from a third-year engineering design course within the mechanical engineering department at the Pennsylvania State University. The course used in the

52 study is a hands-on engineering design course that teaches the fundamentals of engineering design through a semester-long project. In total, 25 females and 203 males participated in the study; design teams were composed of three to four students with no group exceeding four.

3.4.2 Procedure

At the start of the experiment, a brief overview of the study was provided, and informed consent was obtained. At times one, two, and three participants were asked to complete an idea generation activity, where they were given the following task prompt:

“Teachers are always looking for new and exciting ways to engage their students in the classroom. As low-cost 3D printers are becoming more widely available and easy to use, the opportunities to engage students in hands-on activities in the classroom are profound. The availability of 3D printing for in classroom use can redefine not only what is taught but how students learn.

Design an educational activity that leverages 3D printing to create a novel and engaging hands-on experience in the classroom. Be sure to write each solution on a different piece of paper and use drawings to sketch your ideas. It’s important you do your best and continue working for the full time of the activity.”

Participants were then given 15 minutes to complete the brainstorming activity and were provided with as many idea sheets as necessary. At the conclusion of the 15 minutes, students were instructed to put both their student ID and team ID in the bottom corner of each sheet and pass the sheets to the instructors. These sheets were collected and analyzed using a set of ideation metrics

(see Section 3.4.3).

At Time 4, students presented their final prototypes to their sections. At the conclusion of their final presentations, each team brought their prototype over to one of the research team

53 members who photographed the prototype at three different views: (1) top, (2) right, and (3) isometric. Students were also instructed to upload one-page summaries about their final design and final prototype to the course site prior to the conclusion of the class. All data was collected and reviewed by two independent raters once the course had concluded.

3.4.3 Error! Reference source not found.Application of Existing Metrics

The procedure for scoring the concepts was as follows. The same raters assessed all metrics for a concept set (one concept set is equivalent to the concepts sketched in one section of the course), and the rating process was spread out over a series of work sessions to minimize fatigue.

Inter-rater reliability for each metric was found to be 0.8 or higher. All concepts were grouped by team and section, for example, Concept #T2-1-2-4 would refer to the fourth concept produced by

Team 2 in Section 1 at Time 2. As a part of their training, the evaluators first analyzed a sample set of concepts together to become familiar with the rating procedure. This sample set was a sub- set of the larger concept set and was chosen randomly. The results were discussed in detail with the experiment leader. After the training set analysis and discussion, the raters worked independently to analyze the remaining concepts. The following four metrics were used to evaluate concepts in this study.

Originality

Concept originality was determined using Dean et al.’s originality metric [6], which defines originality as “the degree to which an idea is not only rare but is also ingenious, imaginative, or surprising”. It is important to note that this measure is somewhat relative and only provides data about the rarity of the idea with respect to the data set. Unlike the SVS method, however, it is not

54 completely relative in that raters are asked to also rate the degree to which the idea is ingenious, imaginative, or surprising overall. A four-point Likert-type scale was used to evaluate originality, as suggested by Dean et al., where four was an “idea or feature not expressed before” or “ingenious, imaginative or surprising” and a one was “common, mundane, or boring”.

Effectiveness

The effectiveness of each concept was determined using Dean et al.’s effectiveness metric

[6], which defines effectiveness as “the degree to which the idea will solve the problem”. A four- point Likert-type scale was used to evaluate effectiveness, as suggested by Dean et al., where a four was “reasonable and will solve the stated problem” and a one was “solves an unrelated problem (it would not work even if you could do it)”.

Implementability

The implementability of each concept was determined using Dean et al.’s implementability metric [6], which defines implementability as “the degree to which an idea can be easily implemented”. A four-point Likert-type scale was used to evaluate implementability, as suggested by Dean et al., where four was “easy to implement at low-cost or non-radical changes” and a one was “totally infeasible to implement or extremely financially non-viable”.

Applicability

The applicability of each concept was determined using Dean et al.’s applicability metric

[6], which defines applicability as “the degree to which the idea clearly applies to the stated

55 problem”. Dean et al. argue that applicability is a sub-dimension of relevance, which describes an idea’s ability to satisfy the goals set by the problem solver. A four-point Likert-type scale was used to evaluate applicability, as suggested by Dean et al.; where four was “solves an identified problem that is directly related to the stated problem” and a one was, “intervention is not stated or does not produce a useful outcome”.

3.4.4 Research Questions, Data Analysis, and Results

The purpose of this pilot study was to investigate the discriminatory value and reliability of ideation metrics when applied during concept development to sketches and prototypes. Our primary research objective can be separated into two research questions:

RQ1: Do Dean et al.’s metrics discriminate clearly between designs created during the concept development phase in terms of their originality, effectiveness, implementability, and applicability?

RQ2: Do Dean et al.’s metrics reliably evaluate the originality, effectiveness, implementability, and applicability of as sketches and prototypes evolve during the concept development phase?

In this research context, discriminatory value refers to the ability of the metrics to differentiate or distinguish between the concepts developed or created at different points in time. For example, the originality metric should enable designers and practitioners to differentiate between more and less original concepts developed at each time point. If there is no variability present in the data, it

56 is an indication that the metric in question is not a good discriminatory measure of the proposed construct.

Reliability refers to a metric’s ability to consistently capture informative data regarding a specific construct (i.e., originality, effectiveness, implementability, applicability in this study). We hypothesize that because the sketching sessions occurred after concept selection had taken place and during prototyping activities, the core principles or essential natures of the concepts did not change dramatically. Thus, we would expect originality scores to remain essentially the same across the experimental timespan, as originality is a relative measure of the rarity of a concept within the data set. We would also expect applicability scores to remain essentially the same, as this is a measure of the degree to which the concept applies to the problem statement. If the core of the concept remains the same, its applicability to the problem should not shift significantly.

In contrast to originality and applicability scores, we expect scores in effectiveness and implementability to increase over time within this research context. Effectiveness is a measure of the degree to which the solution solves the problem, and implementability is the degree to which the solution can be easily implemented. We hypothesize that as teams moved through concept development and into their prototyping activities, they were elaborating on the features and details of their designs in order to solve the initial problem statement more effectively. This would theoretically lead to an increase in effectiveness scores. We also argue that because this course emphasized creating prototypes that could enter the market, student teams were (by default) refining and iterating on concepts during prototyping activities to increase the implementability of their designs.

To answer our first research question, we created box and whisker plots for each metric to highlight the distribution of scores for concepts at each time point. Evaluating the distribution of scores at each of the time points will allow us to understand the discriminatory value of each metric.

In other words, using this analysis and data we can assess the ability of these ideation metrics to differentiate between concepts created during the concept development phase of design.

To explore the reliability of these metrics, we calculated Spearman’s rank-order correlations and ran Friedman tests with post-hoc analysis [36] to assess the relationship between each of the metrics at each of the time points. The correlation coefficients provide us with data regarding the existence of a linear relationship between metrics at the various time points and with the characteristics of the final prototype. As previously stated, we expect originality and applicability to remain essentially the same across the experimental timeframe, while we expect the effectiveness and applicability scores to increase. As a result, we would expect strong positive correlations to exist between the originality and applicability scores of a concept at Times 1-3 and the originality and applicability of the final prototype. Likewise, we would expect moderate correlations for effectiveness and implementability at those time points. While we do expect scores to change for effectiveness and implementability, the scores should still be related. In other words, if a concept scores high in effectiveness at Time 1, then we would expect it to continue to score highly (with potential increases) at Times 2 and 3 and for the final prototype.

Friedman tests paired with post hoc analysis were also used to determine if any significant differences existed between distributions of scores at each of the time points. This analysis enables us to determine if significant differences exist between the distributions of scores at each of the time points. We would expect the distributions of scores for each of the metrics to be fairly similar; so, we do not expect to find significant differences between distributions at each of the time points.

Originality

The originality of each concept was calculated at each time point. Because each team submitted only one prototype, yet they generated several ideas at Times 1, 2, and 3, the average originality score for each team was calculated for Times 1, 2, and 3. This was done by summing the originality scores for a team at each time point and dividing the sum of originality scores by the total number of concepts sketched by the team at that time point. Therefore, each team had one score for originality at each of Times 1, 2, and 3 that was used in our analysis.

A Friedman test was run to determine whether there were differences in the distributions of originality scores at Times 1, 2, and 3, and for the final prototype. Pairwise comparisons were performed with a Bonferroni correction for multiple comparisons. The distributions of originality scores were significantly different at the different time points, χ2 (2) = 12.00, p = 0.007. Post hoc analysis revealed statistically significant differences in originality scores between Time 2 (Mdn. =

1.33) and the final prototype (Mdn. = 2.00) (p = 0.004), and between Time 3 (Mdn = 1.33) and the final prototype (p = 0.02). There were no significant differences between the distributions of originality scores at Times 1 and 2, Times 1 and 3, Times 2 and 3, Time 1 and the final prototype, or Time 3 and the final prototype. Figure 3-2 shows the distribution of originality scores at each time point and for final prototypes.

Figure 3-2. The distribution of originality scores at Times 1, 2, and 3.

Using Spearman’s correlation, we created a correlation matrix for the originality scores at each time point to more clearly understand the relationship between originality scores as the design process proceeded (see Table 3-1). There were weak to moderate correlations between originality scores at Times 1, 2, and 3. There was no correlation between originality at any of the time points and the final prototype.

Table 3-1. Correlation Matrix for Originality Scores.

Time 1 Time 2 Time 3 Final Proto. Time 1 Cor. Coef. 1.00 Sig. Time 2 Cor. Coef. 0.360* 1.00 Sig. 0.002 Time 3 Cor. Coef. 0.405* 0.387* 1.00 Sig. 0.000 0.001 Final Proto. Cor. Coef. 0.009 0.001 0.051 1.00 Sig. 0.941 0.990 0.666

From the correlation matrix and the score distributions, we note that while originality scores were generally related to one another at Times 1, 2, and 3, some interesting phenomena occur with the final prototype. First, the distribution of originality scores for the final prototype was significantly different than the distributions of scores at Times 2 and 3. This is an interesting finding, as we would expect that the closer the time points are to one another, the more related the scores would be (because there would be less time for dramatic changes to be made in the design).

We would then expect Time 3 and the final prototype to be highly correlated and have similar distributions for originality scores. By the third time point, the designs were in the final stages of development, and most teams were not significantly changing their concepts from Time 3 to the final prototype. We also observe, from the correlation matrix, that there is no correlation between the originality scores for the final prototype and the originality scores for Times 1, 2, and 3. This is also an interesting finding, and while we can draw no final conclusions, we hypothesize that the originality of the final prototypes is not being captured in the same way as the originality of the concepts at Times 1, 2, and 3. These results indicate that while originality metrics reliably assessed the originality construct from Time 1, 2, and 3, it did not reliably asses the originality construct of the final prototype. While further work is needed to fully understand these findings, the results from our pilot study suggest that the originality metric may not be well suited to reliably capture the originality of ideas generated during prototyping after concept selection.

Looking more closely at Figure 2, we see that while a reasonable distribution of scores exists for Time 1 and the final prototype, we do not see reasonable distributions for Times 2 and 3.

Instead, the distributions for both times are clustered from 1-2.3. We also note that none of the concepts or final prototypes scored higher than a 3 in originality. As we previously noted, Dean et al.’s measure of originality is a measure of more radical creativity, emphasizing the uniqueness or unexpectedness of the concept. While it is a partially relative to the data set, in that raters are evaluating the rarity of the concept in the context of the concept set, it also asks raters to rate, not

61 only the rarity, but the degree to which the idea is imaginative or ingenious. We hypothesize that these additional layers to Dean et al.’s originality construct contributed to the cap of originality scores at 3. Dean et al.’s originality metric did provide discriminatory data about the originality of concepts at Time 1 and final prototypes, but the discriminatory value of the metric for Times 2, and

3 is fairly low. Further work is needed to understand how to measure the originality constructs more reliably across time.

Effectiveness

The effectiveness of each concept was calculated at each time point. Because each team submitted only one prototype, but they sketched concepts at Times 1, 2, and 3, the average effectiveness score for each team was calculated for Times 1, 2, and 3. This was done by summing the effectiveness scores for a team at each time point and dividing the sum by the total number of ideas at that time point. Therefore, each team had one score for originality at Times 1, 2, and 3, respectively.

A Friedman test was run to determine whether there were differences in the distributions of effectiveness scores at Times 1, 2, and 3, and for the final prototype. Pairwise comparisons were performed with a Bonferroni correction for multiple comparisons. The distributions of effectiveness scores were significantly different at the different time points, χ2 (2) = 11.08, p =

0.011; however, post hoc analysis revealed that no statistically significant relationships existed between the effectiveness metrics at the different time points.

Figure 3-3. The distribution of effectiveness scores at Times 1, 2, and 3.

Using Spearman’s correlation, we created a correlation matrix for the effectiveness scores at each time point to more clearly understand the relationship between effectiveness scores as the design process proceeds (see Table 3-2). There were moderate to strong correlations between the effectiveness scores at Times 1, 2, and 3. There was no correlation between effectiveness at any of the time points and the final prototype.

Table 3-2. Correlation Matrix for Originality Scores.

Time 1 Time 2 Time 3 Final Proto. Time 1 Cor. Coef. 1.00 Sig. Time 2 Cor. Coef. 0.463* 1.00 Sig. 0.000 Time 3 Cor. Coef. 0.580* 0.719* 1.00 Sig. 0.000 0.000 Final Proto. Cor. Coef. -0.063 -0.012 -0.192 1.00 Sig. 0.596 0.923 0.117

Comparing our findings from the Friedman test with the correlation matrix provides interesting insight into the data. While the Friedman test showed no significant differences between the distributions of effectiveness scores at Times 1, 2, 3, and the final prototype, the Spearman correlations show that no correlations exist between effectiveness scores at any of the time points and the final prototype. We do see moderate to strong correlations between effectiveness at Times

1, 2, and 3. From these results, it is clear that something is happening to the evaluation of effectiveness for the final prototype that is not occurring at Times 1, 2, and 3. Further work is needed to understand this phenomenon. We hypothesize that the effectiveness construct shifts between sketches and physical representations, such that the effectiveness construct is not being reliably assessed for the final prototype.

Implementability

The implementability of each concept was calculated for each time point and for the final prototype for each team. After reviewing the scores, we quickly realized that this metric was not effective for the evaluation of prototypes or concepts beyond the initial concept generation phases.

By the third time point and final prototype, each concept was rated as a 4 in implementability.

Rating the final prototypes for the “degree to which an idea can be easily implemented” was not appropriate in this context, as each final prototype was effectively implemented. The fact that a physical model had been created, in addition to any supplementary material developed by the students (e.g., directions, websites, etc.), indicated that the concept could be implemented. We did not run Spearman’s correlation because the data violated the assumption of a monotonic relationship between variables. The box plots and distributions for implementability were uninterpretable due to the skewed data and so no further analysis was conducted.

The implementability metric should be adjusted to gather more valuable insights about the final prototype in terms of its implementation. For example, by adjusting it to reflect the ease of production, the implementability metric might provide more useful information about the prototype, since a part of implementing a product at scale would (theoretically) include production processes. Implementability could also be adapted to prototyping by examining the ease with which iterations or changes to the prototype could be created or the ease with which the prototype could be tested. We argue that the implementability of a prototype is a valuable characteristic; however, using a traditional ideation metric to evaluate this characteristic does not appear to be effective when applied during prototyping.

Applicability

The applicability of each concept was calculated for each time point and for the final prototype for each team. Similar to the implementability construct, by the third time point, the raters realized that each concept was receiving a four in this category. We did not run Spearman’s correlation because the data violated the assumption of a monotonic relationship between variables. Similar to the implementability metric, the box plots and distributions for applicability were uninterpretable due to the skewed data, and so no further analysis was conducted.

Because this study was conducted over the course of a semester-long project, in which students were graded upon their ability to solve the original problem statement, each team’s concept applied directly to the original problem statement. In addition to the final prototype, each team also submitted a one-page summary that clearly defined their own problem statement and how their team solved this problem with their prototype. Each of the teams’ problem statements was found to be directly related to the original problem statement. As such, this metric does not provide useful information regarding the applicability of concepts. Because each concept received a score of 4 at

65 the third time point and at the final prototype, and most concepts received a 3 or higher at Times 1 and 2, one could argue that the applicability metric was reliably assessing the applicability of the concepts across time. The discriminatory value of this measure needs to be further investigated, as it provided no informative data about the ideas within the set.

3.4.5 Discussion and Implications for Evaluations of Prototypes

Intuitively, we understand that as a design evolves from ideas to concept sketches to prototypes, the design team should be learning and gathering insights about the design, and the concept itself should be changing in various ways as the team discovers limitations and flaws in their original idea. As we applied Dean et al.’s four metrics to the concepts and final prototypes of student teams, we realized that some of these metrics were able to provide reliable and meaningful information regarding sketched concepts, but these metrics often failed to provide reliable or meaningful information about the physical prototype.

For example, we found inconsistent scores for prototype originality or effectiveness, and there were only weak correlations for the originality scores of the concepts across Times 1, 2, and

3. We also saw that the implementability or acceptability metrics did not differentiate between the implementability and acceptability of the concepts; each concept received the same score at Times

3 and the final prototype, and scores varied only slight at Times 1 and 2.

Figure 3-4. A selection of student work highlighting the evolution of one concept from Time 1 to the Final Prototype.

In order to highlight the impact that inadequate metrics could have on a design, we include an illustrative example of one design and its scores in originality and effectiveness across time points.

Figure 3-4 shows the evolution of this concept from Time 1 to the final prototype for the design challenge studied in this work (develop an educational kit that in some way utilizes 3D printing).

This particular concept was intended to help second-year engineering students in a typical strength of materials class by teaching students about the strength of different beam geometries. In this proposed activity, students design and 3D print their own beam geometries and test the strength of these geometries by inserting the beam into the jig (pictured in the photo of the final prototype).

The students then load the beam with a single point force in the center of the beam. This force is measured with a strain gage and read on an accompanying digital scale (the black electronics box pictured in the photo).

Looking at each sketch and the final prototype, the core of the solution remained the same across the experimental timeframe, while the features and details of the design were further refined and developed at each time point. Intuitively, these concepts should have received similar scores on originality, but they did not. Instead, at Time 1, this concept received an originality score of 1, at Time 2 it received a score of 3, at Time 3 it received a score of 2, and the final prototype received

67 an originality score of 2. This variation indicates that while the originality metric might be working at any one point in time, it does not capture the evolution or dynamic nature of the concept’s originality as it evolves through prototyping. We saw this reflected in the correlation matrix (Table

1), where there were only weak correlations between Times 1, 2, and 3 and no correlations between time points and the final prototype.

Similarly, the effectiveness ratings for this concept should have increased with time, as the concept became better defined. However, we see that at Time 1, this solution received an effectiveness score of 4, at Times 2 and 3 it received a score of 2, and the final prototype received a score of 4. Clearly, these ratings are not measuring effectiveness reliably. While they might be measuring some characteristics about the concepts and prototypes, without knowing exactly what is being measured, this information is of limited value.

From previous work in prototyping, we know that prototypes can help design teams make decisions related to manufacturing processes, materials, sub-system design, interface design, tolerances, usability, technical quality, and many other areas; however, information related to these topics was not captured in the evaluations using traditional ideation metrics. Information about the concepts and prototypes is critical to design teams as they navigate through the design process, specifically during prototyping activities; if a set of metrics existed that provided teams with insights into their design’s development, then teams might be able to more quickly identify when they are headed down a futile (or promising) path.

3.4.6 The Case for Alternative Metrics to Evaluate Prototypes

After reviewing many of the concepts and the scores that each of these concepts received, it is clear that traditional ideation metrics cannot be used to evaluate prototype evolution in new product development. We found that many of the metrics could not reasonably be applied to final

68 prototypes and the few metrics that were applied did not provide any valuable insights about the design. This work highlights the importance of creating or adapting metrics to evaluate prototypes.

By evaluating prototypes against a relevant set of metrics, designers and design teams could gain valuable knowledge about how to further iterate or evolve their concept to increase the likelihood of success in the market.

We adapted metrics from the domains of design theory and methodology and market research to the context of prototypes to evaluate the effectiveness of PFX on a final design’s desirability, feasibility, and viability. Specifically, desirability was linked to user satisfaction and perceived value metrics from the management science and market research literature [72,160]; feasibility was linked to technical quality and feasibility metrics from the ideation literature

[117,146]; and viability was linked to manufacturability metrics from the design for manufacturability and assemble-ability literature [123].

3.4.7 Desirability Metrics: User Satisfaction and User Perceived Value

Evaluating a prototype’s potential user satisfaction and perceived value could provide critical information about future iterations and changes that may need to be made to a product as it is being designed. To evaluate user satisfaction, we used the Delighted-Terrible (DT) scale [72], which Westbrook proved to be valid in a study with consumer products. The single-item DT scale asks users to rate how they feel about a product (or, in our case, a prototype) on a spectrum, with seven distinctive adjectives as anchors: (1) terrible, (2) unhappy, (3) mostly dissatisfied, (4) mixed,

(5) mostly satisfied, (6) pleased, and (7) delighted. Because desirability is inherently a measure of preference, a second scale was used to confirm and support this assessment. This second scale is the PERVAL scale of Sweeney and Soutar [160], which was used to evaluate the perceived value of the prototypes. Sweeney and Soutar’s 19-item scale assesses each of Sheth et al.’s [99] five

69 types of perceived value: 1) social, 2) emotional, 3) functional, 4) epistemic, and 5) conditional. In further validation studies, Sweeney and Soutar found that the functional value subscale (6 items) and emotional value subscale (5 items) were better predictors of user behavior in the context of consumer goods [160]; therefore, we used only those two portions of the PERVAL scale in this study.

3.4.8 Feasibility Metric: Technical Quality

In previous studies of ideation methods, quality has been defined as “a measure of a concept’s feasibility and how well it meets the design specifications” [146]. This definition of quality aligns with our definition of feasibility, and we use this definition as the basis for the feasibility metrics created to evaluate prototypes. Toh et al. [146] measured quality with an anchored multipoint scale that resulted in a quality score based on the feasibility and technical functionality of the design. While this scale is appropriate for evaluating the technical feasibility of conceptual designs, we were able to use a more concrete measure of technical quality in evaluating our physical models. In particular, we used an approach based on Yang’s metric for rating task completion [14] to evaluate each final design’s technical quality.

The main design requirement in this study was to develop a functioning hand-held vacuum, where “functioning” was evaluated by how much rice the vacuum could gather in 10 seconds. We recognize that this specific measure of technical quality could not be applied to all prototypes. That said, we question the practicality of a general “technical quality” metric, given that the technical purpose of a prototype will vary depending on the assumption or function being tested. Design teams will need to identify the core function of the desired design and develop metrics specific to this core function. Functional decomposition methods [132] or a functional basis [63] can be

70 adapted and used to determine the main functionality of a prototype, and subsequently, the most relevant metrics to use for its evaluation.

3.4.9 Viability Metric: Manufacturability

Improved manufacturability, which we use to evaluate viability in this study, has been shown in multiple case studies to lead to direct savings in terms of time and cost. Companies like

Honda and Ingersoll Rand have used DFMA methods for years to decrease product costs and time to market [123,161]. In order to evaluate the manufacturability of each prototype, the critical part count ratio was calculated for each design. Critical part count ratio is a measure of the theoretical minimum part count over the design’s actual part count [123]. Participants calculated the theoretical minimum number of parts for their designs using the criteria from Boothroyd and Dewhurst [123]:

1. During the normal operating mode of the product, the part moves relative to all other parts already assembled. (Small motions do not qualify when they can be obtained through the use of elastic hinges).

2. The part must be of a different material or be isolated from all other parts assembled (for insulation, electrical isolation, vibration damping, etc.)

3. The part must be separate from all other assembled parts; otherwise, the assembly of parts meeting one of the preceding criteria would be prevented.

Traditionally, these criteria are used along with many other metrics to conduct an in-depth review of components, parts, and manufacturing processes, resulting in a final design optimized for cost and time in terms of manufacturing and assembly. However, for our purposes, we only used these three guidelines and developed the critical part count ratio as a faster and more efficient

71 alternative to evaluate prototypes for manufacturability, providing designers with a close estimate or approximation of overall manufacturing burden.

3.4.10 Assessment Summary

Each of the four metrics described in the previous subsections was used to evaluate final designs in a classroom study involving junior-level mechanical engineering students in a design course. To summarize, user satisfaction and perceived value were evaluated by two independent raters using a combined seven-point Likert-type scale. Technical quality was evaluated by measuring the weight of rice collected (in pounds) in a ten-second time span. Manufacturability was measured by calculating the critical part ratio for each final design. Table 3-5 summarizes each of the metrics used in this study.

Table 3-3. Correlation Matrix for Originality Scores.

Metric Measure Reference

User Satisfaction 7 point Likert-type Scale (1 item) Westbrook [72]

User Perceived Value 7 point Likert-type Scale (19 items) Sweeney and Soutar [160]

Technical Quality Pounds of rice collected in 10 seconds Yang et al. [14]

Manufacturability Critical Part Ratio Boothroyd & Dewhurst [123]

3.5 Research Hypotheses

We hypothesize that using PFX throughout the design process will lead to significant improvements in desirability, viability, and feasibility of the end designs. Specifically, we believe that the final design’s perceived value and user satisfaction will increase compared to unstructured

72 prototyping practices as a result of the Prototype for Desirability method. This hypothesis is based on previous research that indicates positive relationships between user feedback in early stage prototype development and user satisfaction [7,48,100]. We also hypothesize that using PFX will increase the technical quality of the final design based on previous research that indicates positive relationships between prototyping strategies and technical quality [16,19,23]. Finally, we hypothesize that the viability of the final design will be improved through the PFX framework, specifically the manufacturability of final designs will be improved. This hypothesis is based on previous research that indicates positive relationships between DFMA methods and final manufacturability of components and designs [119,123].

3.6 Experimental Protocol

To evaluate the impact of PFX on design outcomes, a between subjects experiment was conducted, comparing the prototypes from PFX and control conditions. Figure 3-5 gives a general overview of the experimental flow of the study, including the different activities used in the control and experimental classes.

Figure 3-5. Overview of Experiment Flow

All participants were given the following project statement: ACME Tool Company has a product family of 18V cordless drills, saws, and sanders that have been very successful in the consumer market. Their marketing department recommends expanding the product line to include a cordless handheld vacuum. ACME routinely outsources their engineering work to the lowest bidder. The product will be produced in their factory outside Shanghai. They have invited several product development firms from around the world (including your company) to compete in the design and construction of a demonstration prototype. A jury consisting of corporate executives, typical customers, and investors will judge your design based on its economic potential, aesthetics, ergonomics, and performance. The winning firm will be awarded a lucrative production contract.

Both classes were presented with the same deadlines and course and resource constraints; participants in the experimental class received PFX interventions from Week 8 of the semester to

Week 14 (semesters are 16 weeks long), while participants in the control class were provided with no instructions for specific prototyping methods. Final designs from both classes were due during

Week 16 of the semester. Examples of final designs from control and PFX classes are shown in

Figure 3-6.

Figure 3-6. Example Prototypes from the (A) Control and (B) Experimental Classes

Participants in the experimental class were taught each prototyping method in a two-week learning module based on one of the three PFX lenses. Lectures ranged from 45-90 minutes and covered tools and techniques specific to each PFX method, such as Design for Assembly methods

[119] (for viability), functional decomposition [132] (for feasibility), and Design Thinking strategies [121] (for desirability). Although functional decomposition and DFMA are typically taught in engineering design courses, the key difference in PFX was using these existing models to frame and help focus participants’ “build” efforts. For example, once functional decomposition was complete, participants were instructed to use an Analytical Hierarchy Process [134] to determine the key or core sub-function. Participants were then instructed to build a functional prototype of only that sub-function. Once this prototype was completed, students analyzed the technical quality of this prototype and evaluated whether they could move on to another sub- function or whether re-design was needed. In this way, Prototype for Feasibility (PFF) helped participants work through the technical functionality of their designs and provided a structured and formalized prototyping method. An example of what this process might look like for a design team is shown in Figure 3-7.

Figure 3-7. Example of Prototype for Feasibility Process

During Prototype for Viability (PFV), participants were instructed to reduce the number of non-critical components and increase their design’s critical part ratio by building a new prototype

[123]. Unlike traditional DFMA lessons taught in design courses, which is typically a theoretical

“hands off” lecture, PFV focused on switching between building and analyzing, so that participants were constantly reframing their efforts to build, allowing them to readily see which designs were easier to assemble, manufacture, etc. We hypothesized that this “hands on” experience and real- world understanding of systems would help design teams identify parts to simplify and combine more efficiently than traditional methods.

3.6.1 Participants

A controlled study was conducted with 76 undergraduate engineering students from two sections of a junior-level mechanical engineering design course at the Pennsylvania State

University. Participants were tasked with creating a handheld vacuum, and final designs produced by the end of the course were assessed in the study. Design teams (our experimental unit) were comprised of three to four participants; 10 teams were formed in the experimental class (N = 10), and 15 teams were formed in the control class (N = 15).

3.7 Results

To test our hypothesis that design outcomes are impacted by the type of prototyping methods used by the designer, statistical analyses (Mann-Whitney U and independent samples t- tests) were conducted with the experimental or control condition (i.e., were PFX methods implemented) as the independent variable, and the four product metrics (user satisfaction, user- perceived value, technical quality, manufacturability) as the dependent variables.

3.7.1 User Satisfaction and User Perceived Value

A Mann-Whitney U test [162] was conducted to determine whether there were differences in user satisfaction scores on the Delighted-Terrible single-item scale and in user-perceived value on the PERVAL subscales between the experimental and control classes. The Mann-Whitney U method was chosen because it is a rank-based nonparametric test typically used to determine if there are differences between two groups on an ordinal dependent variable [162,163]. Distributions of the user satisfaction scores for the experimental and control classes were similar, as assessed by visual inspection, see Figure 3-8.

2 User User Satisfaction Score 1

0 Experimental Control

Figure 3-8. Distributions of User Satisfaction Scores in Control and Experimental Groups

The difference in median user satisfaction score was significantly higher in the experimental class (4.25) than in the control class (2.5) (U = 24, z = 2.849, p = 0.004). Distributions of the user-perceived value for experimental and control classes were also similar, as assessed by visual inspection, see Figure 3-9. The difference in median user-perceived value score was

77 significantly higher in the experimental class (4.137) than in the control class (3.406) (U = 28.5, z

= 2.58, p = 0.008).

1 User Perceived User Perceived Value MeanScore

0 Experimental Control

Figure 3-9. Distribution of Perceived Value Scores in Experimental and Control Groups (6 pt. Likert-type Scale)

The results of our analysis support our hypothesis that a final design’s perceived value and user satisfaction will increase as a result of applying the Prototype for Desirability (PFD) method compared to unstructured prototyping practices. This finding highlights the advantage of using structured prototyping methods over unstructured prototyping methods by increasing the perceived value and satisfaction from end users. For industry, increasing user satisfaction and perceived value will likely result in higher ROIs and fewer failed products.

3.7.2 Technical Quality Results

An independent-samples t-test was run to determine whether there were differences in the weight of rice collected by the final designs of the control and experimental classes. There were

78 outliers in the data, as determined by inspection of a boxplot, and the distributions of rice (in pounds) were not normally distributed. Because of these conditions, a Mann-Whitney U test was run as a non-parametric alternative statistical analysis to the independent samples t-test.

Distributions of the pounds of rice collected for the control and experimental classes were similar, as assessed by visual inspection, see Figure 3-10. The difference in the median amount of rice collected was not statistically significant between the control and experimental classes (U = 102, z

= 1.16, p = 0.246) using an exact sampling distribution for U. The effect size, d = 0.300, was moderate, indicating that we may find statistically significant results with a larger sample size in future studies.

1.4

1.2

0.8

0.6

0.4

Amount Rice of inLbs per 10 sec. 0.2

0 Experimental Class Comparison Class

Figure 3-10. Distribution of Amount of Rice Collected by Final Designs in Lbs.

Although the designs from the experimental class did not significantly outperform designs in the control class in terms of the amount of rice collected, we do believe there was a trend toward significance when considering our moderate effect size (d = .300). We believe that with a larger sample size, we might see significant results, and that our data point to practical, real-world benefits

79 from using PFX methods to increase the technical quality of final designs. These results are not surprising; as noted previously engineering design traditionally focuses on functional design and technical quality, so we would still expect to see adequate levels of technical quality in the control group. What is interesting is that shifting the focus of students in PFX sections did not detract from the technical qualities of their designs, but may have even added to the technical quality. Further research is needed to determine more specifically how PFX methods could create real benefits in terms of technical quality for industry and practicing engineers when developing new products.

3.7.2.1 Manufacturability Results

An independent-samples t-test was run to determine whether there were differences in the critical part ratios of final designs between the control and experimental classes. There were no outliers in the data, as assessed by inspection of the boxplots. Critical part ratios for the control and experimental classes were normally distributed, as assessed by Shapiro-Wilk's test (p > .05).

The critical part ratio was higher in the experimental class (M = .196, SD = 0.134) than the control class (M = .532, SD = 0.171), and this difference was statistically significant (M = 0.336, 95% CI

[0.201, 0.470], t(23) = 5.506, p >.000, d = .754). Distributions of critical part ratios for the experimental and control classes are shown in Figure 3-11.

0.9

0.8

0.7

0.6

0.5

0.4

0.3

Critical Critical CountPart Ratio 0.2

0.1

0 Experimental Class Comparison Class

Figure 3-11. Distribution of Critical Part Count Ratio for Final Designs

These results indicate that designs in the experimental group had fewer non-critical parts compared with the control group, supporting our hypothesis that a final design’s manufacturability

(reduced assembly time, in this case) will increase as a result of applying the Prototype for Viability

(PFV) method compared to unstructured prototyping practices. This highlights an advantage of using PFX methods over traditional prototyping methods in increasing the manufacturability of final designs. For industry, increasing manufacturability can decrease time to market and overall production costs, increasing the profit margin on new products [123].

3.8 Implications

The main objective in this study was to investigate the impact of applying the Prototype for X (PFX) framework on design outcomes in terms of desirability, feasibility, and viability. Our results show that participants who engaged with and implemented PFX produced end designs that were significantly more desirable and viable. The work discussed here not only evaluated the effect

81 of PFX on functional prototypes, but also highlighted an important gap in the assessment of prototypes. Research in engineering design has focused primarily on the early stages of design, such as concept generation [61,164,165] and concept selection [142,146,166], and as a result, there are an overwhelming number of ideation metrics. We also found that a number of metrics to evaluate products in the market place exist and are used across the marketing [71,72] and management science [10,31] disciplines. However, there is a significant lack of metrics that are appropriate or applicable for the evaluation of functional prototypes; while we used a compilation of simple, practical metrics here, there is clearly more work needed in this area.

The prototyping strategies used by participants and professionals remain largely unstudied, and as a result, there is a lack of tools and methods to guide participants and professionals through a more effective prototyping process. The work presented here contributes to the understanding of structured prototyping methods and how these methods can positively affect the end design’s desirability, feasibility, and viability.

3.9 Limitations

Our study was limited in its scope due to constraints placed on the experiment resulting from the nature and timing of the course. These included the following: using a sample of convenience for the intervention class (N = 10 teams); the ordering of the PFX methods could not be randomized; and the lectures, materials, and instructions in the control class were outside the influence of the primary researcher. The intervention sample was also a sample of convenience, as the primary researcher was teaching this section; this means that the effect the lecturer had on the class could not be evaluated independently and may have influenced design outcomes. In Chapter

5, we randomize control classes and experimental classes to remove this confounding variable.

3.10 A Look Back and a Look Ahead

In this chapter we demonstrated the impact that the PFX framework has on final designs.

Traditional ideation metrics were first evaluated for their applicability to functional prototypes; these metrics were found to be inappropriate for prototypes. We then reviewed the metrics used in this study to evaluate the functional prototypes designed in a junior-level mechanical engineering design course. We found that students in a class where the PFX framework was implemented produced end designs that were in general more desirable and viable, and trending towards more feasible designs. In the next chapter we explore what (if any) effects a structured and holistic prototyping framework, like PFX, might have on the designers themselves during prototyping activities.

Chapter 4

Assessing the Impact of PFX on Students’ Prototyping Awareness

4.1 Introduction

In this chapter, we attempt to answer one of our three main research questions: how are designers’ prototyping awareness affected by a structured and holistic prototyping framework?

Based on literature in engineering design and psychology, we hypothesize that structured prototyping methods such as PFX could increase novice designers’ and engineering students’ feelings of control throughout the prototyping process and could therefore lead to an increase in creative output, higher levels of motivation, and an increase in the quality of final designs. As an initial step in testing this hypothesis, we investigate the prototyping awareness of engineering students exposed to the PFX framework under different conditions. In particular, we investigated whether the engineering students’ prototyping awareness changed during the prototyping process and whether those changes were related to the ordering of PFX activities. As a means to evaluate students’ prototyping awareness, the Prototyping AWareness Scale (PAWS) was developed and distributed in a junior-level mechanical engineering design course. Development of this scale, along with results from the first administration of the instrument, are discussed in this chapter.

4.2 Self-Efficacy and Awareness in Prototyping

While most prototyping research is focused on the design artefacts produced, a handful of researchers have investigated the mental processes of those doing the prototyping, including their self-efficacy. Gerber et al. [13,20] studied the psychological experience of prototyping and found

84 that the production of low-fidelity prototypes enabled designers to learn from failure, created a sense of productivity, and increased creative self-efficacy. Results of their ethnographic study were based on observations and field notes from studying a single design team, making it difficult to generalize findings to a broader design context. Self-efficacy measures have been used in engineering design contexts as an alternative to case studies for researchers attempting to measure motivation, feelings of control, and self-doubt, at scale. Self-efficacy refers to an individual’s assessment of his/her ability to arrange and complete courses of action for a given task (Bandura,

1986; 1997).

According to Bandura’s theory of self-efficacy, the level of self-efficacy for an undertaking or task is affected by motivation, perceptions of control, and self-doubt. Individuals in uncertain environments with low perceptions of control of that environment struggle with motivation, creativity, and persistence in the face of obstacles [70,167]. Bandura [167] found that the opposite is true when individuals are faced with uncertainties yet perceive themselves to be in control of the situation. Specifically, Deci [168] showed that perceptions of high control in uncertain activities can lead to higher cognitive flexibility and creativity. Engineering design and design tasks are characterized by ill-formed problems and uncertainty [169,170]. Carberry and Ohland define design tasks as “the applied or practical component of engineering, which consists of several processes used in devising a system, a component, or a protocol to meet an identified need” [171].

By this definition, prototyping activities qualify as design tasks; thus, in keeping with Bandura’s findings [167], an individual’s prototyping self-efficacy or belief in their ability to arrange and complete prototyping actions directly affects their prototyping outcomes. We believe that a structured and holistic prototyping framework could provide designers with a greater sense of awareness and control during uncertain prototyping activities.

Carberry et al. [171] developed an engineering design self-efficacy instrument; however, only one item on their scale refers directly to prototyping activities: construct a prototype [171].

Because their instrument is intended to measure the self-efficacy of a designer throughout the entirety of the design process, the instrument does not provide any detailed information about specific design activities, such as prototyping. In contrast, Dow et al. [6] measured task-specific self-efficacy during prototyping using a self-assessment report. Specifically, Dow et al. was measuring self-efficacy related to the creation of digital prototypes, along with openness to critiques and feedback on design artefacts. The prototyping self-efficacy scale used in Dow et al.’s study was based on previous self-efficacy measures from education and asked participants to rate their abilities to create advertisements, understand design problems, detect problems in a design idea, and incorporate feedback into a design idea. Aside from Dow et al.’s measure, which focused predominately on feedback and design iterations, and Carberry et al.’s scale, which does not provide sufficient details about self-efficacy during prototyping activities, a comprehensive scale that measures self-efficacy and behavioral awareness during prototyping activities does not exist.

To address this gap, we developed the Prototyping Awareness Scale (PAWS) to measure designers’ awareness or mindfulness of their prototyping behaviors during prototyping activities.

4.3 Measuring Prototyping Awareness with the Prototyping Awareness Scale (PAWS)

Awareness or mindfulness is defined by Langer and colleagues [172,173] as “an open, assimilative “wakefulness” to cognitive tasks, in which thought is used flexibly to create new categories, draw distinctions, and seek multiple perspectives”. We believe that awareness in engineering design is the first step towards the development of self-efficacy beliefs. While previous scales related to prototyping awareness informed our work, they are inadequate for the in-depth study we need. For this reason, a new scale – the Prototyping AWareness Scale (PAWS) – was developed to measure designers’ awareness of their prototyping behaviors during prototyping activities in greater depth.

PAWS was created using Messick’s unified theory of validity [174,175] and best practices in survey design [176]. In highlighting the importance of an instrument’s validity in the context of score interpretation, Messick notes, “The construct validity of score interpretation comes to undergird all score-based inferences” [174]. Score interpretation is dependent upon the validity evidence collected for the instrument itself, making the rigor of the development process for assessment instruments of critical importance. Content relevance and representativeness, which refer to the range and limits of content coverage (i.e., the boundaries of the construct domain to be assessed), must be considered first when developing a sound instrument.

For the PAWS, we limited the content to prototyping activities relevant to the overall desirability, feasibility, and viability of the product based on our use of the Human-Centered Design innovation framework [121]. This model highlights the importance of a product’s desirability, feasibility, and viability for overall long-term success in the market. Creating a scale based on a model for product development or innovation model is in keeping with previous scale development practices, as Carberry et al. [171] used an eight stop product design process proposed by the

Massachusetts Department of Education (DoE) Science and Technology/Engineering Curriculum

Framework as the basis for their engineering design self-efficacy scale. As such, when generating items for the PAWS, we considered prototyping activities related to these domains, such as testing with users [22,79,104], testing the technical quality [14,16,67], simplifying a design for manufacture [27,123], testing the usability of a design [90,101,102], and testing the interaction of various subsystems [3,12,63].

4.3.1 PAWS: Desirability Sub-Scale

Scale items related to desirability in PAWS were generally focused on users, user interfaces, usability, and user-perceived value of the prototype, and are shown in Table 4-1. These

87 items were created by examining prototyping best practices from the fields of human computer interaction, Design Thinking, human centered design, engineering design, and interaction design.

Five items were generated that described prototyping activities and behaviors related to the desirability of a product. Item one, empathized with users, was based on Design Thinking practices

[66,106], which begin the design process with an empathy stage, encouraging designers to empathize with end users in order to truly understand the problem. The second item, thought about the user’s needs, comes from best practices in engineering design [12] and human centered design

[121]; both fields emphasize the importance of not only gathering users’ needs but reflecting upon these needs at various stages throughout new product development. The third and fourth items, reflected on user feedback about the design and built items I felt would increase user value, were created based on research from human computer interaction [8,78,177] and interaction design

[74,135]. Both fields highlight the importance of user testing and feedback, particularly when iterating on the usability of a design. Human computer interaction literature stresses that “user facing features” should add user value to the product or these features should not be built [178,179].

In order to account for any social biases and in keeping with survey development best practices, one reverse scored item was included in each subscale to account for acquiescence bias, or the tendency of respondents to agree with all the questions [180,181]. The reverse scored item, did not care about the aesthetics of the design, was based on literature from consumer psychology

[102,182], which has studied the effect of aesthetics and form on user response to products and prototypes. Table 4-1 summarizes the items and sources for these items reviewed in this section.

Table 4-1. Summary of Items for Desirability Sub-Scale

When building my prototype I… Sources Empathized with users [45,66,94,104,121] Thought about the user’s needs [12,121,131,178] Reflected on user feedback about the design [6,8,107] Built features I felt would increase user value [178,179] Did not care about the aesthetics of the design [43,102,182]

4.3.2 PAWS: Feasibility Sub-Scale

Items related to feasibility in PAWS were generally focused on technical quality, functionality, and overall feasibility of the prototype and feasibility items are shown in Table 4-2.

These items were created by examining prototyping best practices from the fields of engineering design and engineering education. Five items were generated that described the prototyping activities and behaviors related to the feasibility of a product. All of these items draw from best practices for the design of electro-mechanical physical products. It was important to make the distinction between digital and physical prototypes in our work, because the technical quality of an application or website is not evaluated in the same way as the technical quality of a gear train for example. The two items explored how subsystems would work and determined one or more technical function to test come from traditional engineering design resources like Ulrich and

Eppinger’s textbook, Product Design and Development [12], which reviews functional decomposition and black box diagrams as methods to generate sub-systems and pinpoint critical functions. The third item, thought about the technical functionality of my design, was based on literature from engineering design and mechanical design which emphasizes the importance of technical quality [132]. The fourth item, used technical knowledge to layout or design the prototype, was based on literature from engineering management and management science, which has proposed using engineering knowledge to plan out the optimal prototyping path [29,30,183].

As previously mentioned, one reverse scored item was created for each subscale, for the feasibility subscale this was, was not concerned with the functionality of the prototype. This was based on literature from engineering design and mechanical design, as was the third item. Table 4-2 summarizes the items and sources for these items reviewed in this section.

Table 4-2. Summary of Items for Feasibility Sub-Scale

When building my prototype I… Sources Thought about the technical functionality of the design [132] Explored how subsystems would work [12,63,132] Used technical knowledge to layout or design the prototype [29,30,183] Determined one or more technical function to test [12,63,132] Was not concerned with the functionality of the prototype [132]

4.3.3 PAWS: Viability Sub-Scale

Items related to viability in PAWS were generally focused on the overall manufacturability, design for scale production, and marketability of the prototype and viability sub-scale items are shown in Table 4-3. These items were created by examining prototyping best practices from the fields of design for manufacture and assembly (DFMA), marketing, and management science. Five items were generated that describe the prototyping activities and behaviors related the viability of the product. All of the items in this sub-scale are related to the economic viability and resource management needed to bring a product to the market. The two items, considered the expected return on investment my design would produce and contemplated how my design would fit in the market were drawn from best practices in marketing [71,73], where in practitioners evaluate the competitive climate that their concept or idea will enter once on the market. The two items, thought about the manufacturability of my design and did not consider the mass manufacture of the product, stem from best practices from the field of DFMA [118,119]. Boothroyd and Dewhurst, who wrote

90 the first guidelines for DFMA best practices, emphasize the importance of designing for manufacture in the early stages of product development, including prototyping [119]. The final item in the PAWS viability subscale, thought about which available resources could be used to bring the design to market, was based on literature from management science that reviews optimal resource management for faster progress through concept creation to market entry [27,31]. Table

4-3 summarizes the items and sources for these items reviewed in this section.

Table 4-3. Summary of Items for Viability Sub-Scale

When building my prototype I… Sources Thought about the manufacturability of my design [118,119] Contemplated how my design would fit in the market [71,73] Considered the expected return on investment my design would produce [71,73] Thought about which available resources could be used to bring the design to market [27,31] Did not consider mass manufacture of the product [118,119]

4.3.4 Summary of the PAWS

In completing the Prototyping AWareness Scale (PAWS), users rated their agreement with the fifteen statements in the subscales on a five-point Likert-type scale (1 = strongly disagree, 5 = strongly agree) as they consider the processes, behaviors, and mindsets they engaged in during the prototyping process. The entire scale is shown in Appendix E. Before beginning the scale, students were provided with the following instructions: Please answer each of the following questions based on your mindset or thought process as you prototyped. No response is “right” or “wrong.” Your answers are confidential, and your participation is voluntary.

Overall, the PAWS exhibited a Cronbach's alpha of 0.723, indicating a high level of internal consistency. The desirability and viability awareness constructs were found to have a high level of internal consistency as determined by a Cronbach's alpha of 0.748 and 0.808, respectively.

The feasibility awareness construct was found to have a moderate level of internal consistency as determined by a Cronbach's alpha of 0.602. We recognize this moderate level of internal consistency for the feasibility subscale as a limitation of our study and discuss this in Section 4.7.

4.4 Research Hypotheses

The experiment described in this chapter was guided by the following research question: does a structured prototyping framework influence the prototyping awareness of engineering students? We conducted a controlled experiment in a junior-level mechanical engineering design course in which the Prototype for X framework was implemented. Our hypotheses with respect to prototyping awareness and PFX are as follows:

 H1: Engineering students exposed to the PFX methods will be more aware of

prototyping methods than students who are not exposed to these methods (control).

 H2a: Engineering students will be more aware of desirability prototyping methods

after being exposed to PFD.

 H2b: Engineering students will be more aware of feasibility prototyping methods

after being exposed to PFF.

 H2c: Engineering students will be more aware of viability prototyping methods

after being exposed to PFV.

The experimental protocol used to test these hypothesis is described next.

4.5 Experimental Protocol

PAWS was administered four times in the experimental class, once after each alpha prototype was due and once when the final beta prototype was due. PAWS was administered once

92 in the two control classrooms when the final beta prototype was due; because the main researcher did not have control over these two sections, PAWS could only be administered at the conclusion of the course. Results from all four distributions of the scale in the experimental classroom were compared in order to evaluate how prototyping awareness changed with respect to time and the relevant PFX lens. Results from the final distribution of the PAWS from the experimental classroom are compared with results from PAWS in the two control classes in order to evaluate the difference in awareness between control and experimental classrooms at the conclusion of the course.

Figure 4-1 gives a general overview of the experiment with respect to time. The ordering of the PFX methods was not randomized in this pilot study, and a sample of convenience was used; however, in the second study, presented in Chapter 5, the ordering of PFX lenses was randomized to eliminate sequencing or time effects as a potential variable and a large sample size was used.

Figure 4-1. Overview of Experiment Flow

Students in the experimental class were taught each prototyping method in a two-week learning module that was based on one of the three PFX lens. Lectures ranged from 45-90 minutes and covered tools and techniques specific to each PFX lens, such as DFMA methods for viability,

93 functional decomposition for feasibility, and Design Thinking strategies for desirability. Students were instructed to build and test prototypes optimized with respect to each lens. For example, during the Prototyping for Viability module, students were instructed to increase their design’s critical part ratio by building a new prototype using fewer non-critical components; critical part ratio is a metric used in DFMA that describes the theoretical minimum number of parts to total number of parts [25,161].

4.5.1 Participants

Participants were juniors in mechanical engineering at the Pennsylvania State University, and the experiment took place in the latter half of the semester, when students began to build and test prototypes. Two samples are reported in this chapter: (1) Sample A was composed of 30 students from the experimental class, and (2) Sample B was composed of 60 students, 30 from each of the two control classes.

4.6 Results

4.6.1 Within Subjects Experiment

A Friedman test [163] was run on each subscale to determine if there were differences in prototyping awareness throughout the implementation of the PFX lenses. Pairwise comparisons were performed using SPSS 2012 with a Bonferroni correction for multiple comparisons [163].

Desirability prototyping awareness was significantly different at the four different time points throughout the PFX methods, χ2(2) = 24.611, p < .000. The distribution of desirability prototyping awareness scores at all four time point is shown in Figure 4-2. Post-hoc analysis revealed

94 statistically significant differences in desirability awareness from Time 2 (Mdn = 3.5) to Time 4

(Mdn = 4.2) (p < .000) and from Time 3 (Mdn = 3.6) (p = .007) to Time 4. There was not a statistically significant difference in desirability awareness between Time 1 and Time 4, Time 1 and Time 3, or Time 1 and Time 2.

Feasibility prototyping awareness was statistically significantly different at the four different time points throughout the PFX methods, χ2(2) = 51.05, p < .000. The distribution of feasibility prototyping awareness scores at all four time point is shown in Figure 4-3. Post-hoc analysis revealed statistically significant differences in feasibility awareness from Time 1 (Mdn =

3.0) to Time 2 (Mdn = 4.4) (p < .000), Time 1 to Time 4 (Mdn = 4.8) (p < .000), and Time 3 (Mdn

= 4.0) (p < .000) to Time 4. There was not a statistically significant difference in feasibility awareness between Time 1 and Time 3, Time 3 and Time 2, or Time 2 and Time 4.

Viability prototyping awareness was statistically significantly different at the four different time points throughout the PFX methods, χ2(2) = 38.83, p < .000. The distribution of viability prototyping awareness scores at all four time point is shown in Figure 4-4. Post-hoc analysis revealed statistically significant differences in viability awareness from Time 1 (Mdn = 2.2) to

Time 3 (Mdn = 3.7) (p < .000), Time 1 to Time 4 (Mdn = 3.6) (p < .000), Time 2 (Mdn = 2.2) (p <

.000) and Time 3, and Time 2 and Time 4 (p < .000). There was not a statistically significant difference in viability awareness between Time 1 and Time 2 or Time 3 and Time 4.

2 Mean Score on Desirability participant per Subscale Desirability on Score Mean 1 Time 1 Time 2 Time 3 Time 4

Figure 4-2. Mean Desirability Subscale Score at T1, T2, T3, and T4

2 Mean Score on Feasibility Subscale per particioant per Subscale Feasibility on Score Mean

1 Time 1 Time 2 Time 3 Time 4

Figure 4-3. Mean Feasibility Subscale Score at T1, T2, T3, and T4

2 Mean Score on Viability Subscale per participant per Subscale Viability on Score Mean

1 Time 1 Time 2 Time 3 Time 4

Figure 4-4. Mean Viability Subscale Score at T1, T2, T3, and T4

By reviewing Figures 4-2 to 4-4 along with the results from statistical analysis, the following observations can be made in relation to our first hypothesis.

 H2a: Engineering students will be more aware of desirability prototyping methods

after being exposed to PFD. There were statistically significant results comparing

Time 1 (the time PFD was implemented) with Time 2 and Time 3, but we did not

see a statistically significant difference between Time 4 and Time 2 and Time 3.

This means that students were more aware of desirability during the first prototype

construction (Time 1) as compared with the second and third alpha prototype

construction (Time 2 and Time 3, respectively). These findings support our

hypothesis that engineering students will be more aware of desirability prototyping

methods after being exposed to PFD. In other words, students were able to

successfully shift awareness from Time 2 and Time 3, and they focused on—or

were more aware of—the desirability of their design at the first time (Time 1).

 H2b: Engineering students will be more aware of feasibility prototyping methods

after being exposed to PFF. We found statistically significant differences on the

feasibility awareness subscale at Time 2 compared to Time 1 and Time 3. We also

saw a statistically significant difference between Time 4 and Time 1 and Time 3.

This means that engineering students were significantly more aware of feasibility

during the second alpha prototype (Time 2) as compared with the first and third

alpha prototypes (Time 1 and Time 3 respectively). This also means that the

students were significantly more aware of feasibility during the final prototype

construction (Time 4) as compared with the first and third alpha prototypes. These

findings support our hypotheses that students will be more aware of feasibility

prototyping methods after being exposed to PFF. In other words, the engineering

students were able to successfully shift awareness from Time 1 and Time 3 and

focus on feasibility during construction of the second alpha prototype (following

the PFF learning module). The students were also able to shift back to feasibility

for the final prototype.

 H2c: Engineering students will be more aware of viability prototyping methods

after being exposed to PFV. We found statistically significant differences on the

viability awareness subscale at Time 3 compared to Time 1 and Time 2. We also

saw a statistically significant difference between Time 4 and Time 1 and Time 2.

This means students were statistically significantly more aware of viability during

the third alpha prototype (Time 3) as compared with the first and second alpha

prototypes (Time 1 and Time 2 respectively). This also means that students were

statistically significantly more aware of viability during the final prototype

construction (Time 4) as compared with the first and second alpha prototypes.

These findings support our hypotheses that students will be more aware of viability

prototyping methods after being exposed to PFV. In other words, students were

able to successfully shift awareness from Time 1 and Time 2 and focus on viability

during construction of the third alpha prototype (following the PFV learning

module). The students were also able to shift back to viability for the final

prototype. Reviewing Time 4 more closely reveals that students were statistically

significantly more aware across all three dimensions—desirability, feasibility, and

viability—during construction of the final prototype. This means that engineering

students used prototyping methods or practices from each of three previous alpha

prototype phases throughout the construction of the final prototype.

4.6.2 Between Subjects Experiment

A Mann-Whitney U [162] test was performed to determine if there were differences in desirability awareness sub-scores between the experimental class and the control class. This statistical method was chosen because it is a rank-based nonparametric test that is typically used to determine if there are differences between two groups on an ordinal dependent variable [163].

Prototyping awareness was measured with a five-point Likert-type scale, making it an ordinal dependent variable. This eliminated an independent samples t-test as an option, as it requires a normal distribution and continuous dependent variable. Median prototyping awareness scores per item were not significantly different between the experimental and control classes except on three items: 1) thought about the technical functionality of the design (U = 953, p = .001), 2) explored how subsystems would work (U = 867, p = .040), and 3) was not concerned with the functionality of the prototype (U = 502, p = .021). All three items are within the feasibility subscale of the PAWS.

This data indicates that students in the experimental group were significantly more aware about the

99 technical functionality of their designs (with respect to these three items) as compared to the control groups. The following observations can be made in relation to our second hypothesis:

 H1: Engineering students exposed to the PFX methods will be more aware of

prototyping methods than students who are not exposed to these lenses. In general,

we saw few statistically significant differences on PAWS items between the

experimental and control classes. In particular, only three items yielded

statistically different results across the control and experimental groups. We know

that the experimental group scored higher across these three items; however, at this

point we cannot say for certain why we saw these results. In future work we will

attempt to parse this data out further and investigate the causes of these differences.

We will collect more data in the next experiment to determine whether these

differences might be due to (a) course constraints, (b) efficacy of student

prototyping awareness, (c) social biasing, or (d) any number of other variables. We

detected no differences in students’ awareness of desirability and viability, but at

this stage in the experiment, we are unable to define the reasons behind this finding.

We will consider how these parts of PFX can be emphasized even stronger and

more clearly next time to determine whether the way in which they are presented

will yield significant results. Further research is required to explore these findings

and rule out alternative explanations.

4.7 Implications

The results of our experiment indicate that a statistically significant relationship exists between prototyping awareness, as measured by PAWS, and structured prototyping frameworks such as PFX. The purpose in this chapter was to present preliminary findings on the relationship

100 between the three prototyping lenses, desirability, feasibility, and viability, and engineering students’ prototyping awareness. Based on our statistical analysis, we found support for our first hypothesis, meaning that students were able to successfully shift their prototyping awareness to focus on either desirability, feasibility, or viability with respect to the PFX framework. This is an important finding because it indicates that students can successfully engage with and implement new prototyping frameworks to develop more desirable, feasible, and viable end designs. We found no statistically significant difference between the control and experimental groups in terms of general awareness about prototyping. This is an interesting and unexpected finding, as data from a separate experiment (discussed in Chapter 3) showed that designs from the control classes performed significantly lower in metrics related to desirability, feasibility, and viability; in other words, the students’ design outcomes seem to show that students in the control group are not as aware of the PFX lenses as the experimental group [25]. The disparity in the findings from the two experiments, reviewed in Chapters 3 and 4, could indicate that some social bias is at play, with students answering items based on what they think their prototyping awareness levels should have been as opposed to what they actually were. This finding could also indicate that students are not aware of their own knowledge gaps and feel that their prototyping behaviors and practices are at a mastery level. The instructors for each control class were different, and so we felt it necessary to compare the scores from control classes; however, there were no differences between the two control sections on PAWS scores.

This work benefits design educators because it helps shed light onto the thought processes and behaviors student engineers engage in during the prototyping process. PAWS can be used to determine if engineering students are focusing on desirability, feasibility, or viability and it can help educators re-focus student teams onto problem areas or blind spots. Our work also helps students and educators take some of the guesswork and mystery out of the prototyping process. The

PFX framework guides student teams to prototype and iterate on designs more effectively,

101 ultimately leading to designs that are more desirable, feasible, and viable. Future work will help determine how generalizable PFX methods are by exploring how PFX affects designs and students within different design contexts and problems. Our goal is to create a prototyping framework that enhances traditional design courses and adds to students’ skillsets and tools for use in industry.

4.8 Limitations

This study was limited in its scope due to a lower internal consistency on the feasibility subscale and constraints placed on the experiment due to the nature and timing of the course. The moderate internal consistency makes it difficult to attribute changes on that subscale to one larger construct, namely, feasibility. Constraints placed on the experiment due to the nature and timing of the course include a sample of convenience for the experimental class (N =30); PFX lenses could not be randomized; and the lectures, materials, and instructions in two control classes were outside the control of the primary researcher. The experimental sample was a sample of convenience, as the primary researcher was teaching this section; this means that the effect the lecturer had on the class could not be evaluated independently and may have influenced student and design outcomes.

Because the researcher only had control over one section, the order of the PFX methods was set to be (1) desirability, (2) feasibility, and (3) viability; the study reviewed in Chapter 5 includes a partial factorial experimental design to evaluate what (if any) effects the order of PFX methods has on student and design outcomes. We were also unable to distribute a pre-test in the control sections so we do not have any data on the students self-efficacy beliefs and prior prototyping experience.

This is a limitation of the study because potentially the students in the control classes could have started the course with higher levels of self-efficacy and more prototyping experience which could have affected the prototyping awareness results. In the experiment discussed in Chapter 5 we were able to distribute a pre-test to each of the samples included in the study. Finally, lectures, materials,

102 and instructions for the final prototype within the control sections were outside the control of the primary researcher; so, other methods may have affected the results from these sections. In our second study presented in the next chapter, the course content was controlled throughout each prototyping phase.

4.9 A Look Back and a Look Ahead

In this chapter we evaluated the impact of a structured prototyping framework on engineering students’ prototyping awareness. We found that engineering students’ prototyping awareness shifts throughout new product development as they develop their prototypes. We also saw that students in control and experimental classrooms rated themselves as equally aware of prototyping behaviors, even though final designs did not reflect this awareness. This phenomenon is explored in our second study, discussed in Chapter 5. In the following chapter we explore what effect, if any, the sequencing of PFX lenses has on design and designer outcomes.

103

Chapter 5

Assessing the Impact of PFX Sequence on Designs and Designers

5.1 Introduction

In Chapters 3 and 4, we studied the effect that the PFX framework has on both product and people outcomes. In this chapter, we are interested in understanding how the sequence of PFX lenses affects product outcomes and prototyping awareness. In order to study these phenomena, we implemented two different sequences of PFX lenses in a junior-level mechanical engineering design course, in addition to studying a control group. We begin this chapter by presenting the research hypotheses that guided our work and then present the results from each study.

5.2 Research Hypotheses

The experiment described in this chapter was guided by the following research question: does the order or sequence of PFX lenses affect the feasibility, viability, and desirability of an end design and the prototyping awareness of designers? In Chapter 3, we demonstrated the positive effect that a structured prototyping framework could have on design outcomes in comparison to unstructured prototyping. We assert that design outcomes will be improved independently of the ordering of PFX lens because each of the teams will be exposed to all three PFX lenses and structured methods; thus, timing should not affect the final design outcomes. The PFX lens sequence will, however, affect prototyping awareness throughout prototyping activities. In Chapter

4, we demonstrated that engineering students’ prototyping awareness shifted relevant to the PFX lens being implemented. Similar to Chapter 4, we collected data on prototyping awareness throughout PFX implementation, and we hypothesize that students’ prototyping awareness will

104 shift relevant to the lens they are being exposed to at any given time point. Thus, the order of lenses will affect the prototyping awareness of students throughout prototyping activities.

To test this, we conducted a controlled experiment in nine sections of a junior-level mechanical engineering design course in which the Prototype for X framework was implemented.

Our hypotheses with respect to prototyping awareness and PFX are as follows:

 H1a: Engineering students exposed to PFX lenses will be more aware of

prototyping behaviors than students not exposed to PFX lenses.

 H1b: Engineering students will be more aware of prototyping behaviors relevant

to PFX lenses.

 H2: The sequence of PFX will influence prototyping awareness throughout

prototyping activities

 H3: The sequence of PFX will not affect product outcomes; specifically there will

be no significant difference in designs produced using PFX lenses regardless of

lens order.

 H4: Designs produced using PFX lenses will outperform designs in the control

group.

5.3 Experimental Protocol

In order to evaluate the impact of PFX lens sequencing on prototyping awareness and product outcomes, a between subjects’ experiment was conducted in a junior-level mechanical engineering design course. Nine sections of the 232 student class were used in this study and were divided into three samples (three sections per sample). We had one control group and two PFX groups with different lens order, lens order along with data collected is shown in Figure 5-1. The two sequences of PFX tested were PFD-PFF-PFV, or DFV for short, and PFF-PFV-PFD, or FVD

105 for short. The ordering DFV is proposed and used as the “best” path in Design Thinking

[121,125,128] for product innovations; meanwhile, the sequence FVD reflects the traditional engineering approach offered in many engineering design texts [12].

All participants were given the following project description:

Teachers are always looking for new and exciting ways to engage their students in the classroom. As low-cost 3D printers are becoming more widely available and easy to use, the opportunities to engage students in hands-on activities in the classroom are profound. Engineering students can 3D print design prototypes, anthropologists can 3D print specimens and fossils, visual artists can create new shapes and forms of art, and students in mathematics can 3D print complex functions to better understand them. The availability of 3D printing for in-class use can redefine not only what is taught but how students learn.

Despite the pervasiveness of this technology, many teachers remain apprehensive about using and adopting 3D printing in the classroom. Therefore, the objective in this project is to design, develop, fabricate, and test an educational activity that leverages 3D printing to create a novel and engaging hands-on experience in the classroom. The activity should be easily packaged into a “ready-to-go” kit that could be easily shipped or distributed. The kit should cost less than

$50 and include everything that is needed for the activity (e.g., step-by-step instructions, CAD files, measurement tools and devices, purchased parts/components, sample results, guidelines for the teacher). The kit should be designed for the appropriate target audience (e.g., middle school students, high school students, college-aged students, visually impaired students, adult learners) and demonstrate a scientific, mathematical, or engineering principle suitable for the target age.

You will work in teams of 3-4 to design, develop, fabricate, and test the educational kit over the course of the semester. The Department of Mechanical & Nuclear Engineering and the new Maker Commons in Pattee/Paterno Library will provide access to 3D printing on MakerBot

Replicator (Generation 5) at no charge. Each team will also have $50 to develop and create their

106 educational kit for testing. Contacts to local schools, teachers, hands-on science camps, and other resources will be made available as needed. Your ME 340 instructor will guide you through the design and development process, and your team’s final presentation at the end of the semester will be a “pitch” to either (i) launch your own start-up or (ii) license your kit to an existing education technology firm. The team with the best “pitch” in each section will be eligible to receive $500 to advance their kit (e.g., prepare for a Kickstarter campaign), and the best team from each section will be invited to be judged at the College of Engineering Design Showcase. The winning team at the Design Showcase will be eligible for an additional $3000.

Figure 5-1. Timeline and data collected in each sample

107

All classes were presented with the same deadlines and course and resource constraints; participants in Sample A received PFX interventions following the first sequence (PFD-PFF-PFV), and participants from Sample B received PFX interventions following the second sequence (PFF-

PFV-PFD) from Week 8 of the semester to Week 14 (semesters are 16 weeks long). Participants in the control class were provided with no instructions for specific prototyping methods. Each section had three alpha prototypes and one beta prototype due on the same dates, as highlighted in Figure

5-1. Final designs from both classes were due during Week 16 of the semester. Table 5-1 provides an overview of the data collected at each of the time points; in the final column of this table, we differentiated whether the data was collected from the individual or whether the data was related to the final product produced by the design team.

Table 5-1. Data Collected throughout Experiment

Time (Weeks in the Semester) Data Collected Product or Individual Week 2 Design Self-Efficacy Individual Week 2 Creative Self-Efficacy Individual Week 2 Demographic Data Individual Week 9 PAWS Individual Week 12 PAWS Individual Week 15 PAWS Individual Week 16 User Satisfaction Rating Product Week 16 User Perceived Value Rating Product Week 16 Effectiveness Rating Product Week 16 Critical Print Ratio Product Week 16 Critical Part Count Ratio Product

5.3.1 Participants

Participants were juniors in mechanical engineering at the Pennsylvania State University, and the experiment took place in the latter half of the semester, when students began to build and test prototypes. Three samples are reported in this work: (1) Sample A is composed of 87 students,

78 male and 6 female, from three PFX sections with order D, F, V, (2) Sample B is composed of

108

88 students, 77 male and 11 female, from three PFX sections with order F, V, D, and (3) Sample C is composed of 60 students, 51 male and 9 female, from three control sections.

5.3.2 Data Collection

At the start of the semester, we distributed a survey across each sample in order to understand if there were differences in the distributions in creative self-efficacy or engineering design self-efficacy. Creative self-efficacy has been shown to positively impact creative performance [184], and engineering design self-efficacy has been linked to modeling abilities

[171]. In order to ensure that each sample had similar distributions of self-efficacy beliefs, so that these factors could be eliminated as potential variables, we distributed the Creative Self-Efficacy scale from Tierney and Farmer [184], and the Engineering Design Self-Efficacy scale from

Carberry and Ohland [171]. Using a Kruskal-Wallis H test [185] we found that there was no statistically significant difference in creative or engineering design self-efficacy across sections.

Data from Samples A, B, and C were collected at four separate time points. At Time 1, the prototyping awareness scale was distributed to all students and the first alpha prototype was due in all samples. At Time 2, the prototyping awareness scale was distributed, the second alpha prototype was collected, and responses to two open-ended questions was collected. The two open-ended questions were: 1) Did you receive any form of feedback or perform ant tests on your first alpha prototype? If yes, please describe how you gathered this feedback or how you performed the tests and 2) How did the development of your first alpha prototype affect the development of your second alpha prototype? If you performed any tests or gathered feedback, please describe how this affected the design as well. These questions were asked in conjunction with PAWS in order to evaluate students’ perception of prototype evolution and to gather further insights into students’ prototyping awareness.

109

At Time 3, the prototyping awareness scale was distributed, the third alpha prototype was collected, and responses to two open ended questions were collected (note the questions at Time 3 were amended to read: 1) Did you receive any form of feedback or perform ant tests on your second alpha prototype? If yes, please describe how you gathered this feedback or how you performed the tests and 2) How did the development of your second alpha prototype affect the development of your third alpha prototype? If you performed any tests or gathered feedback, please describe how this affected the design as well). At Time 4, photos of each prototype, prototyping performance data (prototyping metrics), and a one-page summary of the prototype and final concept were gathered. Prototyping performance data includes data regarding the user satisfaction, perceived user value, effectiveness, and manufacturability of the final design. The process for data collection and the data collected is displayed in Figure 5-1.

5.4 Results: PFX Sequence and Product Outcomes

In a between subjects experiment, we compare performance of final prototypes from

Samples A, B, and C using four prototyping metrics. As in Chapter 3, in order to evaluate user satisfaction and manufacturability, we used the Delighted-Terrible (DT) scale [72] and the

PERVAL scale [160] and calculated the critical part count of final designs [123]. The nature of this design challenge was quite different from that discussed in Chapter 3; it is much easier to evaluate the technical quality of a vacuum cleaner rather than the technical quality of an educational kit that utilizes 3D printing to teach students. In an attempt to evaluate in some way, the quality of the teaching activity we created two new metrics for the design challenge. Because the core

“function” of the kits was to effectively communicate the intended lesson, we asked two independent expert raters to rate each kit using a five-point Likert-type scale. We asked these raters

“How well does the educational kit or tool effectively communicate the intended lesson?”, and

110 responses were record from “extremely well” to “not well at all”. We also gathered data on the way in which 3D printing was used in the prototypes. Specifically, we counted how many parts or components were 3D printed in the educational kits and asked raters to evaluate how many parts were 3D printed but should not have been. For example, a simple cylinder or disk is more easily manufactured using traditional manufacturing methods as opposed to additive manufacturing methods. We were interested in gathering this information because a key component of the initial constraints was designing an educational kit that effectively incorporated 3D printing; so we were interested in understanding how many components were 3D printed in each sample and how well students understood the capabilities and appropriateness of 3D printing technology. We recognize that evaluating the effectiveness with a five-point Likert type scale makes the evaluation of technical quality subjective; however, we were constrained by the nature of the design challenge.

To test our third hypothesis and determine whether design outcomes are impacted by the sequence of prototyping methods used by the designer, statistical analyses were conducted in order to evaluate the differences between the three samples across the five product metrics (user satisfaction, user-perceived value, effectiveness rating, 3D printing efficiency, manufacturability).

5.4.1 User Satisfaction and Perceived Value

A Kruskal-Wallis H test [185] was conducted to determine whether there were differences in user satisfaction scores on the Delighted-Terrible (D-T) single-item scale and in user-perceived value on the PERVAL subscales between the experimental and control classes. This statistical method was chosen because it is a rank-based nonparametric test typically used to determine if there are differences between three or more groups on an ordinal dependent variable [163].

Distributions of the user satisfaction scores for Samples A, B, and C are shown in Figure 5-2. The difference in median user satisfaction scores was significantly higher in Samples A and B (6.00

111 and 6.00 respectively) than in the control class, or Sample C (4.5) (χ2(2) = 39.716, p < 0.0005).

There was no statistically significant difference in the distribution of D-T scores between Samples

A and B. Recall that Samples A and B were both exposed to PFX lenses; Sample A had PFX sequence of PFD-PFF-PFV and Sample B had PFX sequence of PFF-PFV-PFD.

Figure 5-2. Distributions of User Satisfaction Scores in Samples A, B, and C

Distributions of the mean user-perceived value score for Samples A, B, and C are highlighted in Figure 5-3. The difference in median user-perceived value score was significantly higher in Samples A and B (6.09 and 6.00 respectively) than in sample C (4.909) (χ2(2) = 37.708, p < 0.0005). Table 5-2 highlights the observed differences across each item of the PERVAL scale.

Significant differences were observed across nine out of nineteen items on the PERVAL. Items has poor workmanship and would not last a long time were reverse-scored. So higher scores indicate disagreement with this statement, or higher levels of quality. Note, outliers were removed for statistical analysis.

112

Figure 5-3. Distribution of Perceived Value Scores in Samples A, B, and C

Table 5-2. Significant difference observed across items of PERVAL Scale (bold and italics indicate significant difference observed)

PERVAL Items Mean Score Mean Score Mean Score p Sig. Differences Sample A Sample B Sample C Between C & B Has consistent quality** 5.96 6.30 4.46 >.0005 and C & A Is well made 5.82 6.19 5.38 .023 Between C & B Has poor workmanship (RS)* 5.77 6.09 5.68 .383 No differences Has an acceptable standard of 5.82 6.26 5.51 .063 No differences quality Differences Would not last a long time (RS)** 5.07 5.98 2.84 >.0005 between all groups Would perform consistently** 5.50 6.09 4.84 .002 Between C & B Differences Is one that I would enjoy** 6.32 5.69 4.19 >.0005 between all groups Differences Would make me want to use it** 6.29 5.69 4.43 >.0005 between all groups Is one that I would feel relaxed 6.16 6.01 4.95 .007 Between C & A about using** Between C & B Would make me feel good** 5.71 5.54 4.41 >.0005 and C & A Would give me pleasure 5.86 5.37 5.14 .080 No differences

The results of our analysis partially support our third hypothesis that a final design’s perceived value and user satisfaction will not be impacted by the sequence of PFX lenses. While there was no difference in user satisfaction and mean user-perceived value scores between Samples

A and B, there were differences across items of the PERVAL scale. This finding highlights the advantage of using PFX methods over unstructured prototyping methods by increasing the perceived value and satisfaction from end users. For industry, increasing user satisfaction and

113 perceived value will likely result in higher ROIs and fewer failed products. For items would not last a long time, is one that I would enjoy, and would make me want to use it, there were significant differences between all the Samples. Sample B was rated higher in the item would not last a long time and Sample A was rated higher in items is one that I would enjoy and would make me want to use it. There were also a few items in which only Sample A or B were significantly different than

Sample C. For example, for the item, would perform consistently there was a statistically significant difference between Samples B and C; however, there was no difference between Samples A and C on this item. These differences indicate that the nature of the perceived value of final prototypes in

Samples A and B is somehow different. Reviewing the items, we find that prototypes from Sample

B were typically rated higher in items that were related to the technical quality of the design, while items from Sample A were rated higher in items related to the emotional impact of the design. This is an interesting finding and should be explored further in future work.

5.4.2 Effectiveness

Recall that each kit’s effectiveness was rated by two raters working independently; the scale asked, “How well does the educational kit or tool effectively communicate the intended lesson?”, and responses were record from “extremely well” to “not well at all”. A Kruskal-Wallis

H test [185] was conducted to determine whether there were any significant differences in effectiveness ratings across Samples A, B, and C. There were no statistically significant differences on the effectiveness rating between any sample pairs (χ2(2) = 2.297, p > 0.317). We believe that this may have been due to measurement error, because this design challenge was more open-ended and there was not a clear-cut way to evaluate the technical quality of the final design. The PERVAL scale does ask raters to evaluate the quality of the designs, and we did see significant differences

114 on some of these items, thus we feel our scale did not effectively capture data on the effectiveness of the educational kits.

Figure 5-4. Distribution of Effectiveness Ratings in Samples A (D-F-V Group), B (F-V-D Group), and C (Comparison Class) [6 = highly effective, 1 = not effective at all]

In order to evaluate the efficiency of 3D print use we calculated the critical print ratio for

(푁−푛) each prototype. The critical print ratio is defined as where N = total number of printed parts 푁 and n = number of printed parts that should not have been 3D printed. The number of parts that were 3D printed and should not have been 3D printed refers to components that could have been manufactured more efficiently using traditional subtractive manufacturing processes. For example, in Figure 5-5 the components were 3D printed although they could have been produced more efficiently using a laser cutter. This ratio serves as an indication of the appropriateness of 3D printing use and highlights the student teams’ ability to select the most efficient manufacturing processes.

115

Figure 5-5. An Example of Components that Should not have been 3D Printed

A Kruskal-Wallis H test [185] was conducted to determine if there were differences in critical print ratios across Samples A, B, and C. We found that there were statistically significant differences in critical print count ratios between samples (χ2(2) = 11.090, p = .004). Specifically, we found that Sample A (Mdn = 1.00, p = .003) and Sample B (Mdn = 1.00, p = .033) both had significantly higher critical print ratios as compared to Sample C (Mdn = .400). There was no statistically significant difference between critical part count ratios between Samples A and B.

Figure 5-6. Distribution of Critical Print Ratio in Samples A, B, and C

These results are interesting and indicate that students may be unable to properly select appropriate production processes. While we observed that designs produced in Samples A and B

116 had fewer unnecessary prints as compared with Sample C, we cannot say for sure why this difference exists. None of the lenses implemented in the PFX classes (Samples A and B) were directly linked to proper 3D printing use, and more research is needed to explore this phenomena in more detail.

5.4.3 Manufacturability Rating

A Kruskal-Wallis H test [185] was conducted to determine if there were differences in in critical part count ratios across Samples A, B, and C. We found that there were statistically significant differences in critical part count ratios between samples (χ2(2) = 16.900, p < .0005).

Specifically, we found that Sample A (Mdn = .750, p = .002) and Sample B (Mdn = .733, p < .005) both had significantly higher critical part count ratios as compared to Sample C (Mdn = .333).

There was no statistically significant difference between critical part count ratios between Samples

A and B.

Figure 5-7. Distribution of Critical Part Count Ratio in Samples A, B, and C

117

These results support our third hypothesis that PFX lens order would not affect product outcomes, in particular the products manufacturability. We found that Sample C, the control or

“prototyping in the wild” group, had a significantly lower critical part count ratio as compared with

Samples A and B; this means that the designs in Sample C would potentially be costlier and complex to manufacture and assemble.

5.4.4 Discussion of Results and Limitations

At the start of this study, we were interested in answering one research question: does the sequence of PFX lenses affect or influence final product outcomes. We started with the hypothesis that the sequence of PFX lenses would not influence product outcomes and that final prototypes produced in PFX classes would outperform prototypes produced in control classes. We found evidence to partially support both of these hypotheses.

From the user satisfaction ratings, we found that there was no statistically significant difference between satisfaction ratings of the final prototypes produced in Sample A and Sample

B. Recall the lens order in Sample A was PFD-PFF-PFV, and the lens order in Sample B was PFF-

PFV-PFD. This supports our first hypothesis that there would be no difference in product outcomes due to the sequence of PFX lenses. While we found no statistically significant difference between mean perceived value scores of the final prototypes produced in Sample A and Sample B, we did find statistically significant differences across some of the items between Samples A and B. This partially supports our hypothesis, implying that while raters perceived the value of prototypes in

Sample A and Sample B to be equal, the nature of the value was different. Prototypes from Sample

A in general outperformed prototypes in Sample B and C in items related to the emotional value of the product, for example “would make me feel good”. Prototypes from Sample B in general outperformed prototypes in Sample A and C in items related to the quality of the product, for

118 example “is well made” or “would not last a long time (reverse scored)”. Future work will explore the multi-faceted nature of perceived value and user satisfaction, specifically focusing on the impact the sequence of PFX lenses have on the final characteristics of a design.

We also found that there were no statistically significant differences in effectiveness ratings across samples. This does not support our hypothesis that students in experimental classrooms would outperform those in control classrooms. Because of the nature of the design challenge, determining appropriate metrics to evaluate technical quality was difficult. Unlike the study discussed in Chapter 3, the evaluation of technical quality in this study was unfortunately extremely subjective. We were unable to use a validated scale because an appropriate scale does not exist. We believe that this was a major limitation of the study. Future work should explore how to effectively evaluate technical quality of prototypes in more broad and ill-defined design challenges. In the real world, designers are often working on challenges like the challenge presented in this study, and it may be difficult to assess technical quality objectively. It is critical to understand how we might evaluate technical quality in design challenges such as this.

We also evaluated the ability of the student teams to effectively use 3D printing in their educational kits, as this was one of the constraints in the initial problem statement. In order to evaluate this, we calculated the critical print ratio for each prototype; this was done by counting the total number of 3D printed components, subtracting the number of unnecessary prints, and dividing by the total number of 3D printed components. This ratio was used to eliminate the effect having more parts would have on the analysis. We found that prototypes produced in Samples A and B had significantly higher critical print ratios as compared with Sample C. In other words, this means that students in Samples A and B were able to more effectively and efficiently leverage the 3D printing technology to support and enhance their designs as opposed to simply using the technology because it was an initial project constraint.

119

Finally, we found that students in Samples A and B produced prototypes with significantly higher critical part count ratios as compared to the prototypes produced in Sample C. This suggests that the prototypes in Samples A and B were less complex and would theoretically be less of a manufacturing burden. This supports our second hypothesis that prototypes produced in the PFX classrooms (Samples A and B) would outperform prototypes produced in the control class (Sample

C). There was no statistically significant difference between prototypes produced in Samples A and

B in critical part count ratios, which supports our first hypothesis.

5.5 Results: PFX Sequence and Prototyping Awareness

In a between subjects experiment, we compare Prototyping AWareness Scale (PAWS) results from Samples A, B, and C to evaluate whether groups ranked higher on certain items of the

PAWS scale, indicating a higher level of prototyping awareness in either desirability, feasibility, or viability from using PFX. We evaluated the difference between groups at all three time points which would also shed light on whether the lens of PFX has an impact on prototyping awareness.

In a within subjects experiment, we were interested in answering how prototyping awareness is impacted by PFX sequence and explored the evolution of prototyping awareness throughout time within each sample. This experiment allows us determine if students prototyping awareness shifted throughout the 8-week implementation of PFX. The qualitative results were used to cross-check our quantitative results as well as potentially shed light onto the rationale or reasoning behind any trends in the data.

120

5.5.1 Between Subjects Results

Results from the between subjects experiment are reported next. These data were taken at three separate time points as discussed in Section 5.3.2 and highlighted in Figure 5-1. Scores on the PAWS were compared for the three groups: Sample A, Sample B, and Sample C. Kruskal-

Wallis H [185] tests were run at each time point; the Kruskal-Wallis H test is a rank-based nonparametric test used to determine if there are statistically significant differences between two or more groups of an independent variable on a continuous or ordinal dependent variable. This test was used as opposed to a one-way ANOVA because we have an ordinal dependent variable [163].

Our independent variable is PFX order, and the dependent variable is score across PAWS items.

Although Sample C was not exposed to PFX we still include their data in this analysis as a frame of reference.

Time 1

A Kruskal-Wallis H test was run to determine if there were differences in item scores between the three groups of participants at Time 1, specifically Sample A (n = 87), Sample B (n =

85), and Sample C (n = 56). Distributions of PAWS item scores were similar for all groups, as assessed by visual inspection of boxplots. Median scores were significantly different between groups for five items, namely (1) reflected on user feedback about the design, (2) thought about the technical functionality of the design, (3) explored how subsystems would work, (4) determined one or more technical functions to test, and (5) was not concerned with the functionality of the prototype. The distribution of scores for reflected on user feedback about the design was significantly higher in Sample A and Sample C, as compared to Sample B, χ2(2) = 27.341, p <

0.0005. The distribution of scores for thought about the technical functionality of the design and explored how subsystems would work was significantly higher in Samples B and C as compared to

121

Sample A, χ2(2) = 8.214, p = 0.016, and χ2(2) = 23.506, p < 0.0005, respectively. The distribution of scores for determined one or more technical functions to test, and was not concerned with the functionality of the prototype was significantly higher in Sample B as compared to Samples A and

C, χ2(2) = 15.210, p < 0.0005, and χ2(2) = 11.702, p = 0.003 respectively.

To summarize, Sample A ranked significantly higher on one item from the desirability subscale and ranked significantly lower than Samples B and C on four items from the feasibility subscale of the PAWS. Interestingly, the control group (Sample C) rated themselves relatively high across both desirability and feasibility subscales. All three groups ranked themselves low on the viability subscale, indicating that they were not aware or thinking about prototyping behaviors related to viability.

Time 2

Distributions of PAWS item scores were similar for all groups, as assessed by visual inspection of boxplots. Median scores were significantly different between groups for three items, namely (1) reflected on user feedback about the design, (2) explored how subsystems would work, and (3) used technical knowledge to layout or design the prototype. The distribution of scores for reflected on user feedback about the design was significantly higher in Sample A, as compared to

Samples B and C, χ2(2) = 28.078, p < 0.0005. The distribution of scores for explored how subsystems would work was significantly higher in Sample B as compared to Samples A and C,

χ2(2) = 7.372, p = 0.025. The distribution of scores for used technical knowledge to layout or design the prototype was significantly higher in Sample B as compared to Samples A and C, χ2(2) =

11.630, p = 0.003.

To summarize, Sample A again ranked significantly higher on one item from the desirability subscale, and ranked significantly lower than Samples B and C on four items from the feasibility subscale of the PAWS.

122

Time 3

Distributions of PAWS item scores were similar for all groups, as assessed by visual inspection of boxplots. Median scores were statistically significantly different between groups for one item, namely, reflected on user feedback about the design. The distribution of scores for reflected on user feedback about the design was significantly higher in Sample A, as compared to

Samples B and C, χ2(2) = 10.689, p = .005.

Summary of Between Subjects Results

Summarizing these results, we found that there were few items with significantly different distributions between samples at each of the time points. These results do not support our hypothesis that engineering students in Samples A and Samples B would be significantly more aware of prototyping behaviors as compared to engineering students in the control classes, Sample

C. These results also do not support our hypothesis that the sequence of PFX lenses would impact prototyping awareness.

5.5.2 Within Subjects Analysis

A Friedman test was run on each subscale score to determine whether there were differences in prototyping awareness throughout the semester in Samples A, B, and C. For each sample, we compared students’ scores on the desirability, feasibility, and viability subscale scores at Times 1, 2, and 3. Pairwise comparisons were performed (SPSS Statistics, 2012) with a

Bonferroni correction for multiple comparisons.

123

Sample A

For Sample A (PFX lens sequence PFD-PFF-PFV), desirability, feasibility, and viability prototyping awareness subscale scores were significantly different at the different time points during the semester [χ2(2) = 15.967, p < 0.0005, χ2(2) = 36.738, p < 0.0005, χ2(2) = 39.723, p <

0.0005, respectively]. Post hoc analysis revealed the following statistically significant differences:

From Alpha Prototype 1 to Alpha Prototype 2, desirability scores were significantly different for Sample A, namely, they increased from Time 1 (Mdn = 19.00) to Time 3 (Mdn =

20.00) but not from Time 1 to Time 2. Desirability subscale scores were also significantly different between Time 2 (Mdn = 20.00) and Time 3 (Mdn = 20.00). In other words, participants in Sample

A rated themselves as being less aware of desirability prototyping behaviors during creation of the first and second alpha prototypes as compared to creation of the third alpha prototype..

From Alpha Prototype 1 to Alpha Prototype 3, feasibility scores were significantly different for Sample A, namely, they increased from Time 1 (Mdn = 18.00) to Time 2 (Mdn = 20.00) but not from Time 2 to Time 3. Feasibility subscale scores were also significantly different between

Time 1 and Time 3 (Mdn = 20.00). In other words, participants in Sample A rated themselves as being less aware of feasibility prototyping behaviors during creation of the first alpha prototype as compared to creation of the second and third alpha prototypes.

From Alpha Prototype 1 to Alpha Prototype 3, viability scores were significantly different for Sample A, namely, they increased from Time 1 (Mdn = 14.00) to Time 2 (Mdn = 16.00) and from Time 2 to Time 3 (Mdn = 17.00). Viability subscale scores were also significantly different between Times 1 and 3 and between Times 2 and 3. In other words, participants in Sample A continuously increased in their awareness of viability prototyping behaviors during creation of the first, second, and third alpha prototypes throughout the semester. Figures 5-8 to 5-10 highlight the distribution of scores for each subscale after each alpha prototype was complete.

124

Scale Scale Score - 15

10 Desirability Desirability Sub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-8. Distribution of Desirability Prototyping Awareness Scores for Sample A

Scale Scale Score - 15

10 Feasibility Feasibility Sub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-9. Distribution of Feasibility Prototyping Awareness Scores for Sample A

125

20 Scale Scale Score - 15

10 ViabilitySub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-10. Distribution of Viability Prototyping Awareness Scores for Sample A

Sample B

For Sample B (PFX lens ordering PFF-PFV-PFD), there were no statistically significant differences observed between distributions of desirability and feasibility subscale scores. There was a significant difference observed for the viability subscale between Times 1 and 2 and between

Times 1 and 3. From Alpha Prototype 1 to Alpha Prototype 3, viability scores were significantly different for Sample B, namely, they increased from Time 1 (Mdn = 15.00) as compared to Time

2 (Mdn = 17.00) and Time 3 (Mdn = 17.00). Viability subscale scores were not significantly different between Times 2 and 3. In other words, participants in Sample B increased in their awareness of viability prototyping behaviors from the creation of the first alpha prototype to the second, but their awareness levels remained the same from the second to the third alpha prototype.

Figures 5-11 to 5-13 highlight the distribution of scores for each subscale for Sample B after each alpha prototype was complete.

126

Scale Scale Score - 15

10 Desirability Desirability Sub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-11. Distribution of Desirability Prototyping Awareness Scores for Sample B

Scale Scale Score - 15

10 Feasibility Feasibility Sub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-12. Distribution of Feasibility Prototyping Awareness Scores for Sample B

127

20 Scale Scale Score - 15

10 ViabilitySub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-13. Distribution of Viability Prototyping Awareness Scores for Sample B

Sample C

For Sample C (the control class), desirability (χ2(2) = 21.855, p < 0.0005), feasibility (χ2(2)

= 9.557, p < 0.008), and viability (χ2(2) = 23.156, p < 0.0005) prototyping awareness subscale scores were statistically significantly different at the different time points during the semester. Post hoc analysis revealed the following statistically significant differences:

From Alpha Prototype 2 to Alpha Prototype 3, desirability scores were significantly different for Sample C, namely, they increased from Time 2 (Mdn = 19.00) to Time 3 (Mdn =

20.00) but not from Time 1 to Time 2. Desirability subscale scores were also significantly different between Time 1 (Mdn = 19.00) and Time 3 (Mdn = 20.00). In other words, participants in Sample

C rated themselves as being less aware of desirability prototyping behaviors during creation of the first and second alpha prototypes as compared to creation of the third alpha prototype.

From Alpha Prototype 1 to Alpha Prototype 2, feasibility scores were significantly different for Sample C, namely, they increased from Time 1 (Mdn = 19.00) to Time 2 (Mdn = 20.00) but not

128 from Time 2 to Time 3. Feasibility subscale scores were not significantly different between Time

1 and Time 3 (Mdn = 20.00). In other words, participants in Sample A rated themselves as being less aware of feasibility prototyping behaviors during creation of the first alpha prototype as compared to creation of the second alpha prototype.

From Alpha Prototype 1 to Alpha Prototype 3, viability scores were significantly different for Sample C; namely, they increased from Time 1 (Mdn = 15.00) to Time 3 (Mdn = 18.00).

Viability subscale scores were not significantly different between Times 1 and 2 and between

Times 2 and 3. Figures 5-14 to 5-16 highlight the distribution of scores for each subscale for Sample

C after each alpha prototype was complete.

Scale Scale Score - 15

Desirability Desirability Sub 0 Alpha One Alpha Two Alpha Three

Figure 5-14. Distribution of Desirability Prototyping Awareness Scores for Sample C

129

Scale Scale Score - 15

5 Feasibility Feasibility Sub 0 Alpha One Alpha Two Alpha Three

Figure 5-15. Distribution of Feasibility Prototyping Awareness Scores for Sample

Scale Scale Score -

ViabilitySub 5

0 Alpha One Alpha Two Alpha Three

Figure 5-16. Distribution of Viability Prototyping Awareness Scores for Sample C

Summary of Within Subjects Results

Summarizing the results, we found that each sample had some significant differences in prototyping awareness across time. Specifically, we saw that Samples A and C had differences in desirability, feasibility, and viability prototyping awareness across time, while Sample B only had

130 significant differences in viability prototyping awareness. This is an interesting finding, as we expected that the prototyping awareness would shift accordingly with the PFX lens being implemented in Samples A and B (recall Sample A ordered the PFX lenses as PFD-PFF-PFV and

Sample B used the ordering PFF-PFV-PFD). These results partially support our hypothesis that the sequence of PFX lenses will influence prototyping awareness throughout time.

5.5.3 Results from Qualitative Data Set

While quantitative statistical analyses of the PAWS data provide insights into students’ ability to identify prototyping behaviors related to desirability, feasibility, and viability, qualitative analysis of open-ended questions add depth to our understanding of their prototyping mindsets and awareness. The open-ended responses were coded using qualitative content analysis methods

[109,110]. The response set (413 total responses, 201 at Time 2 and 212 at Time 3) was read carefully, and the researchers used open coding methods [186] to derive categories of responses.

Segments of text that identified (1) the audience the prototype was tested with, (2) the purpose the prototype served, (3) the methods used to test the prototype, and (4) the aspects of the prototype that were iterated on, were highlighted. Appropriate tags were assigned to these segments. In the open coding of all students’ responses, 17 tags were identified across four categories; a full list of the tags within categories is provided in Table 5-3, along with illustrative examples and a brief description of the tag. Two independent raters evaluated the responses, and inter-rater reliability was found to be above 0.75 for all categories. Specifically, Audience had an inter-rater reliability of 0.758, Purpose had an inter-rater reliability of 0.796, Method had an inter-rater reliability of

0.882, and Evolution had an inter-rater reliability of 0.905. Once all responses had been tagged, the results were examined for the percentage of tags within Samples A, B, and C.

131

Table 5-3. Category and tags with Descriptions and Example from Qualitative Findings

Category Tag Description Example Audience: With whom Team The prototype was not tested with Yes; the only feedback on the was the prototype tested? anyone outside of the team first alpha prototype was from our team… User The prototype was tested with the end Yes; We gathered feedback by users identified through customer needs having users try out the and project description product…

Peer The prototype was tested with peers in or Yes; I gave my second alpha out of the class for general feedback prototype to my roommates and asked them to interact with it as if they needed it… TA The prototype was tested with the TA of We asked the TA about it and the lab section for general feedback were given some good ideas about the design of the prototype Purpose: Why was the Quality The prototype was tested to evaluate the We had to fit the pieces together, prototype built? technical quality of the design (i.e. to test them and evaluate the material strength, tolerance, or basic tolerances… function) Appeal The prototype was tested to evaluate the We used a questionnaire along appeal of the concept to users with our prototype to test what features of the card people liked Usability The prototype was test to evaluate the We tested how quickly our kit ease of use of the concept could be assembled and how much directions our users needed… Method: How was the Break The prototype was tested by attempting …tested the strength by prototype tested? to break it dropping the prototype and standing on it… Build The prototype was tested by building or We allowed users to experiment using the educational kit with different heights of the ramp and slide objects down it, just the general function of the concept Demo The prototype was tested by presenting Our team presented our idea in the concept or physical model to a group multiple classrooms and surveyed the students after Evolution: How did the Desirability The prototype was changed to make it We took what people were most prototype evolve? more appealing to users concerned with and focused on making that better Functionality The prototype was changed to improve Our first design displayed a non- the technical quality or basic functional set of gears, but the functionality sizes and gear ratios were estimates. For our second prototype we calculated the correct gear sizes and positions and made it work. Manufacturability The prototype was simplified for easier We realized it was difficult to manufacture or tolerances/interfaces assemble the prototype with so were improved to account for many parts, so we reduced the manufacturing methods. number of parts and reoriented the parts so they were easier to 3D print. Usability The ease of use of the prototype was After our customer interacted improved upon with the prototype without instructions, we decided to include detailed instructions and simplify the activity to make it simpler

132

Prototype Audience: With whom was the prototype tested?

The first category that emerged from the open-ended response data was the identification of an audience with which the prototype was tested. We found these audiences typically fit within four bins: Team, User, Peer, and TA. Team refers to instances in which the team tested or built the prototype for themselves, and they did not share or test the prototype with anyone outside of the original design team. Responses marked as Team typically included text about building the system, design, or kit and sharing/testing it only with team members. Responses marked as User refer to instances in which the team identified users of their product and brought their prototype to these users for testing. Responses marked as User typically included text about presenting the design or concept to a group of end users or customers or allowing “users/customers” to interact with the prototype. Responses marked as Peer refer to instances in which the team tested or presented their prototype with/to roommates, classmates, or friends. Responses marked as Peer typically included text about checking or gathering feedback specifically from friends about the idea or concept before continuing with the design. Responses marked as TA refer to instances in which the team used their prototype as a way to facilitate feedback from the TA or professor of the course. Responses marked as TA typically included text about presenting the prototype to a TA or professor and asking for advice on the design.

At Time 2, when the open-ended questions were first distributed in conjunction with the second PAWS administration, 49.1% of responses were tagged as User, 38.9% were tagged as

Team, 9.5% were tagged as Peer, and 2.4% were tagged as TA. Within Sample A, the most frequently tagged audience was User (86.3%). Within Samples B and C, the most frequently tagged audience was Team (B: 86.4%, C: 75.7%). Figure 5-17 highlights the distribution of tags across all three samples at Time 2. At Time 3, when the open-ended questions were distributed for the second time in conjunction with the third and final PAWS administration, 42.8% of responses were tagged as User, 33.7% were tagged as Team, 17.5% were tagged as Peer, and 6% were tagged as TA.

133

Within Sample A at Time 3, the most frequently tagged audience was user (61.4%). Within

Samples B and C, the most frequently tagged audience was Team (B: 46.9%, C: 67.5%). Figure 5-

18 highlights the distribution of tags across all three samples at Time 3.

100 90 80 70 60 50 40 30 20

Percent Percent of Response Sample in 10 0 Team user friend TA

Sample A Sample B Sample C

Figure 5-17. Distribution of Audience Tags at Time 2

100 90 80 70 60 50 40 30 20

Percent Percent of Responses inSample 10 0 Team user friend TA

Sample A Sample B Sample C

Figure 5-18. Distribution of Audience Tags at Time 3

134

From Figures 5-17 and 5-18 we see that students in sample A reported testing their alpha prototypes with users more frequently than students in Samples B or C at Times 2 and 3. From the figures, we also see that students in Sample C reported testing their prototype with their team most frequently at both time points. Students in Sample B reported testing prototypes with their teams most frequently at Time 2 and 3; however, we do see a slight shift in the frequencies for the other categories., i.e., more students at Time 3 in Sample B reported testing with users than in Time 2.

These results are interesting when we take into account the PFX lenses being implemented at Times

1, 2, and 3 in Samples A and B. Recall the PFX sequence in Sample A was PFD-PFF-PFV and the

PFX sequence in Sample B was PFF-PFV-PFD.

By reviewing Figure 5-17 we see that Sample A first implemented PFD, which emphasizes testing with user. We would then expect to see Sample A have a higher frequency of testing Alpha

Prototype 1 with users, which we see reflected in Figure 5-18. Sample A then implemented PFF and finally to PFV, which focuses on technical quality and manufacture, respectively. We would expect to see a shift in Sample A towards more frequently testing with team or TA as we observed teams often tested with themselves or with the TA when they were working to improve the technical quality of their designs. We do see a slight increase in these categories at Time 3; however, from

Figure 5-18 we still see the most frequent category for Sample A is user. Sample B first implemented PFF, and so we would expect to see a higher percentage of students reporting testing with team or TA, which is reflected in Figure 5-17. Sample B was then introduced to PFV and finally PFD; so, we would expect to see a slight shift to the user category. Reviewing Figure 5-18, we do see a slight shift to users in Sample B. Students in Sample C seemed to consistently test with their team and there are not many differences in the distribution of audience from Time 2 to Time

3. These results partially support our hypothesis that the sequence of PFX lenses would affect the prototyping awareness of students with respect to the lenses being implemented.

135

Prototype Purpose: Why was the prototype created?

A second trend in the coding of the open-ended responses was students’ identification of a central assumption for which their prototype was created. We found that the perceived purpose of each team’s prototype typically fit within three categories: Quality, Appeal, and Usability. Quality referred to the technical functionality of the design and included responses that cited testing materials, interfaces, or basic utility as the main purpose of the prototype. Responses tagged as

Quality typically included text about building a working model of a subsystem, testing the tolerances of the manufacturing method, or evaluating the strength of materials or components.

Appeal referred to the general appeal of the concept to customers and included responses that cited demoing the prototype and gathering general feedback on the concept. Responses tagged as Appeal typically included text about presenting a non-functioning or slightly functioning representation of the concept to a group of users and gathering feedback in the form of surveys or interviews.

Usability referred to the ease of use of the concept and included responses that cited testing the prototype directly with users. Responses tagged as Usability typically included text about allowing users to interact with the prototype with little to no instruction in order to understand how much or how little instructions and re-design were necessary to improve the usability of the design.

At Time 2, when the open-ended questions were distributed in conjunction with the second

PAWs administration, 36.3% of responses were marked with the purpose of Quality, 32.6% were marked as Likability, and 31.1% were marked as Usability. Within Sample A, the most frequently tagged purpose was Likability (52%). Within Samples B and C, the most frequently tagged audience was Quality (B: 66.7%, C: 41.2%). Figure 5-18 highlights the distribution of tags across all three samples at Time 2. At Time 3, when the open-ended questions were distributed for the second time in conjunction with the third and final PAWs administration, 33.3% of responses were marked with the purpose Quality, 25.8% were marked as Likability, and 40.9% were marked as

Usability. Within Samples A and B, at Time 3, the most frequently tagged purpose was Usability

136

(A: 38.7%, B: 53.3%). Within Sample C, the most frequently tagged purpose was Quality (81.6%).

Figure 5-19 highlights the distribution of tags across all three samples at Time 3.

100 90 80 70 60 50 40 30 20

Percent Percent Responses of inSample 10 0 quality likability usability

Sample A Sample B Sample C

Figure 5-19. Distribution of Purpose Tags at Time 2

100 90 80 70 60 50 40 30 20

10 Percentage Percentage of Responses inSample 0 quality likability usability

Sample A Sample B Sample C

Figure 5-20. Distribution of Purpose Tags at Time 3

137

From Figures 5-19 and 5-20 we see that students in Sample A reported testing their alpha prototypes in order to evaluate the Appeal and Usability more frequently than students in Sample

B or C at Time 2 and Time 3. From the figures we also see that students in Sample B reported testing their prototypes to evaluate the Quality of their design most frequently at Time 2; however, they reported testing their prototype for Usability most frequently at Time 3. Students in Sample C reported testing their prototypes for both Quality and Appeal, with a fair amount also testing their prototype’s Usability. We see a drastic shift at Time 3 in Sample C as a majority of the students reported testing their prototypes for Quality at this time.

These results are interesting when we take into account the PFX lenses being implemented at Times 1, 2, and 3 in Samples A and B. We would expect to see Sample A have a higher frequency of testing the alpha prototype’s Appeal and Usability at Time 2 because the first PFX lens the students were exposed to was desirability; we see this reflected in Figure 5-19. In Figure 5-20 we see a more even distribution in Sample A, as more students reported testing their prototype for

Quality. This also aligns with the PFX lenses being implemented at the time, which were PFF

(Prototype for Feasibility). Sample B implemented PFF first, and so we would expect to see a higher percentage of students testing the prototypes for Quality, which is reflected in Figure 5-19.

Sample B was then introduced to PFV and finally PFD; so, we would expect to see a slight shift in the purpose of the test. We see this in Figure 5-20, with most of the students in Sample B testing the Usability of their designs at Time 3. These results partially support our hypothesis that the sequence of PFX lenses would affect the prototyping awareness of students.

Prototype Methods: How was the prototype Tested?

We found that the method each team used to test their prototype could be separated into three categories: Break, Build, and Demonstrate (Demo for short). Break referred to instances in which the team cited attempts to break or destroy materials, components, or features of their design

138 in order to gather insights into flaws or weak points. Responses categorized as Break typically included text about the strength of the materials being used and cycles to failure of the design. Build referred to instances in which the team cited the need to simply build or create a proof of concept system or sub-systems in order to understand if their idea was technically feasible. Responses categorized as Build typically included text about the team or end users building or putting the prototype together in order to evaluate the general feasibility of the concept. Demo referred to instances in which the team cited demonstrating or presenting their prototypes to groups of users or peers for feedback about the concept.

At Time 2, when the open-ended questions were distributed in conjunction with the second

PAWS administration, 6.8% of responses were tagged as Break, 51.7% were tagged as Build, and

41.5% were tagged as Demo. Within Sample A, the most frequently tagged method was Demo

(87.5%). Within Samples B and C, the most frequently tagged method was Build (B: 55%, C:

60.7%). Figure 5-21 highlights the distribution of tags across all three samples at Time 2.

At Time 3, when the open-ended questions were distributed for the second time in conjunction with the third and final PAWS administration, 5.8% of responses were tagged as Break,

69.8% were tagged as Build, and 24.4% were tagged as Demo. Within Samples A, B, and C, at

Time 3, the most frequently tagged method was Build (A: 52.1%, B: 87.5%, and C: 76.7%). Figure

5-22 highlights the distribution of tags across all three samples at Time 3.

139

100 90 80 70 60 50 40 30 20

10 Percentage Percentage of Responses inSample 0 break build demo

Sample A Sample B Sample C

Figure 5-21. Distribution of Method Tags at Time 2

100 90 80 70 60 50 40 30 20

10 Percentage Percentage of Responses inSample 0 break build demo

Sample A Sample B Sample C

Figure 5-22. Distribution of Method Tags at Time 3

From Figures 5-21 and 5-22 we see that students in Sample A reported testing their alpha prototypes by demonstrating the design (Demo) more frequently than students in Sample B or C at

Time 2, but they shifted and began testing their prototype by building the kit (Build) at Time 3.

140

From the figures we also see that students in Sample B and C reported testing their prototypes by building them (Build) at both Times 2 and 3.

These results are interesting when we take into account the PFX lenses being implemented at Times

1, 2, and 3 in Samples A and B. We would expect to see Sample A have a higher frequency of testing the alpha prototype by demonstrating it to users (Demo) as they were implementing the desirability lens first; we see this reflected in Figure 5-21. In Figure 5-22, we see a dramatic shift as more students reported testing their prototype by building it (Build) in Sample A. This also aligns with the PFX lens being implemented at the time, which was PFF (Prototype for Feasibility).

Sample B implemented PFF first, and so we would expect to see a higher percentage of students reporting that they built the prototype in order to test/evaluate technical quality; this is seen in

Figure 5-21. We do not see a similar shift in Sample B or C between Time 2 and Time 3. These results partially support our hypothesis that the sequence of PFX lenses would affect the prototyping awareness of students.

Prototype Evolution: How did the prototype develop?

In their open-ended responses, students clearly identified four ways that prototypes were improved upon or evolved since completion of the previous prototype: Desirability, Functionality,

Manufacturability, and Usability. Responses marked as Desirability typically contained text that mentioned improvements made to the prototype in order to make the design, product, or prototype more appealing to users. We also saw students cite the need to improve the technical quality of their designs, and responses that included similar sentiment were sorted into the Functionality category. Responses that were sorted into Functionality typically included text that referenced the need to improve the basic performance of the design. Students also identified the manufacturing methods used to create their prototypes as one area for improvement; responses sorted into the

Manufacturability bin typically referenced improvements to the CAD models or design that would

141 improve the production of components using 3D printing. Finally, the Usability category typically included responses that were related to the ease of use of the product; here, in general, students were concerned with improving the intuitiveness or instructions of the design.

At Time 2, when the open-ended questions were distributed in conjunction with the second

PAWS administration, 15.2% of responses were tagged as Desirability, 29.7% were tagged as

Functionality, 33.3% were tagged as Manufacturability, and 21.8% were tagged as Usability.

Within Sample A, the most frequently tagged mode of evolution was Functionality (31.3%). Within

Samples B and C, the most frequently tagged mode of evolution was Manufacturability (B: 42.5%,

C: 50%). Figure 5-23 highlights the distribution of tags across all three samples at Time 2.

At Time 3, when the open-ended questions were distributed for the second time in conjunction with the third and final PAWS administration, 22.2% of responses were tagged as

Desirability, 32.3% were tagged as Functionality, 32.9% were tagged as Manufacturability, and

12.6% were tagged as Usability. Within Sample A at Time 3, the most frequently tagged mode of evolution was manufacturability (30.9%). Within Samples B and C, the most frequently tagged mode of evolution was Functionality (B: 42.9%, C: 43.1%). Figure 5-24 highlights the distribution of tags across all three samples at Time 3.

142

Figure 5-23. Distribution of Evolution Tags at Time 2

Figure 5-24. Distribution of Evolution Tags at Time 3

143

From Figures 5-23 and 5-24 we see that students in Samples A and B had similar distributions for the improvements that they planned to make to their design based on their prototype. Students in Sample C reported that the majority of improvements made to the design would be in areas related to the designs’ Functionality or the Manufacturability of the prototype.

Samples A and B while higher in these two areas as well, also had a fair amount of responses categorized as Desirability and Usability. These results are interesting when we take into account the PFX lenses being implemented at Times 1, 2, and 3 in Samples A and B.

The final open-ended question asked students how they improved upon or changed their most recent prototype based on their previous prototype. So at Time 2, most students in Sample A reported that they had attempted to improve the technical Quality of the design, and Time 3 they reported that improvements were made in the Manufacturability of the prototype. This aligns with the lenses being used at both of those times, namely PFF and PFV, respectively. Students in Sample

B at Time 2 reported that they had focused on improving the overall Manufacturability of their prototypes for the second alpha prototype. Again this aligns with the PFX lens that was being implemented at this time, namely, PFF. It appears that students in Sample C, in general made, improvements in the same areas from prototype to prototype as we see very little of that sample reporting making any improvements related to Desirability or Usability at Time 2 or Time 3. These results partially support our hypothesis that the sequence of PFX lenses would affect the prototyping awareness of students.

5.5.4 Discussion of Results and Limitations

At the start of this study we were primarily interested in answering one research question: does the sequence of PFX lenses affect or impact students’ prototyping awareness? We started with the hypothesis that the sequence of PFX will impact prototyping awareness and that students will

144 be more aware of prototyping behaviors relevant to the lens to which they are being exposed. We also hypothesized that students in PFX classes would be more aware of prototyping behaviors as compared to students in control classes.

We saw in Sample A that participants rated themselves as being less aware or engaged with desirability prototyping behaviors during creation of the first and second alpha prototypes as compared to creation of the third alpha prototype. We also saw that participants in Sample A rated themselves as being less aware or engaged with feasibility prototyping behaviors during creation of the first alpha prototype as compared to creation of the second and third alpha prototypes.

Finally, we saw that participants in Sample A continuously increased in their awareness or engagement with viability prototyping behaviors during creation of the first, second, and third alpha prototypes throughout the course of the semester.

In Sample B, we saw that participants rated themselves as being less aware of viability prototyping behaviors during the creation of the first alpha prototype as compared to creation of the second and third alpha prototype. We did not observe any other statistically significant differences. We saw very interesting trends throughout Sample B as the implementation of the PFX framework progressed and these trends were different than the phenomena observed in Sample A.

In general we noticed that the qualitative results in Sample B often matched or were similar to the qualitative findings from Sample C at Time 2; however, at Time 3 we often saw the frequency of tags shift in Sample B responses, and this shift was often closer to the trends we found in Sample

After reviewing the qualitative results, we found that Sample A, in general, had very different distributions of tags across all categories as compared to Samples B and C. In Sample A, we saw at the time of the second PAWS distribution, students responded most frequently as building prototypes for users for the purpose of testing either the likability or usability of their prototype. Often students demonstrated their prototype, and we found that their responses for which

145 aspects of the prototype changed or evolved were fairly evenly distributed across all four evolution tags, with slightly higher frequencies in Functionality and Usability. At the time of the third PAWS administration, we found that the frequency of tags across categories shifted for Sample A. At the third time point, while Sample A still cited Users most frequently as the audience for their prototype, the were fairly evenly distributed across purpose of prototype with only a marginally higher frequency in usability as compared with Quality and Likability and tested these prototypes by building them (Build), as opposed to demoing them (Demo).

For Sample B at Time 2, students built prototypes (Build) for their team most frequently and indicated that these prototypes were created in order to evaluate the technical Quality of their overall design. However, at the third administration of PAWS, we saw that the audience for Sample

B’s prototypes shifted, and while Team was still the most frequently tagged audience, the frequency of tags was more evenly distributed across the other categories with a notable increase in testing with Users, which can be seen in Figure 5-16. We also saw a shift in the purpose of Sample B’s prototypes at Time 3 with most of the responses being sorted into Usability. For Sample C, we saw that the distribution of tags across categories predominately remained the same from Time 2 to

Time 3, with one notable difference occurring in purpose. At the third distribution of the PAWS, there was a spike in the purpose category, and we noticed that most students in this sample were testing the technical Quality of their design.

Based on the quantitative data we found partial support for our first hypothesis, but we were unable to find support for our second hypothesis (which was: the sequence of PFX will not affect product outcomes; specifically there will be no significant difference in designs produced using PFX lenses regardless of lens order). In the within subjects experiment, we found that in

Sample A students self-reported prototyping awareness shifted, however we did not see this shift in Sample B. When we look at the qualitative data, we find a different story. Based on the responses to the open-ended questions we saw that students in both Samples A and B were shifting their

146 prototyping behaviors in accordance with the lenses that were being implemented. In general, we did not see the same shifts happening in the control section, or Sample C. These finding from the quantitative data and qualitative data could indicate that the current version of the PAWS is inducing some form of bias that is impacting the way students answer the scale. We hypothesize that students across groups viewed all the behaviors presented in PAWS as behaviors they ought to be aware of during prototyping. As such, students may have responded with what they believed they should be doing. Thus, our results might not be representative of what students are actually aware of during prototyping activities. Future iterations of the PAWS will experiment with a change in the initial phrase from “when developing my prototype…” to “How often did you or your team engage in the following behaviors when developing your prototype?”. This would then change the scale from agreement to frequency, which could help reduce some of the bias by forcing students to think back to actual instances of these behaviors and count them.

Based on these findings, we hypothesize that, in general, students “prototyping in the wild” are more likely to focus on the technical quality and functionality of their design, and ignore the desirability or viability of the product. These results do indicate that the sequence of PFX lenses does affect prototyping awareness and prototyping behaviors, but further work is needed to understand fully how the sequence of lenses is impacting prototyping awareness.

5.6 Implications

The results of our findings indicate that the sequence of PFX lenses might have some effect on product outcomes. This is an important finding because if students or designers wish to increase certain aspects of a product (for example, the quality or emotional value), then it is possible that they could adjust the sequence of PFX lenses to optimize these outcomes. Future work will explore how PFX lenses are affecting the evolution of product characteristics by not only evaluating final

147 prototypes or design artifacts but evaluating the prototypes and design artifacts produced throughout the design process. This will provide insights into how the nature of the product is shifting and what, if any, effect PFX lens ordering has on this shift.

We also found partial support for our third hypothesis that students’ prototyping awareness would be effected by the sequence of PFX lenses. While we did not see many significant differences in PAWS scores between samples at Times 1, 2, and 3 we did find that within samples prototyping awareness was shifting with time. We found partial support for our hypothesis from our qualitative analysis, which revealed that students reported different audiences for their prototypes, different purposes their prototypes served, different methods to test their prototypes, and different ways their prototype evolved based on which sample they were in.

This is an important finding for two reasons. First, this indicates that further work is needed to refine PAWS in order to fully capture the prototyping awareness of students and designers. We found that the results from the qualitative analysis were not reflected in the scale results, which could indicate that some flaws exist within the measure based on how students are being asked to assess their prototyping awareness. Second, these findings suggest that prototyping awareness can be shifted accordingly with PFX lenses. We feel that higher prototyping awareness is a first step in developing higher prototyping self-efficacy, which can lead to better prototyping outcomes and higher levels of motivation throughout prototyping activities. If PFX can shift designers’ prototyping awareness through desirability, feasibility, and viability behaviors, then designers would theoretically be able to implement the PFX lens in which they wish to increase prototyping self-efficacy. Future work is needed to fully explore the impact the sequence of PFX lenses has on prototyping awareness.

148

5.7 A Look Back and a Look Ahead

Engineering students are expected to develop innovative products that solve some of the world’s toughest challenges as soon as they enter the workforce. The Accreditation Board for

Engineering and Technology (ABET) requires engineering graduates to have “an ability to design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability”

[187]. A thorough understanding and mastery of the design process is necessary to solve these difficult challenges; however, one of the most critical stages of the design process, prototyping, has remained largely unstructured and unstudied. Our work aligns with the National Science

Foundation’s second strategic goal “stimulate innovation and address societal needs through research and education” [188]. By educating engineering students to develop products that incorporate societal, economic, and technical perspectives, we are encouraging the development of innovative solutions and individuals. Engineers and students exposed to this work will be able to design a technically feasible product or system that meets the needs of a complex global market and user base.

In this chapter, we investigated the impact the sequencing of PFX lenses has on product and people outcomes. We found that PFX lens order impacts certain qualities of the product and has some impact on prototyping awareness. Future work is needed to validate these findings and dig deeper into the phenomena behind them. In the next chapter, we discuss the contributions of this work in engineering design and engineering education and review future areas of research.

149

Chapter 6

Contributions and Future Work

6.1 Summary and Conclusions

As previously stated, $141.8 billion is invested by large companies in product research and design activities (Cooper, 2001), and we noted that 40-50% of that money is spent on cancelled or failed products. Prototyping is clearly one of the most critical activities during the design process, and it can represent one of the largest sunk costs during product development for many companies.

Our work has the potential to reduce the amount of failed products by improving prototyping practice. PFX enables designers to focus their prototyping efforts, allowing for more effective prototyping activities that lead to deeper insights about the product being designed. We believe

PFX will help designers identify critical issues with the design that may lead to failure earlier in the design process as compared to traditional prototyping techniques. Ultimately, this could reduce the failure rate of new products entering the market.

In Chapter 1, prototyping research from the fields of engineering design, engineering education, management science, and human computer interaction was reviewed with a focus on developing specifications for structured and holistic prototyping frameworks. Existing prototyping frameworks and models were introduced, and the need was made clear for a structured and holistic prototyping framework. Specifically, we found that a prototyping framework should (1) encourage iterative prototype development early and often, (2) enable design teams to quickly select a prototyping focus, (3) enable the development of prototypes that engage the user and the design team to maximize insights, and (4) promote a prototype appropriate for the level of interaction desired by the design team. To validate this need, we conducted a small pilot study to understand engineering students’ perceptions of prototypes and prototyping activities. We found that

150 engineering students hold imprecise and incomplete perceptions about the purpose and value of prototypes and prototyping activities within the design process. Finally, the research tasks and objectives of this work were presented.

Chapter 2 introduced the Prototype for X (PFX) framework and presented the elements of the framework and the initial lenses—desirability, feasibility, and viability—to test its utility. We began by reviewing the theoretical underpinnings that PFX draws on, such as the Human Centered

Design framework, Design Thinking methods, and Design for X. We then reviewed the elements at the core of the PFX framework, namely, frame for x, build for x, and test for x. We concluded this chapter by reviewing the initial lenses used to evaluate the impact of the PFX method on both design and designer outcomes, prototyping for desirability, prototyping for feasibility, and prototyping for viability.

A case study evaluating the impact of PFX on product outcomes was discussed in Chapter

3. We began this chapter by reviewing the applicability of traditional ideation metrics to functional prototypes; these metrics were found to be inappropriate for prototypes, and new metrics related to a prototype’s desirability, feasibility, and viability were proposed. We then reviewed a case study evaluating the impact of PFX in a junior-level mechanical engineering design course. We found that the PFX framework significantly impacts a final design’s desirability and viability; functionality was not significantly different given the nature of the study’s design challenge.

In Chapter 4 we used the same case study to evaluate the impact of PFX on students’ prototyping awareness. We introduced the PAWS, or Prototyping AWareness Scale, as a method to measure designers’ awareness of their prototyping behaviors. By examining engineering students in the PFX class, we found that engineering students’ prototyping awareness shifts during prototyping activities. We also saw that students in control and PFX classrooms rated themselves as equally aware of prototyping behaviors, even though final designs did not reflect this awareness.

151

Having validated that PFX had some impact on the final design and engineering students’ prototyping awareness, we explored the question: how does the sequence of PFX lenses impact product outcomes and prototyping awareness in Chapter 5. We implemented two different sequences of PFX lenses in a junior-level mechanical engineering design course in addition to studying a control group within the same course. We found that the lens ordering does affect certain facets of the prototype and student’s prototyping awareness.

6.2 Research Contributions

The contributions from this research lie primarily in the field of engineering design, specifically in the area of prototyping during the new product design process. They include:

 Establishment of four specifications for a structured and holistic prototyping framework

o Based on an extensive literature review, four specifications were proposed that

could ensure prototyping is taught and practiced effectively in academic and

professional settings. While previous prototyping frameworks meet some of the

specifications, none meet all of them. These specifications can be used as a

roadmap for future researchers exploring structured prototyping frameworks and

formal methods for prototyping during new product development.

 Development of the Prototype for X framework

o Prototype for X (PFX) is a novel framework that guides designers during

prototyping activities throughout the design process. We developed three phases

of PFX, namely, frame for x, build for x, and test for x. Through these phases, PFX

helps designers focus their resources and efforts on building prototypes that test

core assumptions and lead to deeper and richer learning about specific aspects of

the design at the time of testing. Drawing from Human-Centered Design, Design

152

for X, and Design Thinking, we confirmed that prototyping methods focused on

three lenses of product design (i.e., desirability, feasibility, and viability) can

positively impact final design outcomes. Up to this point, no previous work in the

published prototyping literature has focused on these lenses or proposed phases

similar to frame, build, test—or validated the impact of a structured prototyping

process on an end design’s outcome(s).

 Creation of alternative metrics to evaluate prototypes

o Little to no prototyping research has explored the impact of a structured

prototyping framework on the multi-faceted nature of products. Previous research

has traditionally used binary metrics to indicate a successful or unsuccessful design

or prototype. Using a variety of metrics from management science, engineering

design, and marketing studies we highlighted the need for more applicable and

relevant prototyping metrics.

 Creation of a scale to measure prototyping awareness

o PAWS was created to evaluate students’ prototyping awareness. While previous

research has explored the psychological experience of prototyping or the design

self-efficacy of students, these works could either not be scaled or did not provide

the depth necessary to explore prototyping awareness in a large junior-level design

class. PAWS was successfully tested in this work and future areas for improvement

identified.

 Validation of the effect of a structured prototyping framework on the many facets of a

product

o While previous work has explored the impact of prototyping frameworks or

methods on the success or failure of end designs, these studies have traditionally

used binary metrics to evaluate this impact. We validated that a structured

153

prototyping framework impacts end designs across a variety of facets, and we

confirmed that a structured prototyping framework can positively impact the

desirability, feasibility, and viability of end designs.

 Partial validation of the effect of a prototyping framework on the prototyping awareness of

novice designers

o We found that students’ prototyping awareness shifts during the new product

development process by exploring students’ open-ended responses to two

prototyping questions. We saw that students in both the control and experimental

classrooms rated themselves as equally aware of their prototyping behaviors, even

though final designs did not reflect this awareness. From these findings, we suggest

further development of PAWS to capture students’ prototyping awareness more

accurately.

 Exploration of the effect of lens sequence on product and people outcomes

o The results of our findings indicate that the sequence of PFX lenses might have

some effect on product outcomes. No other prototyping research has explored the

effect of timing and framing on product outcomes.

o We also found that students’ prototyping awareness was effected by the sequence

of PFX lenses. Our findings suggest that prototyping awareness can be shifted

accordingly with PFX lenses. No other prototyping research has explored the shift

in a designer’s awareness of prototyping throughout the product design process.

This work is the first step in understanding this shift.

154

6.3 Limitations and Shortcomings

This work was limited due to the population of subjects and the nature and scope of design challenges investigated in this work. Because we only tested PFX with an engineering student population in a junior-level mechanical engineering design course, we are unable to generalize these results to professional settings. We assert that this work is still useful, however, as it highlights the utility of PFX as an educational tool to improve prototyping practices in novice designers in engineering. Future work will implement PFX in professional settings; so, findings can be generalized to industry.

Our work was also limited by the nature of both design challenges reported on in this work.

The first design challenge (convert a hand-held drill into a hand-held vacuum) was more defined and well scoped than the second design challenge (create an educational kit that utilizes 3D printing in some way). The difference in constraints between each design challenge most likely affected the results reported on in Chapters 3-5. Because the second design challenge, reported in Chapter

5, was more open-ended than the challenge reported on in Chapter 3, we were unable to find a standard or quantitative way to evaluate the technical quality of final designs. This, once again, highlights the importance of developing rigorous and informative metrics to evaluate prototypes, independent of design challenges.

Finally, this work was limited by the scope of the design challenges presented to students.

While these challenges were more complex than the design tasks reported on in previous prototyping studies [14,55,80,107], they were still not representative of the complexities present in professional or industrial design settings. While our work highlighted the potential impact, a structured and holistic prototyping framework might have on end designs and designers’, further work is needed to fully explore the influence of PFX in real world design settings.

155

6.4 Areas for Future Work

Prototyping represents one of the largest sunk costs during new product development, yet it is one of the least researched design activities. We have been motivated in this graduate research to study both designers and design outcomes during prototyping from the perspective of design thinking in order to gain insight into both the decisions that occur and the products that are produced. This work has led to new insights and new understanding of prototyping and its role in product design and development.

This work has focused on developing a structured and holistic prototyping framework within engineering design in order to help translate innovative ideas into successful products and end designs. Our goal was to develop a structured and holistic prototyping framework that can be used in industry to more efficiently and effectively prototype by integrating best practices from related literature. Future work will focus on validating the PFX framework in industry settings and studying the effects of PFX on the designers’ understanding of prototyping, the decisions made during prototyping, and the artefacts produced during prototyping. The following areas within design theory and methodology should be explored in future work.

6.4.1 Prototype for X: Translation to Industry and Beyond

We are interested in integrating PFX into real-world design environments and exploring prototyping practices within industry. Currently, PFX has been shown to positively impact student designs and prototyping awareness. In order to continue studying this work, we will extend studies into industries, specifically those that develop physical products (as opposed to digital). In future research we will work to answer the following questions regarding the translation of PFX to industry:

156

1) What methods are currently used by designers during prototyping and how might

PFX be integrated into existing practice? We would like to study design teams in

industry throughout prototyping activities in order to identify gaps in current practices.

This will enable further development of PFX and provide valuable insights into the

most useful applications of a prototyping framework for industry. Ultimately, this

research will aid the translation of PFX from an academic context to an industry

context.

2) Does PFX affect end designs as well as the designers themselves in industry? Using

industry partners, we will study how PFX is implemented in design teams and explore

the effects of PFX on both the designers themselves, as well as the end designs.

Previous work has studied the effects of PFX on student teams and designs [2] and

results indicate that PFX leads to improvements in manufacturability and user

satisfaction; however, student teams and semester long projects do not realistically

mimic design in industry. By studying the effects of PFX in industry, we can validate

or invalidate these early findings and evolve PFX into a design tool that is more readily

used. Answering this question will help validate and improve the PFX framework and

create useful partnerships with industry partners for future work.

3) How can PFX be adapted to fit the needs of a particular industry or company?

Working with industry partners, we will focus on tailoring the PFX framework for

desired outcomes, such as designs that are easier to manufacture. The goal of this work

would be to develop a library of PFX applications that led to specific designer and

design outcomes to be used in both academia and industry during prototyping.

157

6.4.2 Prototype for X: Development of Appropriate and Rigorous Metrics for Assessing Prototyping

In order to evaluate the effectiveness of structured prototyping frameworks, such as PFX, it is necessary to evaluate the prototypes developed using objective measures of relevant criteria or characteristics of the design. While studying the effectiveness of PFX, we realized that a critical gap exists in design research, namely, that appropriate metrics to evaluate functional prototypes do not exist. There has been a multitude of studies proposing metrics for the evaluation of ideas during concept generation and selection, and similarly, metrics exist to evaluate products already on the market. Metrics to evaluate the design artefacts that exist “between” these phases do not exist.

Ideation metrics, while useful for the selection of the most novel or feasible idea or design concept, do not provide designers with useful insights into prototypes, because they lack the depth and perspective necessary to thoroughly evaluate a physical prototype. Metrics used to evaluate final designs, such as user-perceived value or user satisfaction, often require a level of fidelity or “polish” that most prototypes will not reasonably be able to achieve. In future work we will continue to develop a set of rigorous and appropriate metrics to evaluate prototypes. Specifically, we are interested in developing a set of metrics that can be design-independent and will provide designers and design teams with deeper insights into their prototypes.

158

References [1] Cooper, R. G., 2001, Winning at new products: accelerating the process from idea to

launch, New York, NY.

[2] Camburn, B., Dunlap, B., Gurjar, T., Hamon, C., Green, M., Jensen, D., Crawford, R., Otto,

K., and Wood, K., 2015, “A Systematic Method for Design Prototyping,” Journal of

Mechanical Design, Vol. 137, No. 8, pp. 81-102.

[3] Wall, M. B., Ulrich, K. T., and Flowers, W. C., 1992, “Evaluating prototyping technologies

for product design,” Ressearch in Engineering Design, Vol. 3, No. 3, pp. 163–177.

[4] Buchenau, M., and Suri, J. F., 2000, “Experience prototyping", Proceedings of the

Conference on Designing Interactive Systems Processes Practices Methods and

Techniques, August 17-19, New York, NY, ACM, Vol. 3, pp. 424–433.

[5] Liu, L., and Khooshabeh, P., 2003, “Paper or interactive?: A study of prototyping techniques

for ubiquitous computing environments,” Proceedings from Computer Human Interactions

Extended Abstracts on Human Factors in Computing Systems, Fort Lauderdal, FL, ACM,

pp. 1030–1031.

[6] Dow, S., Glassco, A., and Kass, J., 2011, "The effect of parallel prototyping on design

performance, learning, and self-efficacy", Stanford Tech Report, Vol. 10, No.1, pp 1-10.

[7] Dow, S., Fortuna, J., Schwartz, D., Altringer, B., and Klemmer, S., 2011, “Prototyping

Dynamics : Sharing Multiple Designs Improves Exploration, Group Rapport, and Results,”

Design Thinking Research: Measuring Performance in Context (H. Plattner, C. Meinel, L.

Leifer). Springer Berlin Heidelberg, Berlin, Germany, pp. 47-70.

[8] Walker, M., Takayama, L., and Landay, J., 2002, “High-Fidelity or Low-Fidelity, Paper or

Computer? Choosing Attributes when Testing Web Prototypes,” Proceedings from the

Human Factors and Ergonomic Society Annual Meetnig, Los Angeles, CA, Vol. 46, No. 5,

pp. 661–665.

159

[9] Doke, E. R., and Swanson, N. E., 1995, “Decision variables for selecting prototyping in

information systems development: A Delphi study of MIS managers,” Information and

Management, Vol. 29, No. 4, pp. 173–182.

[10] Smith, R. . P., and Eppinger, S. D., 1997, “A Predictive Model of Sequential Iteration in

Engineering Design,” Management Science, Vol. 43, No. 8, pp. 1104–1120.

[11] Dyson, 2013, “James Dyson,” Insid. Dyson [Online]. Available:

http://content.dyson.co.uk/insideDyson/#jamesdyson.

[12] Montini, L., 2014, “Hardware Startup LittleBits Embraces the Internet of Things,” Inc.

[Online]. Available: http://www.inc.com/laura-montini/modular-hardware-company-

littlebits-embraces-the-internet-of-things.html. [Accessed: 01-Apr-2017].

[13] Camburn, B. A., Dunlap, B. U., Kuhr, R., Viswanathan, V. K., Linsey, J. S., Jensen, D. D.,

Crawford, R. H., Otto, K., and Wood, K. L., 2013, “Methods for Prototyping Strategies in

Conceptual Phases of Design: Framework and Experimental Assessment,” ASME 2013

Design Engineering Technical Conferences-Design Theory and Methodology Conference,

June 5-9, Portland, OR, ASME, Paper No. DETC2013/DTM-13072

[14] Martin, A. S., 2015, “A little bit of creativity: How littleBits fueled electronic

experimentation at TEDWomen,” TedBlog [Online]. Available: http://blog.ted.com/how-

littlebits-fueled-electronic-experimentation-at-tedwomen/. [Accessed: 01-Apr-2017].

[15] Elliker, T., 2016, “Ringly Dominates by Thinking Like a Customer,” Mirror [Online].

Available:https://www.mirror.me/ringly-dominates-by-thinking-like-a-customer/.

[Accessed: 01-Apr-2017].

[16] Christie, E. J., Jensen, D. D., Buckley, R. T., Menefee, D. A., Ziegler, K. K., Wood, P. K.

L., and Crawford, R. H., 2012, “Prototyping Strategies : Literature Review and

Identification of Critical Variables,” Proceedings from the 2012 American Society of

Engineering Education Annual Conference, June 9-12, San Antonio, TX.

160

[17] Gerber, E., 2009, “Prototyping: Facing Uncertainty Through Small Wins,” Proceedings of

the 17th International Conference on Engineering Design, August 24-27, Palo Alto, CA,

Vol. 0, No. 8, pp. 333–342.

[18] Rhinow, H., 2012, “Design Prototypes as Boundary Objects in Innovation Processes,”

Proceedings of the 2012 International Conference on Design Research Society, July 1-4,

Bangkok, Thailand, Vol. 1, No. 1, pp. 1–10.

[19] Gabrysiak, G., Giese, H., and Seibel, A., 2011, “Towards Next Generation Design Thinking:

Scenario-Based Prototyping for Designing Complex Software Systems with Multiple

Users,” Design Thinking: Understand – Improve – Apply (H. Plattner, C. Meinel, L. Leifer).

Springer Berlin Heidelberg, Berlin, Germany, pp. 219–236.

[20] Dunlap, B. U., Hamon, C. L., Camburn, B. A., Crawford, R. H., Green, M. G., and Wood,

K. L., 2014, “Hueristics-Based Prototyping Strategy Formation - Development and Testing

of a New Prototype Planning Tool.” ASME 2014 Mechanical Engineering Congress and

Exposition: Systems, Design, and Complexity, November 14-20, Montreal, Canada, ASME,

Paper No. IMECE2014-39959

[21] Viswanathan, V. K., and Linsey, J., 2011, “Design Fixation in Physical Modeling: An

Investigation on the Role of Sunk Cost,” ASME 2011 Design Engineering Technical

Conferences-Design Theory and Methodology Conference, August 28-31, Washington,

D.C., ASME, Paper No. DETC2011-47862

[22] Menold, J., Simpson, T. W., and Jablokow, K. W., 2016, “The Prototype for X Framework:

Assessing the Impact on Desirability, Feasibility, and Viability of End Designs,” ASME

2016 Design Engineering Technical Conferences-Design Theory and Methodology

Conference, August 24-26, Charlotte, NC, ASME, Paper No. DETC2016-60225.

[23] Menold, J., Jablokow, K. W., Simpson, T. W., and Waterman, E. A., 2016, “The Prototype

for X (PFX) Framework: Assessing Its Impact on Students´ Prototyping Awareness,”

161

Proceedings from the 2016 American Society of Engineering Education Annual Conference,

June 26-June 30, New Orleans, LA.

[24] Ben-Arieh, D., and Qian, L., 2003, “Activity-based cost management for design and

development stage,” International Journal of Product Economics, Vol. 83, No. 2, pp. 169–

183.

[25] Krishnan, V., and Ulrich, K. T., 2001, “Product Development Decisions: A Review of the

Literature,” Management Science, Vol. 47, No. 1, pp. 1–21.

[26] Thomke, S., and Bell, D. E., 2001, “Sequential Testing in Product Development,”

Management Science, Vol. 47, No. 2, pp. 308–323.

[27] Thomke, S., 1998, “Managing experimentation in the design of new products,”

Management Science, Vol. 44, No. 6, pp. 743–762.

[28] Dahan, E., and Mendelson, H., 2001, “An Extreme-Value Model of Concept Testing,”

Management Science, Vol. 47, No. 1, p. 102-122.

[29] Tornberg, K., Jämsen, M., and Paranko, J., 2002, “Activity-based costing and process

modeling for cost-conscious product design: A case study in a manufacturing company,”

International Journal of Product Economics, Vol. 79, No. 1, pp. 75–82.

[30] Loch, C. H., Terwiesch, C., and Thomke, S., 2001, “Parallel and Sequential Testing of

Design Alternatives,” Management Sciecne, Vol. 47, No. 5, pp. 663–678.

[31] Terwiesch, C., and Loch, C., 2004, “Collaborative prototyping and the pricing of custom-

designed products,” Management Science, Vol. 50, No. 2, pp. 145–158.

[32] Lemons, G., Carberry, A., Swan, C., Jarvin, L., and Rogers, C., 2010, “The benefits of

model building in teaching engineering design,” Design Studies, Vol. 31, No. 3, pp. 288–

309.

[33] Yang, M. C., 2005, “A study of prototypes, design activity, and design outcome,” Design

Studies, Vol. 26, No. 6, pp. 649–669.

162

[34] McKim, R. 1972, Experiences in Visual Thinking, Belmont, CA.

[35] Perry, M., and Sanderson, D., 1998, “Coordinating joint design work: the role of

communication and artefacts,” Design Studies, Vol. 19, No. 3, pp. 273–288.

[36] Sass, L., and Oxman, R., 2006, “Materializing design: The implications of rapid prototyping

in digital design,” Design Studies, Vol. 27, No. 3, pp. 325–355.

[37] Crilly, N., Moultrie, J., and Clarkson, P. J., 2004, “Seeing things: Consumer response to the

visual domain in product design,” Design Studies, Vol. 25, No. 6, pp. 547–577.

[38] Yang, M. C., and Daniel, J., 2005, “A study of prototypes, design activity, and design

outcome Maria,” Design Studies, Vol. 26, No. 2, pp. 649–669.

[39] Chamorro-Koc, M., Popovic, V., and Emmison, M., 2008, “Using visual representation of

concepts to explore users and designers’ concepts of everyday products,” Design Studies,

Vol. 29, No. 2, pp. 142–159.

[40] Häggman, A., and Yang, M. C., 2013, “The Influence of Timing in Exploratory

Prototypiung and Other Activities in Design Projects," ASME 2013 Design Engineering

Technical Conferences-Design Theory and Methodology Conference, June 5-9, Portland,

OR, ASME, Paper No. DETC2013-12700 .

[41] Macomber, B., and Yang, M., 2011, “The Role of Sketch Finish and Style in User

Responses To Early State Design Concepts,” ASME 2011 Design Engineering Technical

Conferences-Design Theory and Methodology Conference, August 28-31, Washington,

D.C., ASME, Paper No. DETC2011-48714.

[42] Elsen, C., Häggman, A., Honda, T., and Yang, M. C., 2012, “Representation in early stage

design: an analysis of the influence of sketching and prototyping in design projects,” ASME

2012 Design Engineering Technical Conferences-Design Theory and Methodology

Conference, August 12-15, Chicago, IL, ASME, Paper No. DETC2012-70248.

[43] Genco, N., Johnson, D., Hoelttae-Otto, K., and Seepersad, C. C., 2012, “A Study of the

163

Effectiveness of Empathic Experience Design As a Creativity Technique,” ASME 2011

Design Engineering Technical Conferences-Design Theory and Methodology Conference,

August 28-31, Washington, D.C., ASME, Paper No. DETC2011-48256.

[44] Neeley, L. W., Lim, K., Zhu, A., and Yang, M. C., 2013, “Building fast to think faster:

exploiting rapid prototyping to accelerate ideation during early stage design,” ASME 2011

Design Engineering Technical Conferences-Design Theory and Methodology Conference,

August 28-31, Washington, D.C., ASME, Paper No. DETC2011-12635.

[45] Maria C. Yang;, 2007, “An Analysis of Sketching Skill and Its Role in Early Stage

Engineering Design,” Journal of Mechanical Deisgn, Vol. 129, No. 5, pp. 476–482.

[46] Sefelin, R., Tscheligi, M., and Giller, V., 2003, “Paper prototyping - what is it good for?: a

comparison of paper- and computer-based low-fidelity prototyping,” Proceedings from

Computer Human Interactions Extended Abstracts on Human Factors in Computing

Systems, Fort Lauderdal, FL, ACM, pp. 778–779.

[47] Brereton, M., and Mcgarry, B., 2000, “An Observational Study of How Objects Support

Engineering Design Thinking and Communication: implications for the design of tangible

media,” Proceedings of the SIGCHI conference on Human Factors in Computing Systems,

April 1-6, Hague, The Netherlands, pp. 217-224.

[48] Baecker, R., Nastos, D., Posner, I. R., and Mawby, K., 1993, “The User-centered iterative

design of collaborative writing software,” Proceedings of the INTERACT '93 and CHI '93

Conference on Human Factors in Computing Systems, April 24-29, Amsterdam, The

Netherlands, pp. 399–405.

[49] McCurdy, M., Connors, C., Pyrzak, G., Kanefsky, B., and Vera, A., 2006, “Breaking the

Fidelity Barrier - An Examination of our Current Characterization of Prototypes and an

Example of a Mixed-Fidelity Success,” Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems, April 22-27, Montreal, Canada, pp. 1233–1242.

164

[50] Tohidi, M., Buxton, W., Baecker, R., and Sellen, A., 2006, “Getting the right design and

the design right,” Proceedings of the SIGCHI Conference on Human Factors in Computing

Systems, April 22-27, Montreal, Canada, pp. 1243–1252.

[51] Sellen, K. M., Massimi, M. A., Lottridge, D. M., Truong, K. N., and Bittle, S. A., 2009,

“The people-prototype problem: understanding the inteaction between protoype format and

user group,” Proceedings of the SIGCHI Conference on Human Factors in Computing

Systems, April 4-9, Boston, MA, pp. 635-638.

[52] Viswanathan, V. K., and Linsey, J. S., 2013, “Training future designers: A study on the role

of physical models,” Proceedings from the ASEE Annual Conference and Exposition, June

23-26, Atlanta, GA.

[53] Viswanathan, V. K., and Linsey, J. S., 2009, “Enhancing student innovation: Physical

models in the idea generation process,” Proceedings from 39th Frontiers in Education

Conference, October 18-21, San Antonio, TX.

[54] Wright, S. M., Silk, E. M., Daly, S. R., Jablokow, K. W., and Yilmaz, S., 2015, “Exploring

the effects of problem framing on solution shifts: A case study,” Proceedings from the 122nd

ASEE Annual Conference and Exposition, June 15-19, Seattle, WA.

[55] Hammon, C. L., Green, M. G., Dunlap, B. U., Camburn, B. A., Crawford, R., & Jensen, D.,

2014, “Virtual or Physical Prototypes? Proceedings from the 121st ASEE Annual

Conference and Exposition, June 14-17, Indianapolis, IN p. 9025.

[56] Aurigemma, J., Chandrasekharan, S., Nersessian, N. J., and Newstetter, W. C., 2013,

“Design in the Wild,” ASEE Prism, Vol. 22, No. 7, pp. 45-55.

[57] Kazmer, D. O., 1998, “On the Divergence of Case Studies and Hardware Prototypes in

Active Learning,” Proceedings from ASEE Northeast Regional Meeting, Amherst, MA.

[58] Rodriguez, W., 1991, “Building Real and Virtual Design-Prototypes,” Proceedings from

the ASEE Annual Conference and Exposition, pp. 1199–1203.

165

[59] Toh, C. A., and Miller, S. R., 2014, “The Impact of Example Modality and Physical

Interactions on Design Creativity,” Journal of Mechanical Design, Vol. 136, No. 9.

[60] Viswanathan, V. K., and Linsey, J. S., 2012, “Physical Models and Design Thinking: A

Study of Functionality, Novelty and Variety of Ideas,” Journal of Mechanical Design, Vol.

134, No. 9., pp.1004-1017.

[61] Stone, R. B., and Wood, K. L., 2000, “Development of a Functional Basis for Design,”

Journal of Mechanical Design, Vol. 122, No. 4., pp. 359-368.

[62] Jang, J., and Schunn, C. D., 2012, “Physical Design Tools Support and Hinder Innovative

Engineering Design,” Journal of Mechanical Design, Vol. 134, No. 2. p. 41001.

[63] Viswanathan, V. K., and Linsey, J. S., 2013, “Role of Sunk Cost in Engineering Idea

Generation: An Experimental Investigation,” Journal of Mechanical Design, Vol. 135, No.

12.

[64] Jobst, B., and Meinel, C., 2014, “How Prototyping Helps to Solve Wicked Problems,”

Design Thinking Research-Building Innovation Eco-Systems (H. Plattner, C. Meinel, L.

Leifer). Springer Berlin Heidelberg, Berlin, Germany pp. 105–113.

[65] Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., and Klemmer, S. R., 2012,

“Parallel prototyping leads to better design results, more divergence, and increased self-

efficacy,” Design Thinking Research: Studying Co-Creation in Practice (H. Plattner, C.

Meinel, L. Leifer). Springer Berlin Heidelberg, Berlin, Germany, pp. 127–153.

[66] Gabrysiak, G., Giese, H., and Beyhl, T., 2012, “Virtual multi-user software prototypes III,”

Design Thinking Research: Studying Co-Creation in Practice (H. Plattner, C. Meinel, L.

Leifer). Springer Berlin Heidelberg, Berlin, Germany, pp. 263–284.

[67] Sadler, J., Shluzas, L. A., Blikstein, P., and Srivastava, S., 2016, “Can Anyone Make a

Smart Device? Evaluating the Usability of a Prototyping Toolkit for Creative Computing,”

Design Thinking Research: Takig Breakthrough Innovation Home (H. Plattner, C. Meinel,

166

L. Leifer). Springer Berlin Heidelberg, Berlin, Germany, pp. 147–160.

[68] Sadler, J., Shluzas, L., Blikstein, P., and Katila, R., 2015, “Building blocks of the maker

movement: Modularity enhances creative confidence during prototyping,” Design Thinking

Research: Making Design Thinking Foundational (H. Plattner, C. Meinel, L. Leifer).

Springer Berlin Heidelberg, Berlin, Germany, pp. 141–154.

[69] Anderson, E. W., Fornell, C., and Lehmann, D. R., 1994, “Customer Satisfaction, Market

Share, and Profitability: Findings from Sweden,” Journal of Marketing, Vol. 58, No. 3, pp.

53–66.

[70] Westbrook, R. A., 1980, “A Rating Scale for Measuring Product/Service Satisfaction,”

Journal of Marketing, Vol. 44, No. 4, pp. 68–72.

[71] Guido, G., Prete, M. I., and Sammarco, L., 2010, “Prototypes versus examples: A new

model of online credibility for commercial websites,” Journal of Targeting, Measurement,

and Analysis for Marketing, Vol. 18, No. 1, pp. 3–16.

[72] Cartwright, A. J., 1997, “Interactive prototyping—a challenge for computer based design,”

Research in Engineering Design, Vol. 9, No. 1, pp. 10–19.

[73] Eckert, C., Clarkson, P. J., and Zanker, W., 2004, “Change and customisation in complex

engineering domains,” Research in Engineering Design, Vol. 15, No. 1, pp. 1–21.

[74] Chen, C., Kasof, J., Himsel, A., Dmitrieva, J., Dong, Q., and Xue, G., 2005, “Effects of

Explicit instruction to ‘be creative’ across domains and cultures,” Journal of Creative

Behavior, Vol. 39, No. 2, pp. 89–110.

[75] Gero, J. S., 2011, “Fixation and commitment while designing and its measurement,” Journal

of Creative Behavior, Vol. 45, No. 2, pp. 108–115.

[76] Lim, Y.-K., Stolterman, E., and Tenenberg, J., 2008, “The anatomy of prototypes,” ACM

Transactions on Computer-Human Interaction, Vol. 15, No. 2, pp. 1–27.

[77] Svanaes, D., and Seland, G., 2004, “Putting the users center stage: Role playing and low-fi

167

prototyping enable end users to design mobile systems,” Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems, April 24-29, Vienna, Austria pp.

479–486.

[78] Gill, C., Sanders, E., and Shim, S., 2011, “Prototypes as Inquiry, Visualization and

Communication,” Proceedings of the 13th International Conference on Engineering and

Product Design Education, London, England, pp. 672–677.

[79] Ulrich, K. T., and Eppinger, S. D., 2012, Product Design and Development. New York, NY.

[80] Gero, J. S., 1990, “Design Prototypes: A Knowledge Representation Schema for Design,”

AI Magazine, Vol. 11, No. 4, pp. 26.

[81] John, B. E., and Salvucci, D. D., 2005, “Multipurpose prototypes for assessing user

interfaces in pervasive computing systems,” IEEE Pervasive Computing, Vol. 4, No. 4, pp.

27–34.

[82] Sauer, J., Seibel, K., and Rüttinger, B., 2010, “The influence of user expertise and prototype

fidelity in usability tests,” Applied Ergonomics, Vol. 41, No. 1, pp. 130–140.

[83] Rudd, J., Stern, K., and Isensee, S., 1996, “Low vs. high-fidelity prototyping debate,”

Interactions, Vol. 3, No.1, pp. 76–85.

[84] Brereton, M., 2004, “Distributed cognition in engineering design: Negotiating between

abstract and material representations,” Design Representation (G. Goldschmidt, W. Porter).

London, England, pp. 83–103.

[85] Do, E. Y. L., and Gross, M. D., 2001, “Thinking with diagrams in architectural design,”

Artificial Intelligence Review, Vol. 15, No. 2, pp. 135–149.

[86] Viswanathan, V., and Linsey, J., 2011, “Understanding physical models in design cognition:

A triangulation of qualitative and laboratory studies,” Proceedings from 39th Frontiers in

Education Conference, October 18-21, San Antonio, TX.

[87] Viswanathan, V., Atilola, O., Esposito, N., and Linsey, J., 2014, “A study on the role of

168

physical models in the mitigation of design fixation,” Journal of Engineering Design, Vol.

25, No. 3, pp. 25–43.

[88] Walker, M., Takayama, L., and Landay, J. A., 2002, “High-Fidelity or Low-Fidelity, Paper

or Computer? Choosing Attributes When Testing Web Prototypes,” Proceedings of the

Human Factors and Ergonomics Society Annual Meeting, Septemeber 15-18, Los Angeles,

CA, pp. 661–665.

[89] Lim, Y.-K., Pangam, A., Periyasami, S., and Aneja, S., 2006, “Comparative analysis of

high- and low-fidelity prototypes for more valid usability evaluations of mobile devices,”

Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing

Roles, October 14-18, Oslo, Norway, pp. 291–300.

[90] Guimaraes, T., and Saraph, J. V., 1991, “The role of prototyping in executive decision

systems,” Information Management, Vol. 21, No. 5, pp. 257–267.

[91] Buchenau, M., Francisco, I. S., and Suri, J. F., 2000, “Experience Prototyping,” Proceedings

of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and

Techniques, August 17-19, New York, NY pp. 424–433.

[92] Schrage, M., 1993, “Culture(s) of Prototyping,” Design Management Journal, Vol. 4, NO.

1, pp. 55–65.

[93] Bogers, M., and Horst, W., 2014, “Collaborative prototyping: Cross-fertilization of

knowledge in prototype-driven problem solving,” Journal of Product Innovation and

Management, Vol. 31, No. 4, pp. 744–764.

[94] Kiriyama, T., and Yamamoto, T., 1998, “Strategic knowledge acquisition: a case study of

learning through prototyping,” Knowledge-Based Systems, Vol. 11, No. 8, pp. 399–404.

[95] Lancaster, A., 2003, “Paper Prototyping: The Fast and Easy Way to Design and Refine User

Interfaces,” IEEE Transactions on Professional Communication, Vol. 47, No. 4, pp. 335–

336.

169

[96] Skogstad, P., and Leifer, L., 2011, A Unified Innovation Process Model for Engineering

Designers and Managers. Berlin, Germany.

[97] Gatsou, C., Politis, A., and Zevgolis, D., 2014, “An exploration to user experience of a

mobile tablet application through prototyping,” International Journal of Computer Science

Applications, Vol. 11, No. 1, pp. 56–74.

[98] Sheth, J. N., Newman, B. I., and Gross, B. L., 1991, “Why we buy what we buy: A theory

of consumption values,” Journal of Business Research, Vol. 22, No. 2, pp. 159–170.

[99] Houde, S., and Hill, C., 1997, “What do prototypes prototype?,” Handbook of Human

Computer Interaction. Cupertino, CA.

[100] Nielsen, J., 1993, Usability Engineering. Mountain View, CA.

[101] Sonderegger, A., Uebelbacher, A., Pugliese, M., and Sauer, J., 2014, “The Influence of

Aesthetics in Usability Testing: The Case of Dual-domain Products,” Proceedings of the

SIGCHI Conference on Human Factors in Computing Systems, April 26-May 1, Toronto,

CA, pp. 21–30.

[102] Bailey, B., 2005, “Paper Prototypes Work as Well as Software Prototypes,”.

[103] Lin, J., and Seepersad, C., 2007, “Empathic lead users: the effects of extraordinary user

experiences on customer needs analysis and product redesign,” ASME 2007 Design

Engineering Technical Conferences-Design Theory and Methodology Conference,

September 4-7, Las Vegas, NV, ASME, Paper No. DETC2007-35302.

[104] Henderson, K., 1991, “Flexible Sketches and Inflexible Data Bases: Visual Communication,

Conscription Devices, and Boundary Objects in Design Engineering,” Science Technology

and Human Value, Vol. 16, No. 4, pp. 448–473.

[105] Vetterli, C., Hoffmann, F., Brenner, W., Eppler, M. J., and Uebernickel, F., 2012,

“Designing Innovation: Prototypes and Team Performance in Design Thinking,”

Proceedings from Action for Innovation from Experience Conference, June 17-20,

170

Barcelona, Spacin, pp. 1–11.

[106] Dow, S. P., Glassco, A., and Kass, J., 2009, “The effect of parallel prototyping on design

performance, learning, and self-efficacy,” Stanford Tech Report, Vol. 10, No. 1.

[107] Gerber, E., and Carroll, M., 2012, “The psychological experience of prototyping,” Design

Studies, Vol. 33, No. 1, pp. 64–84.

[108] Loughry, M. L., Ohland, M. W., and Woehr, D. J., 2014, “Assessing Teamwork Skills for

Assurance of Learning Using CATME Team Tools,” Journal of Marketing Educucation,

Vol. 36, No. 1, pp. 5–19.

[109] Hsieh, H.-F., and Shannon, S. E., 2005, “Three Approaches to Qualitative Content

Analysis,” Quality Health Resolutions, Vol. 15, No. 9, pp. 1277–1288.

[110] Zhang, Y., and Wildemuth, B. M., 2009, “Qualitative Analysis of Content,” Applications of

Social Research Methods to Questions in Information and Library Science, Vol. 4, No. 2

pp. 421.

[111] Kaulio, M. A., 1998, “Customer, consumer and user involvement in product development,”

Total Quality Management, Vol. 9, No. 1, pp. 141–149.

[112] Frisch, D., 1993, “Reasons for Framing Effects,” Organizational and Behavioral Human

Decision Processes, Vol. 54, No. 3, p. 399.

[113] Tversky, A., and Kahneman, D., 1981, “The framing of decisions and the psychology of

choice,” Environmental Impact Assessment, Technology Assessment, and Risk Analaysis,

Vol. 4, No.1, pp. 453–458.

[114] Silk, E. M., Daly, S. R., Jablokow, K., Yilmaz, S., and Rosenberg, M. N., 2014, “The Design

Problem Framework: Using Adaption-Innovation Theory to Con-struct Design Problem

Statements,” Proceedings from the 121st ASEE Annual Conference and Exposition, June 14-

17, Indianapolis, IN.

[115] Andreasen, M. M., and Hein, L., 1987, Integrated Product Development. London, England.

171

[116] Bucciarelli, L. L., 1994, Designing Engineers. Cambridge, MA.

[117] Ries, E., 2011, The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation

to Create Radically Successful Businesses. New York, NY.

[118] Kuo, T. C., Huang, S. H., and Zhang, H. C., 2001, “Design for manufacture and design for

‘X’: Concepts, applications, and perspectives,” Computers and Industrial Engineering, Vol.

41, No. 3, pp. 241–260.

[119] Boothroyd, G., 1994, “Product design for manufacture and assembly,” Computer Design,

Vol. 26, No. 7, pp. 505–520.

[120] Oviatt, S., 2006, “Human-Centered Design Meets Cognitive Load Theory : Designing

Interfaces that Help People Think,” Proceedings from the ACM international Conference

on Multimedia, Octeober 23-27, Santa Barbara, CA, pp. 871–880.

[121] IDEO, 2011, “Human-Centred Design Toolkit.,” Available online at:

http://www.ideo.com/work/human-centered-design-toolkit/ [Accessed 12.07.2013].

[122] Menold, J., Jablokow, K., Kisenwether, E., and Zappe, S., 2015, “Exploring the Impact of

Cognitive Preferences on Student Receptivity to Design Thinking,” Proceedings from the

121st ASEE Annual Conference and Exposition, June 15-19, Seattle, WA.

[123] Boothroyd, G., Dewhurst, P., and Knight, W. A., 2011, "Product Design for Manufacture

and Assembly", Computer Aided Design, Vol. 26, No. 7, pp. 5050-520.

[124] Kimbell, L., 2012, “Rethinking Design Thinking: Part II,” Design Culture, Vol. 4, No. 2,

pp. 129–148.

[125] Brown, T., 2008, “Design thinking,” Harvard Business Review, Vol. 86, No. 6, pp. 10-14.

[126] Johansson-Skoldberg, U., Woodilla, J., and Cetinkaya, M., 2013, “Design thinking: Past,

present and possible futures,” Creativity and Innovation Management, Vol. 22, No. 2, pp.

121–146.

[127] Reimann, M., and Schilke, O., 2010, “Product Differentiation by Aesthetic and Creative

172

Design: A Psychological and Neural Framework of Design Thinking,” Design Thinking:

Understand – Improve – Apply (H. Plattner, C. Meinel, L. Leifer). Springer Berlin

Heidelberg, Berlin, Germany.

[128] Kelley, T., and Kelley, D., 2013, Creative confidence: Unleashing the creative potential

within us all. New York, NY.

[129] Özbayrak, M., Akgün, M., and Türker, A. K., 2004, “Activity-based cost estimation in a

push/pull advanced manufacturing system,” International Journal of Product Economics,

Vol. 87, No. 1, pp. 49–65.

[130] Qian, L., and Benarieh, D., 2008, “Parametric cost estimation based on activity-based

costing: A case study for design and development of rotational parts,” International Journal

of Product Economics, Vol. 113, No. 1, pp. 805–818.

[131] Maguire, M., 2001, “Methods to support human-centred design,” International Journal of

Human and Computer Studies, Vol. 55, No. 4, pp. 587–634.

[132] Eck, D. Van, 2010, “Explaining and Relating Different Engineering Models of Functional

Decomposition,” Proceedings of Design Research Society Internatinoal Conference, July

1-4, Bangkok, Thailand, pp. 1–11.

[133] Karlsson, C., Nellore, R., and Soderquist, K., 1998, “Black Box Engineering: Redefining

the Role of Product Specifications,” Journal of Product Innovation and Management, Vol.

15, No. 9, pp. 534–549.

[134] Ariff, H., Salit, M. S., Ismail, N., and Nukman, Y., 2008, “Use of analytical hierarchy

process (ahp) for selecting the best design concept,” Journal of Technology, Vol. 49, No. 1,

pp. 1–18.

[135] Boy, G. A., 2013, Orchestrating human-centered design. New York, NY.

[136] Viswanathan, V. K., and Linsey, J. S., 2013, “Design Fixation and Its Mitigation: A Study

on the Role of Expertise,” Journal of Mechanical Design, Vol. 135, No. 5, pp. 51-65.

173

[137] Starkey, E., Toh, C. A., and Miller, S. R., 2016, “Abandoning creativity: The evolution of

creative ideas in engineering design course projects,” Design Studies, Vol. 47, No.1, pp.

47–72.

[138] Menold, J., Jablokow, K. W., and Simpson, T. W., 2017, “Prototype for X (PFX): A holistic

framework for structuring prototyping methods to support engineering design,” Design

Studies, in review.

[139] Shah, J., 2003, “Metrics for measuring ideation effectiveness,” Design Studies, Vol. 24, No.

2, pp. 111–134.

[140] Dean, D. L., Hender, J. M., Rodgers, T. L., and Santanen, E. L., 2006, “Identifying Quality,

Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation,” Journal of

Information Systems, Vol. 7, No. 10, pp. 646–698.

[141] Amabile, T. M., 1996, Creativity in context. New York, NY.

[142] Toh, C. A., and Miller, S. R., 2016, “Choosing Creativity: The Role of Individual Risk and

Ambiguity Aversion on Creative Concept Selection in Engineering Design,” Research in

Engineering Design, Vol. 27, No. 1, pp. 195–219.

[143] Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., and Markman, B.,

2011, “An Experimental Study of Group Idea Generation Techniques: Understanding the

Roles of Idea Representation and Viewing Methods,” Journal of Mechanical Design, Vol.

133, No. 3.

[144] Jansson, D. G., and Smith, S. M., 1991, “Design fixation,” Design Studies, Vol. 12, No. 1,

pp. 3–11.

[145] Viswanathan, V., and Linsey, J. S., 2011, “Design Fixation in Physical Modelling: An

Investigation on the Role of Sunk Cost,” Des. Eng. Tech. Conf.

[146] Toh, C. A., and Miller, S. R., 2015, “How engineering teams select design concepts: A view

through the lens of creativity,” Design Studies, Vol. 38, No. 1, pp. 111–138.

174

[147] Amabile, T. 1983, "The Social Phsychology of Creativity: A Componential

Conceptualization." Journal of Personality and Social Psychology. Vol. 45, No. 2, pp. 357-

367.

[148] Kaufman, J. C., and Baer, J., 2012, “Beyond New and Appropriate: Who Decides What Is

Creative?,” Creat. Res. J., 24(1), pp. 83–91.

[149] Diedrich, J., Benedek, M., Jauk, E., and Neubauer, A. C., 2015, “Are creative ideas novel

and useful?,” Psychology of Aesthetics, Creativity, and Arts, Vol. 9, No. 1, pp. 35–40.

[150] Silvia, P. J., 2011, “Subjective scoring of divergent thinking: Examining the reliability of

unusual uses, instances, and consequences tasks,” Thinking Skills and Creativity, Vol. 6,

No. 1, pp. 24–30.

[151] MacCrimmon, K. R., and Wagner, C., 1994, “Stimulating ideas through creative software,”

Management Science, Vol. 40, No. 11, pp. 1514–1532.

[152] Daly, S., and Christian, J. L., 2012, “Assessing design heuristics for idea generation in an

introductory engineering course,” International Journal of Engineering Education, Vol. 28,

No. 2, pp. 1–11.

[153] Kramer, J., Daly, S. R., Yilmaz, S., Seifert, C. M., and Gonzalez, R., 2015, “Investigating

the impacts of design heuristics on idea initiation and development,” Advances in

Engineering Education,Vol. 4, No. 4.

[154] Rietzschel, E. F., Nijstad, B. A., and Stroebe, W., 2006, “Productivity is not enough: A

comparison of interactive and nominal brainstorming groups on idea generation and

selection,” Journal of Experimental Social Psychology, Vol. 42, No. 2, pp. 244–251.

[155] Ford, C. M., and Gioia, D. A., 2000, “Factors influencing creativity in the domain of

managerial decision making,” Journal of Management, Vol. 26, No. 4, pp. 705–732.

[156] Pugh S, 1991, Total Design, Integrated Methods for Successful Product Engineering.

Wokingham, England.

175

[157] Vinodh, S., Shivraman, K. R., and Viswesh, S., 2012, “AHP-based lean concept selection

in a manufacturing organization,” J. Manuf. Technol. Manag., 23(1), pp. 124–136.

[158] Starkey, E., Gosnell, C., and Miller, S. R., 2015, “Implementing Creativity Evaluation Tools

Into the Concept Selection Process in Engineering Education,” ASME International Design

Engineering and Technical Conferences & Information in Engineering Conference.

[159] Saunders, M. N., Seepersad, C., and Hölttä-Otto, K., 2011, “The Characteristics of

Innovative, Mechanical Products,” J. Mech. Des., 133(February), p. 21009.

[160] Green, S. B., and Salkind, N. J., 2014, “Pearson product- moment correlation coefficeint,”

Using SPSS Windows and Macintosh Analysis and Understanding Data, pp. 233–238.

[161] Sweeney, J., and Soutar, G., 2001, “Consumer perceived value: the development of a

multiple item scale,” Journal of Retail, Vol. 77, No. 2, pp. 203–220.

[162] Boothroyd, G., 1994, “Product design for manufacture and assembly,” Computer Aided

Design, Vol. 26, No. 7, pp. 505–520.

[163] Mann, H. B., and Whitney, D. R., 1947, “On a Test of Whether one of Two Random

Variables is Stochastically Larger than the Other,” Annalytical Math Statistics, Vol. 18, No.

1, pp. 50–60.

[164] Hollander, M., and Wolfe, D. A., 1999, Nonparametric statistical methods. New York, NY.

[165] Daly, S. R., Yilmaz, S., Christian, J. L., Seifert, C. M., and Gonzalez, R., 2012, “Design

heuristics in engineering concept generation,” Journal of Engineering Education, Vol. 101,

No. 4, pp. 601–629.

[166] Yilmaz, S., Daly, S. R., Seifert, C. M., and Gonzalez, R., 2016, “Evidence-based design

heuristics for idea generation,” Design Studies, Vol. 46, No. 1, pp. 95–124.

[167] Toh, C. A., and Miller, S. R., 2016, “The Preferences for Creativity Scale (PCS): Identifying

the Underlying Constructs of Creative Concept Selection," ASME 2016 Design Engineering

Technical Conferences-Design Theory and Methodology Conference, August 24-26,

176

Charlotte, NC, ASME, Paper No. DETC2016-60414

[168] Bandura, A., 1994, “Self-efficacy: The Exercise of Control,” Personal Psychology, Vol. 50,

No. 3, pp. 801-815.

[169] Deci, E. L., and Ryan, R. M., 1985, Intrinsic motivation and self-determination in human

behavior.

[170] Simon, H. A., 1973, “The structure of ill structured problems,” Artificial Intelligence, Vol.

4, No. 3, pp. 181–201.

[171] Buchanan, R., 1992, “Wicked Problems in Design Thinking,” Design Issues, Vol. 8, No. 2,

pp. 5–21.

[172] Carberry, A. R., Lee, H.-S., and Ohland, M. W., 2010, “Measuring Engineering Design

Self‐Efficacy,” Journal of Engineering Education, Vol. 9, No. 9, pp. 71–79.

[173] Langer, E. J., and Moldoveanu, M., 2000, “The construct of mindfulness,” Journal of Social

Issues, Vol. 56, No. 1, pp. 1–9.

[174] Langer, E. J., 1995, “Mindfulness/mindlessness,” The Blackwell Encyclopedia of Social

Psychology, Vol. 5, No. 3, pp. 388–390.

[175] Messick, S., 1989, “Validity,” Educational Measurement, Vol. 2, pp. 13–105.

[176] Messick, S., 1995, “Validity of Psychological Assessment,” American Psychology, Vol. 50,

No. 9, pp. 741–749.

[177] Bandura, A., 2006, “Guide for constructing self-efficacy scales,” Self-efficacy Beliefs

Adolescence, New York, NY, pp. 307–337.

[178] Lim, Y., Stolterman, E., and Tenenberg, J., 2008, “The Anatomy of Prototypes: Prototypes

as Filters, Prototypes as Manifestations of Design Ideas,” ACM Transactions in Computer

Interactions, Vol. 15, No. 2, pp. 1–27.

[179] Beyer, H., 2010, “User-Centered Agile Methods,” Synthesis Lectures on Human-Centered

Informatics, Vol. 3, No. 1, pp. 1–71.

177

[180] Ungar, J., and White, J., 2008, “Agile user centered design,” Journal of Usability Studies,

Vol. 2, No. 3, pp. 112-132.

[181] Hubbard, A. W., 1950, “Phrasing Questions,” Journal of Marketing, Vol. 15, No.1, pp. 48–

56.

[182] Benson, J., and Hocevar, D., 1985, “The Impact of Item Phrasing on the Validity of Attitude

Scales,” Journal of Educational Measures, Vol. 22, No. 3, pp. 231–240.

[183] Reimann, M., Zaichkowsky, J., Neuhaus, C., Bender, T., and Weber, B., 2010, “Aesthetic

package design: A behavioral, neural, and psychological investigation,” Journal of

Consumer Psychology, Vol. 20, No. 4, pp. 431–441.

[184] Ha, A. Y., and Porteus, E. L., 1995, “Optimal Timing of Reviews in Concurrent Design for

Manufacturability,” Management Science, Vol. 41, No. 9, pp. 1431–1447.

[185] Tierney, P., and Farmer, S. M., 2002, “Creative self-efficacy: Its potential antecedents and

relationship to creative performance,” Academy of Management Journal, Vol. 45, No. 6,

pp. 1137–1148.

[186] MacDonald, J. H., 2009, “Kruskal-Wallis Test,” Handbook of Statistics, Vol. 1, pp. 165–

172.

[187] Strauss, A., and Corbin, J. M., 1998, “Open coding,” Basics Qualitative Research, Vol. 1,

pp. 101–121.

[188] Felder, R. M., and Brent, R., 2003, “Designing and Teaching Courses to Satisfy the ABET

Engineering Criteria,” Journal of Engineering Education, Vol. 92, No. 1, pp. 7–25.

[189] Katzenmeyer, C., and Lawrenz, F., 2006, “National Science Foundation Perspectives on the

Nature of STEM Program Evaluation.,” New Directions Evaluations, Vol. 109, pp. 7–18.

178

Appendix A Qualitative Coding Tags from Perceptions of Prototyping Study

Tag Tag Number 1 Size 2 Cost 3 Speed 4 Build 5 Comparable 6 Visualize 7 Ideas 8 Iterate 9 Material 10 Test 11 Early 12 Late 13 Learn 14 Understand 15 Product 16 Manufacture 17 Final 18 Function 19 Users 20 Fidelity 21 Communicate

179

Appendix B Build for X Handouts

180

181

182

Appendix C Test for X Handouts

183

184

185

Appendix D Description of Project Distributed to Students at the Start of the Semester

3D Printing Comes of Age:Educational Kits Using 3D Printing

Motivation:

Teachers are always looking for new and exciting ways to engage their students in the classroom. As low-cost 3D printers are becoming more widely available and easy to use, the opportunities to engage students in hands-on activities in the classroom are profound. Engineering students can 3D print design prototypes, anthropologists can 3D print specimens and fossils, visual artists can create new shapes and forms of art, and students in mathematics can 3D print complex functions to better understand them. The availability of 3D printing for in-classroom use can redefine not only what is taught but how students learn.

Objective:

Project Scope:

186

Replicator (Generation 5) at no charge. Each team will also have $50 to develop and create their educational kit for testing. Contacts to local schools, teachers, hands-on science camps, and other resources will be made available as needed. Your ME 340 instructor will guide you through the design and development process, and your team’s final presentation at the end of the semester will be a “pitch” to either (i) launch your own start-up or (ii) license your kit to an existing education technology firm. The team with the best “pitch” in each section will be eligible to receive $500 to advance their kit (e.g., prepare for a Kickstarter campaign), and the best team from each section will be invited to be judged at the College of Engineering Design Showcase. The winning team at the Design Showcase will be eligible for an additional $3000.

References:

1https://www.mne.psu.edu/3DPrinting/

2http://makercommons.psu.edu

187

Appendix E Prototyping AWareness Scale

Please answer each of the following questions based on your mindset or thought process as you prototyped. No response is “right” or “wrong.”Please consider the full range of responses for each of the items and avoid bunching your responses down either side or down the middle, since these patterns will make it difficult to interpret the results. Your answers are confidential and your participation is voluntary.

When developing my prototype I...

J M J E S S I C A M E N O L D Design Theory and Methodology

[email protected] Department of Mechanical and Nuclear Engineering 484-832-8677 The Pennsylvania State University 316 Biobehavioral Health Building, University Park, PA University Park, Pennsylvania, 16802

EDUCATION RESEARCH

Ph.D., Mechanical Engr. Research Assistant, Engineering Design and Optimization Group, The Pennsylvania State University August 2013-Present Anticipated Graduation: 05/2016

● Evaluated efficacy and effectiveness of structured versus unstructured Dissertation: Prototype for X (PFX): A Prototyping Framework prototyping methods to Support Product Design ● Developed the Prototype for X framework in engineering design to enhance prototyping processes and optimize design outcomes Advised By: ● Analyzed varying prototyping strategies within the Prototype for X Dr. Timothy Simpson, framework and exploring tradeoffs in prototyping decision making [email protected] ● Evaluated prototyping methods within Prototype for X framework on design outcomes specifically related to manufacturability, technical Dr. Kathryn Jablokow quality, user satisfaction, and user perceived value. [email protected] ● Developed metrics to assess the desirability, feasibility, and viability of B.S., Mechanical Engr. early stage prototypes in product development process The Pennsylvania State University ● Developed the Prototyping Awareness Scale to estimate student gains 2009-2013 in prototyping confidence and awareness after engaging in the Prototype for X framework. ● Developed the Engineering Innovativeness Scale to enhance STUDIES individual gains and understanding of innovativeness levels across 20 unique characteristics. STUDY TOPICS: ● Assessed individual traits of engineers throughout the innovation Design Methodologies| Design process. Thinking|Educational Psychology|Human Computer Research Assistant, Human Development Design for Impact Lab Interaction| Anthropometric Engineering Design, August 2015-Present Design|Design and Analysis of Psychometric Measures| ● Developed a digital platform for childcare utilizing RFID technology Electromechanical Design| ● Created human centered design tools such as empathic field guide, Elasticity Theory| Automatic rapid prototyping game, and how might we cards Control Systems |Simulation of ● Directed design thinking workshops to enhance the design process Mechanical Systems| and facilitated over 50 person workshop for final project wrap-up Optimization of Mechanical ● Designed user interface to test usability of location and Structures communication device for first responders during disaster events

● Design of user interface to enhance the intuitiveness of climate GPA: 3.98/4.0 control for industrial buildings