The Pennsylvania State University The Graduate School College of Education
FAILURE IS AN OPTION: REACTIONS TO FAILURE IN ELEMENTARY
ENGINEERING DESIGN PROJECTS
A Dissertation in Curriculum and Instruction
by Matthew M. Johnson
© 2016 Matthew M. Johnson
Submitted in Partial Fulfillment of the Requirements for the Degree of
Doctor of Philosophy
May 2016
ii
The dissertation of Matthew M. Johnson was reviewed and approved* by the following:
William S. Carlsen Professor of Science Education Director of Graduate and Undergraduate Education, C&I Dissertation Advisor Chair of Committee
Gregory J. Kelly Professor of Science Education Associate Dean for Research, Outreach, and Technology
Scott P. McDonald Associate Professor of Science Education Director, Krause Innovation Studio
Dan Sykes Senior Lecturer Director of Analytical Laboratory Instruction
Christine M. Cunningham Special Member Founder and Director, Engineering is Elementary Vice President, Museum of Science, Boston, MA
*Signatures are on file in the Graduate School
iii
ABSTRACT
Recent reform documents in science education have called for teachers to use epistemic practices of science and engineering researchers to teach disciplinary content (NRC, 2007; NRC,
2012; NGSS Lead States, 2013). Although this creates challenges for classroom teachers unfamiliar with engineering, it has created a need for high quality research about how students and teachers engage in engineering activities to improve curriculum development and teaching pedagogy. While framers of the Next Generation Science Standards (NRC, 2012; NGSS Lead
States 2013) focused on the similarities of the practices of science researchers and engineering designers, some have proposed that engineering has a unique set of epistemic practices, including improving from failure (Cunningham & Carlsen, 2014; Cunningham & Kelly, in review). While no one will deny failures occur in science, failure in engineering is thought of in fundamentally different ways. In the study presented here, video data from eight classes of elementary students engaged in one of two civil engineering units were analyzed using methods borrowed from psychology, anthropology, and sociolinguistics to investigate: 1) the nature of failure in elementary engineering design; 2) the ways in which teachers react to failure; and 3) how the collective actions of students and teachers support or constrain improvement in engineering design. I propose new ways of considering the types and causes of failure, and note three teacher reactions to failure: the manager, the cheerleader, and the strategic partner.
Because the goal of iteration in engineering is improvement, I also studied improvement.
Students only systematically improve when they have the opportunity, productive strategies, and fair comparisons between prototypes. I then investigate the use of student engineering journals to assess learning from the process of improvement after failure. After discussion, I consider implications from this work as well as future research to advance our understanding in this area. iv
TABLE OF CONTENTS List of Figures………………………………………………………………………..….vii List of Tables………………………………………………………...…………………viii Acknowledgements…………………………………………………………………….….x
Chapter 1 - Introduction ...... 1 Autobiographical Sketch ...... 3 Engineering Design in Elementary Schools ...... 4 Failure in Engineering Design ...... 7 Failure Types ...... 8 Goals of the Research ...... 11 Research Questions ...... 11 Chapter 2 - Literature Review ...... 13 Engineering Education ...... 13 Motivation ...... 14 Learning Science and Engineering ...... 15 Project-Based Learning ...... 15 Cognitive Load ...... 16 Iteration ...... 17 Failure ...... 18 Failure and Human Response ...... 19 Failure in Engineering ...... 19 Resilience ...... 21 Explanatory style ...... 24 Attribution Theory ...... 25 Feedback ...... 26 Attributional Feedback ...... 27 Praise ...... 28 Studies of Science-in-the-Making ...... 29 Theoretical Framework ...... 31 Chapter 3 - Design and Methodology...... 34 Choice of Research Tradition ...... 34 Educational Context ...... 37 Description of Engineering is Elementary ...... 38 Description of Engineering 4 Children ...... 39 Description of EiE’s To Get to the Other Side: Designing Bridges ...... 40 Description of E4C Civil Engineering (see Table 3.2) ...... 41 Classroom Context and Data Sources ...... 43 Data Analyses ...... 44 Preventing Researcher Bias/Increasing Validity ...... 46 Chapter 4 - Analysis of the Control Curriculum (E4C) ...... 48 Context of the Study ...... 48 Analysis of Classroom Video ...... 49 Failure Types ...... 51 Teacher Reactions to Failure ...... 54 v
Causes of Failure ...... 62 Obstacles to Improvement ...... 67 Summary of Analysis of Classroom Video ...... 73 Analysis of Engineering Journals ...... 74 Comparison of student journals with event maps ...... 75 Summary of Analysis of Engineering Journals ...... 79 Summary of the Analysis of the Control Curriculum ...... 79 Chapter 5 - Analysis of the Experimental Curriculum (EiE) ...... 83 Context of the Study ...... 83 Analysis of Classroom Video ...... 84 Failure Types ...... 86 Teacher Reactions to Failure ...... 88 Causes of Failure ...... 98 Obstacles to Improvement ...... 105 Patterns of interaction between constructs ...... 107 Systematic Improvement ...... 108 Summary: Analysis of Classroom Video ...... 115 Analysis of Student Journals ...... 115 Comparison of Video data to Journals ...... 117 Reliability of the Rubric ...... 120 Considering the Hawthorne Effect ...... 121 Summary: Analysis of Student Journals ...... 123 Chapter 6 - Comparison of the EiE and E4C ...... 124 Research Question 1a. How Do Designs Fail? ...... 125 Similarities ...... 125 Differences ...... 127 Summary of Research Question 1a ...... 128 Research Question 1b. Why Do Designs Fail? ...... 129 Similarities ...... 129 Differences ...... 131 Summary of Research Question 1b ...... 134 Research Question 2. How Do Teachers React to Failure During Engineering Design? ...... 134 Similarities ...... 135 Differences ...... 137 Summary of RQ 2 ...... 139 Research Question 3. How Do the Collective Actions of Students and Teachers Support/Constrain the Students’ Ability to Use Failure to Improve? ...... 139 Support ...... 140 Constraints ...... 144 Summary of RQ 3 ...... 148 Summary of Comparison of EiE and E4C ...... 148 Chapter 7 - Implications, Study Limitations, and Future Research ...... 152 Implications ...... 152 Failure as an Epistemic Practice ...... 152 Improvement ...... 157 vi
Studies of Engineering-in-the-Making ...... 160 Recommendations ...... 161 Study Limitations ...... 163 Future Directions ...... 165 Appendix A – Excerpt from an Event Map ...... 180 Appendix B – A Comparison of Event Maps and Student Journals ...... 181 Appendix C – Interaction of Failure Cause, Type, and Teacher Reaction ...... 187 Appendix D - Scoring Engineering Journals for Improvement ...... 188
vii
LIST OF FIGURES
Figure 1.1 - Three Axes of Failure ...... 9 Figure 2.1 - Resiliency Model (adapted from Richardson, 1990) ...... 22 Figure 4.1- Failure types by class ...... 52 Figure 4.2 - Failure Types by Class ...... 61 Figure 4.3- Failure causes by class ...... 66 Figure 4.4 - Model of Improvement...... 74 Figure 5.1 - Failure Types by Class ...... 87 Figure 5.2 – Descriptive statistics of teacher reaction types by class ...... 97 Figure 5.3– Mark, Lexi, and Rose attempt to build a truss bridge ...... 99 Figure 5.4 - A schematic of a double arch bridge ...... 100 Figure 5.5 - A double-arch bridge ...... 100 Figure 5.6– Lack of understanding of material properties ...... 101 Figure 5.7 – Poor craftsmanship ...... 101 Figure 5.8 - Limitation of materials ...... 103 Figure 5.9 - Causes of failure by class ...... 104 Figure 5.10– Systematic Improvement ...... 114 Figure 5.11– Group Journal Scores by Improvement Cycle ...... 117 Figure 5.12– Comparison of improvement scores of journals and video event maps ...... 119 Figure 6.1 - Failure types ...... 125
viii
LIST OF TABLES
Table 1.1 - Description of Failure Types ...... 8 Table 3.1 - Description of EiE: Bridges ...... 41 Table 3.2 - Description of E4C: Civil Engineering ...... 43 Table 4.1- E4C Classes ...... 49 Table 4.2– Coding Scheme for Failure Type ...... 51 Table 4.3 – Coding Scheme for Reaction Type ...... 54 Table 4.4 - The Manager Reaction Type ...... 55 Table 4.5– The Manager Reaction Type ...... 56 Table 4.6– The Cheerleader Reaction Type ...... 56 Table 4.7– The Cheerleader Reaction Type ...... 57 Table 4.8– The Strategic Partner Reaction Type ...... 59 Table 4.9– Coding Scheme for Causes of Failure ...... 62 Table 4.10 – Lack of Understanding of Science/Technology ...... 63 Table 4.11– Lack of Understanding of Science/Technology ...... 64 Table 4.12– Lack of Understanding of Materials ...... 65 Table 4.13 – Unfair Comparison ...... 68 Table 4.14– Denying the Opportunity for Low-Stakes Failure ...... 71 Table 4.15– Denying Opportunity for Low-Stakes Failure ...... 71 Table 4.16– Comparison of Student Journals with Event Maps (Ms. Lyle & Ms. Flemming) .... 77 Table 4.17– Comparison of student Journals with Event Maps (Mr. Tanner & Ms. Houseman) . 78 Table 5.1– EiE Classes (*Ms. Maddux and Ms. Clay taught in the same school) ...... 84 Table 5.2 – Coding scheme for failure type ...... 86 Table 5.3– Ms. Thomas reacts as a manager ...... 89 Table 5.4– Ms. Maddux reacts as a manager ...... 90 Table 5.5– Ms. Clay reacts as a cheerleader ...... 91 Table 5.6– Ms. Maddux reacts as a cheerleader ...... 92 Table 5.7– Ms. James reacts as a strategic partner ...... 93 Table 5.8– Ms. Clay reacts as a strategic partner ...... 94 Table 5.9– Ms. Thomas acts as a strategic partner pre-emptively ...... 96 Table 5.10– Lack of understanding of Science/Technology ...... 99 Table 5.11– Lack of knowledge of material properties ...... 101 Table 5.12– Poor craftsmanship ...... 102 Table 5.13– Poor Craftsmanship ...... 102 Table 5.14– Limitation of Materials ...... 103 Table 5.15 – Sequential analysis and changes aimed at improvement in T2 ...... 111 Table 5.16 Improvement in Ms. Thomas’ and Ms. Maddux’ Classes ...... 112 Table 5.17– Improvement in Ms. Clay’s and Ms. James’ Classes ...... 113 Table 5.18– Differences between on- and off-camera journal scores ...... 122 Table 6.1 – Comparison of Percentage of Failure Cause Codes ...... 131 Table 6.2– Summary of Class discussion about the String ...... 133 Table 6.3– Teacher reactions to failure ...... 136 Table 6.4– Total failures and teacher responses ...... 137 ix
Table 6.5– Denying Opportunity for Low-stakes Failure ...... 141 Table 6.6– Ms. Clay helps attribute a cause of failure ...... 142 Table 6.7– Unfair comparison ...... 146 Table 6.8– Observed supports and constraints of improvement...... 150
x
ACKNOWLEDGEMENTS My most sincere thanks to… Christy, Janey, and Annie Thank you for your patience and love as I spent so many nights and weekends away from you working on this dissertation.
Don, Cathy, Katie Thank you for your love and support through every failure and success in life.
Sonny and Shirley Thank you for including me in your family. I could not ask for better in-laws.
Bill Thank you for all of your guidance and wisdom through this long process. Thank you for being my friendly skeptic and making me think harder about what I was trying to do.
Greg Thank you for all you have taught me in coursework and in the classroom discourse group. Your help in thinking about methodology was invaluable.
Scott Thank you for your guidance beginning with my Master’s work and continuing on through this study. You always help me remember to connect the theoretical and empirical and not to leave fish on the table.
Dan Thank you for taking me into your lab during those three summers and your continued mentorship now. I truly appreciate all you have done for me over the past seven years.
Christine Thank you for your enthusiasm about engineering education and your willingness to allow me to use your data. I am grateful for the opportunity to work with you and your team.
My colleagues at CSATS, especially Annmarie, Leah, Darlene, Janell, Rosa, Kathy, and Amanda Thank you for all your hard work and encouragement
Penn State Classroom Discourse Group, especially Pete, Arzu, Amy, Carmen, Katie, Yann Thank you for your thoughtful feedback – best of luck in the future!
This material is based upon work supported by the National Science Foundation under Grant No. 1220305. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. 1
Chapter 1 - Introduction
Failures are the finger posts on the road to achievement. – C.S. Lewis
Several adages praise the positive virtues of failure, but very little is known about how students and teachers experience and react to it in the context of engineering design in the K-12 classroom. This study investigates the phenomenon of failure in engineering design projects in the elementary classroom, how students and teachers react to that failure and how collective actions in the classroom promote improvement (or not). This research aims to inform curriculum developers, professional development providers, and pre- and in-service teachers about failure and improvement in engineering projects. Understanding of the extent to which both the experience and support of failure and the opportunity for improvement should be considered in the development and implementation of science and engineering curricula and is timely research.
Chapter One is an introduction to the problem space of failure in engineering projects in formal education settings. Engineering in the K-12 science curriculum is a relatively new idea and although it presents challenges to teachers with little practical experience, it also provides opportunities for learning concepts and epistemic practices of engineering, as well as concepts and practices of science and mathematics. I first characterize types of failure, student and teacher responses to failure, the idea of resilience as growth in the wake of adversity, and how feedback is given, received, and used in future activity. I also describe the aspirations of this research, and will outline the research questions I attempt to answer with this study.
Chapter Two is a literature review to better situate my research within existing scholarship. Resilience is not a new topic of research in psychology, health and physical education, or in performance arts literature. The idea of resilience in schools is typically related to grades and not failure in material activities, but some of the concepts are interrelated. The 2 model developed in this study is compared with similar models of resilience. This chapter also establishes engineering design as a context for studying design failure in the classroom as well as responses to it. Integral to the investigation about the reactions to failure is a better understanding of teacher feedback, so the current views on responses are explored. Perhaps most importantly, I propose that improving after failure is an important epistemic practice of engineers that differs with science practice, but is potentially useful to teaching and learning in the science classroom.
Chapter Three outlines my research plan. As part of a larger education research grant, I have access to an enormous amount of classroom data, student artifacts, and student and teacher interviews. Because I will be choosing theoretical, not random, samples of instances that are recognizable as “failures,” I justify my choice of hermeneutic phenomenological case study as a research tradition and explain how this aligns with my theoretical framework. Since this failure is situated in group work in a classroom environment, I used interactional sociolinguistics
(Gumperz, 2001) as a tool to better understand how the participants experience failure, and will outline techniques to increase the trustworthiness of the research.
Chapter Four analyzes four classrooms engaged in a curricular unit originally designed as a comparison curriculum for a larger efficacy study. The unit is meant to represent a hands-on project representative of engineering units currently available and in use in elementary classrooms. Here I develop an initial model for understanding the complex nature of failure and improvement, for the purpose of applying this model to other classroom engineering projects.
Chapter Five studies four elementary classes using the curriculum, Engineering is
Elementary: Designing Bridges, and the analysis parallels the structure of Chapter Four with one notable exception: It goes further to understand systematic improvement that many student 3 groups were able to achieve, and also demonstrates a way to assess learning through the process of improvement, by analyzing student engineering journals.
Chapter Six compares the two analyses from Chapter Four and Five, using the research questions to frame the structure of the chapter. And Chapter Seven is presented in terms of the implications of this research, limitations of the study, and potential future research to further my understanding of this complex phenomenon.
Autobiographical Sketch
Cresswell (1998) suggests to include an autobiographical statement about experiences that lead researchers to their choice of topic. I have been competitive in all aspects of my life and many examples of failure remain in my memories. I have competed in baseball, pole vaulting, and golf. Each of these activities is rife with opportunities for failure. Hall of famer Ted
Williams said, “Baseball is the only field of endeavor where a man can succeed three times out of ten and be considered a good performer” (Palmer et al., 2006). In golf there is the opportunity for failure on every shot. In pole vault, the competition continues until the jumper fails to complete a height, and ties are decided based on the fewest number of failed attempts. I remember failed attempts much more than my successes, but they did not cause me to want to quit; they motivated me to want to improve. I’m sure much of this attitude comes from the community of family, friends, and teammates I’ve been with during my competitions.
In school, an early incident that affected me was a failed science (engineering) project in
6th grade. I designed a box to protect an egg from a two-story fall. The egg did not survive and we did not get a chance to redesign our devices. At the time, I was embarrassed and jealous of the groups that were successful. In hindsight, though, it seems like a missed opportunity to teach 4 about science concepts like inertia, and more specifically the way to systematically design and evaluate a design to make iterative improvements on it.
I am a former science teacher and a football, baseball, and track and field coach. I feel that teaching is always about more than the skills or content; teaching/coaching is also about helping student athletes through challenges, making the invisible practices of successful people visible by modeling positive ways to react to adverse situations--academic or athletic--and to helping students grow. Although it is possible that the primary predictors of a resilient reaction to failure are internal or ingrained prior to students arriving in the classroom, or are context dependent--a musician may respond differently to a mistake in a concert compared to a failed engineering design--I’m interested in the effect the teacher and/or curriculum designer can have in supporting improvement and learning after failure. As science/engineering educators, those are actions over which we have some control. I believe it is important to know more about not only the effects of supporting failure but the importance of the improvement (redesign) step in curricular development and implementation.
Engineering Design in Elementary Schools
As Rising Above the Gathering Storm (Augustine, 2005) pointed out, the United States has a shortage of qualified students entering the field of engineering, and racial minorities and females are drastically underrepresented. Perhaps more importantly, every citizen should be able to understand and become critical consumers of science and engineering as it permeates so many areas in our lives (NRC, 2012). The acronym STEM is often used to describe the integrated conceptualization of science, technology, engineering and mathematics, though typically the “E” is given much less priority in K-12 education. Recently, many in the community have recognized the need for these topics to be taught beginning in elementary school (Katehi, Pearson, & Felder, 5
2009). While calls like this eventually led to the inclusion of engineering in the current focus on science education in both the Framework for K-12 Science Education (NRC, 2012) and the Next
Generation Science Standards (NGSS) (Lead States, 2013), its inclusion brings with it a new set of problems for the science education community.
For many years, elementary teachers in self-contained classrooms have taught science sporadically or not at all (Weiss, 1978; Stefanich and Kelsey, 1989). A more recent study claims only 20 percent of K-3rd and 35 percent of 4-6th grade teachers in self-contained classes teach science all or most days (Banilower, et al., 2013). Although some of this can be blamed on poor self-efficacy (Bandura, 1977), it might also be attributed in part to the “antagonistic dilemma” described by Duschl (1983). This phenomenon is due to the dichotomy between the focus of introductory science courses on teaching “content,” while science teaching methods courses focus on process. Further complicating matters, the NGSS emphasizes students participating in the professional practices of scientists and engineers in the context of disciplinary core ideas and crosscutting concepts (NGSS Lead States, 2013). Most elementary school curriculum begin each year with the Traditional Scientific Method (TSM) (Windschitl, 2004) as being separate from the content of the rest of the material covered in class, so teaching content using epistemic practices is not only a paradigm shift, but it differs greatly with how most teachers experienced science learning in their own elementary experience.
One response to ineffective and infrequent science teaching in elementary schools is the adoption of curriculum kits such as Full Option Science System (FOSS). This curriculum was originally developed under a National Science Foundation (NSF) grant at the University of
California at Berkeley. Although originally the curriculum mandated 10 hours of professional development training for the teachers (Robardey, Allard, and Brown, 1994), many schools opt to 6 forego the training and rely solely on the boxed curriculum. Teachers are thus likely to use the materials and activities of the curriculum while still maintaining a view of the TSM, and failing to understand the importance of the necessary discourses that are productive in the science classroom (Duschl, Schweingruber, and Shouse, 2007). Without specific understanding of how to teach science, it is unlikely that a curriculum alone will significantly improve science instruction or the understanding of the disciplinary core ideas, crosscutting concepts, and/or the epistemic practices of researchers (NRC, 2012).
The introduction of engineering design will face many of these same challenges. First, teachers have little if any experience teaching (or learning) about engineering, so the frequency and the quality of teaching engineering will likely be low, at least initially. A probable response will be the development of many engineering curriculum kits marketed to school districts as a simple way for their teachers to “teach engineering.” Engineering is Elementary is a curriculum from the Museum of Science in Boston that takes a unique approach. Rather than teaching engineering isolated from science, the curriculum explicitly connects fields of engineering with science concepts typically taught in elementary science courses and includes extensive professional development, as well as resources for teachers to understand engineering better and developing appropriate pedagogical skills (Cunningham, 2009).
Further complicating the teaching of engineering is the portrayal of engineering in the
NGSS (NGSS Lead states, 2013). While the science disciplinary core ideas (DCI) are listed as nouns such as “natural selection,” the engineering DCI are actually practices like “developing a solution.” Cunningham and Carlsen (2014a) suggest that the difference between engineering and science is not the core ideas; rather, the difference lies in how ideas are used in productive, professional work. Without a better understanding of the needs of the teachers and students in 7 engineering education, what engineering instruction looks like in the classroom and how best to design the curriculum and professional development, engineering runs the risk of being misrepresented as a profession, operationalized in a linear, stepwise process done in isolation, and marginalized into discrete sets of declarative knowledge intended for standardized tests.
Failure in Engineering Design
One important practice of engineers is using failure as a part of the learning experience.
Petroski (1985, pg. 9) says, “…the history of structural engineering, indeed the history of engineering in general, may be told in its failures as well as in its triumphs.” While this view of failure is a normative part of practicing engineers, it does not have a clear analogue in science.
Certainly failure occurs in science practice, but there are few books exalting its virtue to scientists (Firestein, 2015). Rather, the mistakes often written about scientists come in the flavor of stories like Alexander Fleming and the discovery of penicillin. In such mythical portrayals, scientific discoveries occur quickly, individually, and because of mistakes, misleading students about how science derives its authority (Allchin, 2003).
Engineering is Elementary (EiE) curriculum was designed to include some “critical components” that the developers feel are important for students to learn about engineering. One such critical component is that students are given the opportunity to use failure constructively.
Currently, EiE is evaluating the efficacy of its curriculum in a large National Science Foundation grant. Part of their data collection in that study is digital video recording of teachers and students engaged in engineering design projects, using both the EiE curriculum and a control curriculum that lacks many of these hypothesized critical components. This context is well suited for the investigation of teachers and students reacting to engineering design failure, their responses, and the effect of a redesign (improvement) phase. 8
All engineering design failures are not the same. To specify the type of engineering design failure I am interested in, I will briefly describe four types of engineering failure that I propose to describe what may occur in classroom engineering projects.
Failure Types
Failures are not all alike. Because conditions associated with failures are likely to affect the types of reactions they elicit and the types of discourse events that occur in response, I developed a three-axis system for considering failure (Table 1.1 and Figure 1.1). This system theorizes three distinct continua on which failures in engineering design fall:
Table 1.1 - Description of Failure Types
Continuum Description High/Low stakes Failures that occur during the planning or initial construction phase is distinct from failure that occurs when the group is presenting its solution to the teacher or the entire class.
Intended/Unintended Failures that occur intentionally are done for a different purpose and have different outcomes compared with unintentional failures. Objective/Subjective Failure can be a matter of perception. Objective failure occurs when a solution does not achieve the desired criteria within the given constraints. Success or failure can be judged based on comparison to other groups’ projects. Although a solution may achieve its desired effect, individuals may see that solution as a failure compared to other groups’ solutions a subjective failure.
9
Figure 1.1 - Three Axes of Failure I think it is important to consider failures on these three axes. The reactions to failure, and the resulting interactions that occur are likely to be related to the type of failure. Additionally, this classification system serves as a theoretical basis for my sampling (see Chapter Three). Lottero-
Perdue and Parry (2014), who are also studying failure in EiE classrooms, describe failure types slightly differently: they consider failure to be either “within the Engineering Design Process” or a “failed final design.” My distinction, when it refers to K-12 engineering, is that students can present a “final” design that can fail, but learning from and improving can occur during an improvement cycle.
This study uses video recordings of students designing, constructing, evaluating, and improving various structures that are meant to be strong and stable. Student engineering journals 10 were also examined, to better understand the learning that occurs after failure. Video recordings of teachers during these designs also help characterize the phenomenon. These classroom activities have recognizable successes and failures. Analyses of these data use a phenomenological lens with a sociocultural understanding of learning (Lave & Wenger, 1991), utilizing interactional sociolinguistic methods (Gumperz, 2001), and content analysis (Bazerman,
2006).
In Chapter Two, I review literature on failure and reactions to it. Concepts like resilience, persistence, and “growth mindset” are all related to reaction to failure and the pursuit of success in future iterations. Exploring these concepts is related to why I am interested in exploring this phenomenon. Failure in a classroom engineering design project is a social process and many factors are likely involved in the ways that student groups (and individual students) react to it.
Many of these elements are likely to be internal characteristics related to any number of nature/nurture factors unrelated to the classroom. As science (engineering) educators, we cannot control such factors, so this study attempts to delve into aspects that can allow a failed design to lead to learning and can eventually result in success.
The aspects under a teacher’s control may allow learning from this type of failure to happen. First, the classroom culture must support failure as a normal aspect of learning. A classroom where improvement upon mistakes is a norm is likely a climate that promotes learning from failure. Related to the climate is the explicit support of the teacher, not only in the reaction but in the scaffolding of it. This requires additional time and limits the amount of instructional material that can “be covered.” Third, it is possible that the structure of a curriculum a teacher uses may be designed so this resilience occurs without the teacher. For example, when a second cycle of design (that includes thoughtful improvement) is built in, it is likely that students will 11 learn even void of explicit teacher support in the process. Without this affordance of an improvement step, student learning from failure is unlikely to lead to further learning. Thus, analysis of classroom culture, teacher and student interactions, and curricular affordances are all important aspects involved in learning from failure.
Goals of the Research
When students engage in open-ended engineering design projects, failure is almost inevitable. For a multitude of reasons, students and teachers may react in predictable (and maybe unpredictable) ways. Some of these reactions will result in negative or neutral experiences for students. However, positive reactions initiated by students and supported by teachers and curricula increase the potential for positive learning experiences in more students. Thus, the idea of learning from failure could also be potentially equity enhancing because students who do not
“get it right” the first time get another chance to improve and subsequently succeed. And by permitting (and encouraging) multiple designs, students come to understand the design is failing, not them. In order for this learning process to be supported by professional development providers and curriculum developers, more needs to be understood about what failure and the ensuing reactions to it look like in classroom settings. To do this, I characterize the types of reactions of students and teachers and how these reactions work together before, during, and after a failed design. Additionally, outcomes related to students engaging in a design improvement are important to understand for curriculum developers.
Research Questions
The overarching research question I am interested in is:
How is failure in K-12 engineering projects experienced and supported by teachers and students as a positive learning experience that promotes engagement and learning? 12
Because this overarching question is so broad, I have narrowed the research questions in this study to these:
1. What is the nature of engineering design failure in elementary school settings? a. How do designs fail? b. Why do designs fail? 2. How do teachers respond to failure during engineering design? 3. How do the collective actions of students and teachers support or constrain the students’ ability to use failure to improve? As I describe in Chapter Three, this study analyzes video recordings of eight classrooms engaged in civil engineering design challenges. As students attempt to build strong and stable structures in those classrooms, there were frequent failures that had different causes, and students and teacher reacted in a variety of ways. In some cases, improvement was supported, while in others it was not. Through a systematic analysis of over 135 hours of classroom video and student engineering journals, I attempt to make small steps toward a better understanding of failure and improvement in engineering design.
13
Chapter 2 - Literature Review
In Chapter One, I described the overarching research question as, “How is failure in K-12 engineering projects perceived as and supported by teachers and students to serve as a positive learning experience that enhances equity and promotes engagement and learning?” I then broke that problem into four questions more appropriate in scale for a dissertation study. The dearth of research of this problem might be attributed to the historic exclusion of engineering design from the national science standards and thus exclusion from the elementary classroom. The recent emphasis placed on engineering practices in A Framework for K-12 Science Education (NRC,
2012) and Next Generation Science Standards (NGSS Lead states, 2013) creates an urgency for research in the field that will inform teachers, curriculum developers, and professional development providers. Theoretical grounding of my research borrows from studies in psychology and anthropology as well as science education. This chapter reviews relevant studies about engineering education, work about failure and responses to it, and studies of feedback and its potential to encourage productive responses to failure. Studies of engineering discourse, like scientific discourse, will increase our understanding of how engineering knowledge is constructed in communities. In the third chapter, I synthesize this literature review and my theoretical framework into a methodological strategy that attempts to answer the research questions and to contribute to the corpus of research in science (and engineering) education.
Engineering Education
A Framework for K-12 Science Education (NRC, 2013) marked two significant changes from prior reform movements. Explicit descriptions of the “epistemic practices” (Kelly 2008,
2011) of scientists was likely done to alleviate the challenges similar to interpreting the term,
“inquiry,” (Loucks-Horsley & Olson, 2000) after implementation of the previous reform 14 document (NRC, 1996). The other major shift was the inclusion of disciplinary content and practices of engineering (NRC, 2012). Some have argued the framers of this document did not make a strong case for the inclusion of engineering practices, and actually misrepresented epistemic practices of engineering (Cunningham & Carlsen, 2014a, Cunningham & Carlsen,
2014b, Cunningham & Kelly, in review). Whatever the reason, the Framework does not present a compelling case to a teacher to include engineering within the science curriculum, so this section reviews some of the literature that supports the notion that engineering should be taught in K-12 schools because it increases content learning, equity, and other benefits such as motivation and other “21st century skills” (Partnership for 21st Century Skills, 2009).
Motivation
It has been found that lessons that allow students to make connections with and apply knowledge from their lives are particularly engaging and motivational to students, especially those from underrepresented minority populations (Doppelt et al., 2008). By providing students with a context, introducing specifications, and asking for design plans, teachers aid in students setting appropriate goals (Barron et al., 1998).
A major benefit of using project based learning (PBL) is in its ability to motivate students to take control of their own learning (Barron, et al., 1998). The Engineering is Elementary (EiE) units that will are studied in this work are meant to be engaging and fun while promoting learning goals in science and engineering. For this to happen, students must see the benefit of embedded activities and how they support learning goals (Harris & Rooks, 2010), and must be engaging enough not to bore students (Blumenfeld, et al., 1991). Because the structure of the four lessons in EiE units are meant to build towards knowledge and skills necessary for a final 15 design challenge, the information is necessary for students to learn and reflects the “need to know” nature of learning skills and concepts (Kolodner et al., 2003).
Learning Science and Engineering An engineering task should be more than simply engaging; it should support conceptual understanding throughout the process (Cunningham and Carlsen, 2014b). Problems in EiE units are meant to be relevant, authentic, and student-centered, in order to engage learners in realistic practices. This productive participation represents one of the four strands of scientific proficiency described by Duschl, Schweingruber & Shouse, (2007), and a successful curriculum depends on these social practices of the field and on using the concepts productively (Duschl,
2008). Students participating in EiE units are scaffolded in reflecting on their learning as well as productively using their new science knowledge, improving their chances to learn and retain information (Kolodner, et al., 2003; Zubrowski, 2002).
Project-Based Learning Project-based learning generally starts with a problem or overarching question that is intended to immerse students in a meaningful, engaging and relevant way in order to teach content and skills (Blumenfeld, 1991). The EiE units are designed with this framework in mind, rather than using hands-on activities without active learning, or “activity without understanding”
(Banilower, Smith, Weiss, and Pasley, 2006). The units are aimed at specific learning goals, scaffold support for both student and teacher learning, provide frequent opportunities for self- assessment, and for social organization of students to promote participation. All of these are aspects of effective PBL instruction (Barron et al, 1998); providing resources and an interesting problem are not sufficient (Barron & Darling-Hammond, 2010). Developers of PBL curricula must consider the learning goals, the disciplinary practices that students will engage in, how to 16 support skill development, and how to connect the practice with learning and assessment
(Krajcik et al., 2008).
A growing body of literature is beginning to demonstrate the effectiveness of PBL in teaching integrated STEM concepts and practices. Thomas (2000) reviewed outcomes of many
PBL initiatives and concluded that although teachers require significant support to engage in
PBL, this strategy is effective in improving self-efficacy and attitudes towards science, enhances communication and problem solving, and shows modest gains in subject matter knowledge.
Similarly, Kolodner, et al. (2003) found that middle school students who engaged in a PBL design challenge outperformed a control group in content understanding and collaboration.
Interestingly, underrepresented minority students who participated in a PBL curriculum made significant gains in achievement and attitudes (Kanter & Kanstantanopolous, 2010), and students from low SES schools benefitted more than high SES schools from a PBL unit (Blanchard, et al.,
2010).
Cognitive Load Cunningham and Carlsen (2014a) also suggest the effectiveness of engineering in elementary classrooms because of the decrease in cognitive load on students and the opportunity to externalize abstract concepts onto tangible physical materials. They discuss work by Levy
(2013), which showed significantly better conceptual understanding about the behavior of liquids when children were given the opportunity to design physical systems. Levy’s quantitative study compared test scores with a control group that participated in inquiry-based learning activities.
She concluded that the “concreteness” of the structure supports learning because results are both tangible and observable (Levy, 2013). Cunningham and Carlsen (2014a) also cite Roth (1996) and his assertion that “it is in building and playing with physical devices that the incompleteness 17 of representations is exposed” (pg. 160) to suggest that students are more able to deal with incomplete understanding of their mental models when these models are easily “experientially testable” (pg. 755).
Another example of design used to increase accessibility to younger learners was described in Penner, et al, (1997). First and second graders engaged in developing a model of their elbow as a way of better understanding structure/function. Students were shown to not only understand that motion was a key element, but constraints on motion were as well. In comparison with a control group, students were better able to evaluate other models, and demonstrated the readiness of young learners to participate in sophisticated model evaluation
(Penner, et al, 1997).
Iteration Embedded in the engineering design process is the practice of iteration. The goal of iteration is improvement, and thus success after failure is more likely when a project allows for iteration. Kolodner (2002) embeds iteration in her Learning by Design curriculum as cycles of investigation or “need to know” with cycles of design/redesign “need to do.” (p. 340). Her theory of learning suggests that we 1) learn best when we are in context of trying to achieve the goals of interest, 2) must interpret our experiences accurately so they become well-articulated memories,
3) apply cases from memory to allow for future learning, 4) can learn from the cases of others as well as our own, and 5) best learn from mistakes when we get immediate feedback (Kolodner,
2006). Thus, having the opportunity to re-design an engineering project provides the opportunity for students to apply cases from previous failure to improve. Similarly, Fortus, et al (2004) embeds iterative cycles in his “Design-Based Science and Student Learning,” but showed learning gains with pre- and post-tests and had no comparison group. 18
Another example demonstrating the benefits of including an iterative design process comes from Sadler, Coyle, and Schwartz (2000), who utilize engineering design competitions to show how iteration allows students to encounter misconceptions and overcome them by repeated testing and redesign. They quantify improvements in design from an initial prototype design
(provided by the teacher) to the design that is achieved through re-design with a measure they call “dynamic range” (Sadler, Coyle, and Schwartz, 2000). For example, one challenge aimed to decrease the difference in temperature between the inside and outside of a model solar house.
The prototype design had a difference of 50º C and the optimum design was 0.1º C for a dynamic range of 400x (Sadler, Coyle, and Schwartz, 2000).
Computer simulations offer an efficient means to iterate through the design cycle, and has been shown to progressively increase student interest in design activities and in the understanding of the scientific concept of center of mass (Svarovsky & Shaffer, 2006). This use of computer simulations to run many iterations increases the learning of students and saves resources. Chesler, et al (2013) demonstrated this in a virtual mentorship program in a course for first year engineering students as a way to increase learning and motivation simultaneously.
Failure The primary reason for iteration is improvement. Designs often fail, so the opportunity to re-design and learn from that failure is a normative aspect of engineering (Petroski, 1992).
However, Cajas (2001) claims that productive use of engineering is rare in the K-12 classroom, because students do not naturally address how or why the failure occurred. He also suggests that there are few studies that actually document which science or engineering concepts are learned through participation in designing projects like bridges, nor is there research on how students plan, test, and explain those failed designs (Cajas, 2001). Kolodner (2006) emphasizes the 19 importance of the feedback students receive from failure, and that two important features that enable students to learn from failure are access to expert cases that they can learn from and the opportunity to try again until they fully understand the concepts and create a successful design. A unique feature of failure in an engineering design context is that it provides feedback to the students independent of the teacher (Cunningham & Carlsen, 2014a). Scientific investigations typically require evaluative feedback from a more-knowledgeable other (Mariage, Englert &
Garmon, 2011). If the bridge collapses, the teacher’s feedback does not need to be evaluative, but can help guide the student through the experience of failure and process of improvement.
Engineering design in K-12 classrooms is a sparsely researched area, specifically in the understanding of failure and students’ and teachers’ response to it. However, it provides an interesting context to study classroom life and the potential for learning. The next section of this review will consider failure in a number of contexts as well as work about people’s responses to it. While none of these studies approach the phenomenon using the methods I employ, they are useful from a conceptual standpoint and potentially as means of analysis.
Failure and Human Response
Failure in Engineering Henry Petroski has written several books about failure in professional engineering. He uses examples of catastrophic engineering failures as case studies for others to learn from, similar to recommendations of Kolodner (2006). One commonly used example in forensic analysis of failure is the Tacoma Narrows Bridge. In 1940, this bridge was opened and became notorious immediately, given the name “Galloping Gertie” because it moved violently with the wind; in fact, thrill seekers would drive across it because it was like a roller coaster (Petroski,
1985). This bridge was filmed during its final moments before it tragically failed, but the film 20 has been used to examine the reasons for its failure. As a result of this catastrophe, bridges are now made out of stronger materials, and designs are routinely aerodynamically tested in wind tunnels to prevent this occurrence in the future (Petroski, 1985). Petroski (1985) also suggests that bridge failure would never happen again if we did not innovate—if all bridges built in the future were identical to successful existing structures—but this would also require a moratorium on innovation and that is clearly not desirable in a society that values both form and function.
Failures can be used as learning experiences for others so that future engineers will keep in mind aspects that another engineer did not. In Design Paradigms, Petroski (1994) makes the case for the use of failures and analyzing their causes as a way to educate engineers to prevent making similar errors. There is a key difference between failure by expert and novice engineers:
Experts typically fail due to overconfidence (Petroski, 1994), while novices fail because of inadequate conceptual understanding, and other reasons I explore in this study. Despite these differences between experts and novices, forensic case studies of classroom engineering failures can be utilized in students’ redesign phase (Kolodner, 2003). Redesign is an important way to learn from failure, but the responses to failure and adversity can and will vary greatly among classrooms.
Certain aspects of failure in engineering have no analogs in science. For example, the measurement of “mean time between failures” is a reliability metric for hardware or other technologies. It is calculated by taking the average time it takes for a certain number of systems to fail. Similarly, there other measurements that are used to describe failure, such as fishing line
(“eight pound test”) or pole vaulting poles (e.g., 14 feet long, rated for 185 pounds). Failure is also important to engineering in the use of engineering standards. These are essentially tools to communicate about products and/or processes. They provide some enforceable means of 21 acceptability when considering what is sellable to consumers, and are in place to protect the public from low-quality products and practices.
In professional engineering, failure is more than just an unexpected and undesirable result to a solution. Failure is a normative feature of the process of engineering, it is a way to rate the dependability of the device or process, and it provides an evaluation point that can be used in the enforcement of products and processes to ensure consumer safety. Similarly, failure in K-12 engineering design should not be viewed only as a negative outcome. Testing devices to failure during the school engineering design process can benefit the final design, and can be used as an evaluative metric for students to make informed, evidence-based decisions.
Resilience In order to progress from failure, we must react appropriately to it. Because little has been written about students failing in K-12 engineering classroom projects, one must find literature about the many types of responses to failures and other adversities. According to
Richardson, et al. (1990), resiliency is “the process of coping with disruptive, stressful, or challenging life events in a way that provides the individual with additional protective and coping skills than prior to the disruption that results from the event” (pg. 34). He uses this definition in the field of health education from a psychological standpoint of responding to life events that cause psychosocial, cognitive, or moral development challenges (Richardson, et al., 22
1990). However, his model of resiliency without facilitating (Figure 2.1) could be useful in
interpreting failure responses in engineering projects.
Figure 2.1 - Resiliency Model (adapted from Richardson, 1990) In this model, the stressor could be a failed design, and individual students will have a
number of innate, learned, or sociocultural protective factors to deal with that. However, the
students are arranged in a group that will work in concert, according to their group’s social and
intellectual norms. Disruption can come in the form of embarrassment, confusion, or frustration
due to lack of meeting intrinsic or extrinsic expectations. Among the four re-integration states
described, only resilient reintegration represents learning from failure. Although in psychology,
homeostatic reintegration signals a return to a state of “normalcy,” in learning it may represent a 23 return to the same state (thus no learning), and maladaptive and dysfunctional reintegration
(Richardson, et al., 1990) entail intellectual and/or physical disengagement with the project.
Interestingly, the terminology used to describe resiliency in this way is strikingly similar to the way conceptual change theorist talk about learning (Posner et al, 1982), although in learning the adversity is evidence that causes cognitive dissonance (Festiger, 1962).
Emmy Werner (1982) reported the following attributes are protective factors that contribute to resiliency: female, robust, socially responsible, adaptable, tolerant, achievement oriented, a good communicator, and having good self-esteem. Although there are a number of other studies that outline the individual factors that promote resiliency, and realizing that resiliency to an event like death in the family is completely unlike a failed bridge project in school, I choose not outline these other studies. The body of work on resiliency in schools with regard to achievement also seems unrelated to the types of resiliency I expect to see from classroom engineering projects. The causes for academic “failure” are different from classroom engineering design failure. For example, failure often occurs because a student does not turn in assignments and consequently receives failing grades. Although there is a need to better understand academic failure, my work is focused on failure during engineering design. However, it is useful to consider that although a classroom will likely contain a range of personalities and reactions to failure, feedback intervention by teachers and peers may have the power to promote resiliency and therefore promote learning after failure.
Other fields of study are also interested in promoting improvement after failure, such as sports psychology. Galli and Vealey (2008) theoretically sampled ten former college and/or professional athletes who experienced major setbacks or adversities. They conducted ethnographic interviews to better understand the athletes’ perceptions and personal experiences 24 of resilience and found sociocultural influences and personal resources to be important keys to resilience. They developed a model originally based on the model of resiliency by Richardson, et al. (1990), and suggested both an initial adverse reaction to the adversity followed by “positive outcomes” such as gaining a new perspective and the realization of a support network to be key in the resilient response (Galli & Vealy, 2008). Similarly, Mummery and others (2004) studied resilience in swimmers by theoretically sampling athletes with success after failure as “resilient,” and compared them with those that did not fit this description. Resilience seems to stem from positive self-reported perceptions of their abilities but were not significantly related to perceived social support (Mummery et al, 2004).
Explanatory style Another consideration in resilient reintegration (Richardson et al., 1990) is explanatory style (Abramson, Seligman, Teasdale, 1978). Originally coined in an article about learned helplessness (Seligman, 1975), explanatory style refers to the way people usually explain good or bad events as a means to better understand psychological processes that account for subsequent results. In a basketball skill competition, students with a more optimistic explanatory style were less anxious, more confident, and performed better than those displaying more pessimism in their causal explanatory style (Martin-Krumm, et al., 2003). Several methodologies have been attempted to measure a connection between explanatory style and resilient athletic performances. Seligman et al. (1990) conducted an experiment where swimmers’ attributional styles were measured with a questionnaire, then subjects were given false results about their times (simulating defeat). In that study, swimmers with pessimistic explanatory styles performed poorly on their next swim, while optimistic swimmers did not (Seligman et al., 1990). These 25 methodologies attempt to control variables and/or cause the subjects to talk about failure, but that adds additional complexity and cannot necessarily be considered typical or normal behavior.
Attribution Theory Explanatory style is closely related to attribution theory (Weiner, 1985). It theorizes the connection between motivation and emotion and the effect of causal attributions on them.
Perceived causes of both success and failure are coalesced into three categories: locus, stability, and controllability. Locus refers to whether the cause is an internal or external dimension, and stability is related to whether the cause is permanent or temporary, and personal controllability considers whether the cause is within the subject’s ability to regulate (Weiner, 1985). Further, these attributions are affected by outcomes (such as failure in an engineering task) and have an effect on future behaviors because they may influence choice, intensity, and persistence of those future behaviors (Le Foll, Rascle, & Higgins, 2007). According to studies on attribution, subjects that attribute failure to internal, controllable, and unstable (ICU) factors, such as a lack of effort, tend to experience more positive motivation from the failure than those attributing failure to external, uncontrollable, and stable (EUS) factors (Weiner, 1985, 1992). Many of these studies correlate positive reactions to failure to the increase in persistence in the subject (Dweck, 1986).
Persistence in this study can be considered the students’ willingness to give their best effort at success, even after an initial engineering failure. Lack of persistence after failure could be construed as an embodiment of learned helplessness (Seligman, 1985), but others have found an increase in performance and persistence after failures, despite ICU attribution patterns (for example, Follette and Jacobson, 1987). Further complicating this issue is the finding that attributions themselves are not stable. Allen (2010) showed that attributions of athletes changed temporally and posits this progression occurs because the reaction is first related to self-esteem 26 but fades over time, meaning attributions tend to become more internal, controllable, and specific.
Related to engineering education, a study of the causes of unequal retention of females in undergraduate engineering programs showed differences in attributions (Goodman et al, 2002).
A major finding of this evaluation was that many women leaving engineering did not leave due to failing grades; often, it was a misinterpretation of the grade or what it signified (Goodman et al, 2002). For example, the practice of grading on a curve posed anxiety for many women who left the field, because low class averages indicated that professors did not care whether they learned what they needed to know, and attributions were external and stable (Goodman et al.,
2002).
Evidently, the resilient reaction to failure is more complicated than interpersonal factors.
Because learning is a sociocultural process, it seems clear that there are many factors involved to cause students who fail an engineering task to be motivated and skilled enough to learn from the mistakes and to correct them in a second attempt, turning failure into success.
Feedback
As a science educator, I am less interested in the wide array of student attributes that are out of our sphere of influence, and more in the types of actions we can take to promote resilient behavior. One aspect of teacher practice that has been researched is the type, amount, and frequency of teacher feedback. One unique feature engineering design projects offer that many inquiry-based science projects do not easily offer interpretable feedback (Cunningham &
Carlsen, 2014b). When a student’s bridge design collapses in a pile of broken Popsicle sticks, no teacher evaluation is necessary; therefore, the type of feedback need not be of the “right/wrong” type, but more “how should you deal intellectually and emotionally with this result” type. Again 27 borrowing from literature of athletic performance, studies about feedback have been conducted to learn more about its effect on future performance.
Attributional Feedback One area of study is about feedback aimed at guiding athletes toward an internal, changeable, and specific style of attribution. An example of this type of research comes from Le
Foll, Rascle, and Higgins (2007), in which thirty novice golfers attempted to make eighteen-foot putts. Groups were given different styles of feedback that were either functional attribution
(internal, controllable, and unstable), dysfunctional attribution (external, uncontrollable, and stable), or non-attributional feedback. The functional attributions group produced improvements in causal attributions about failure, expectations of success, hopefulness, and in persistence after failure (Le Foll, Rascle, & Higgins, 2007). The researchers concluded that the effects of the feedback actually overrode the individual functional/dysfunctional attributional styles (Le Foll,
Rascle, & Higgins, 2007), suggesting a powerful effect of teacher feedback on student resilience in the classroom.
This type of attributional training has been demonstrated to be effective in other fields of study. Perry and Penner (1990) demonstrated its effect in a study investigating its effect on college undergraduates’ academic achievement. Chan (2006) used a reading intervention including reading strategies coupled with attributional retraining and demonstrated improved comprehension, use of reading strategies, and decreased learned helplessness (Weiner, 1985) compared with the group that did not receive the attribution intervention. And Dieser and Ruddel
(2002) showed positive outcomes in measures of personal control and stability attributions of depressed patients in a recreational therapy context. 28
Praise Another type of teacher feedback that has been studied is praise. Hattie and Timperley
(2007) claim that praise, punishment, and extrinsic rewards were the least effective for enhancing achievement. Carol Dweck is a leader in this area, and has shown that praise for students’ abilities has negative consequences for achievement, contrary to the notion that it is useful to maintain academic achievement motivation, behaviors, and strategies (Mueller &
Dweck, 1998). More specifically, the focus of the praise is important to consider. Praise about a student’s ability is likely to promote performance as the primary motivation for participation
(Mueller & Dweck, 1998). In other words, being challenged or learning a lot become less important that appearing smart (Mueller and Dweck, 1998). Another drawback of praising intelligence is that is develops a stability in attribution for failure, but praising effort had the opposite effect – students praised for effort showed greater persistence and success in answering math problems. (Mueller and Dweck, 1998). This suggests an effect of the teachers’ feedback on persistence and resilience.
Based on work like this, Hattie and Timperley (2007) propose a model of feedback. In it, they suggest the purpose for teacher feedback is to “reduce discrepancies between current understandings/performance and a desired goal” (p. 87). The model includes questions that address the types of feedback: 1) Where am I going, 2) How am I going, and, 3) Where to next, and each of the questions work at four levels: 1) task, 2) process, 3) self-regulation, and 4) self
(Hattie & Timperley, 2007). Thus, when analyzing teacher and student interactions surrounding a failure event, it will be important to consider not only what the teacher says and does, but considering the likely motivation for that feedback. A teacher may praise a student who failed in 29 a design in order to try to protect the student’s self-esteem, but could be inadvertently causing her/him to internalize the failure as being caused by external, stable, and uncontrollable factors.
Studies of Science-in-the-Making
So far, I have argued that 1) Engineering design projects in elementary classrooms may be interesting and worthwhile to study; 2) Failure is a normative experience in professional engineering and can likely be a beneficial experience in elementary engineering, despite some key differences; 3) Resiliency and theoretical models of it in psychology may be useful in analyzing reactions to failed engineering projects, even using a sociocultural perspective of learning; and 4) The feedback the teacher, other students, and the project itself provides is a potentially important aspect of promoting resilient reactions to failure. However, the methodologies for most of the studies on resilient responses to adversity typically use self- reporting, surveys, and forced reactions to failure that adds a further layer of complexity and might generate claims that do not accurately reflect true behaviors. Important understandings about how students and teachers collectively experience and react to failure might occur more from naturalistic phenomenology/ethnography rather than from designed experiments and/or forced reactions to attribute causality for failure. To accomplish this, I chose to study groups of students participating in engineering design and analyze students’ interactions with each other, with their teacher, and with the material activities via discourse (verbal and non-verbal). Thus, the next section of my review is of studies that describe classrooms using a similar methodology as I outline in Chapter Three.
Kelly et al. (2000) is an example of an ethnography that uses videotaped classroom activities, artifacts, and interviews and views learning from a sociocultural perspective. As the research study progressed, the researchers’ modified their questions to make them more specific 30 due to analysis of event maps and patterns of interaction (Kelly, 2014). Another study described in Kelly (2014) as an exemplar of sociolinguistic analysis of classroom activities comes from
Sezen, et al (2009). This study utilized video recorded interactions of students, voice-over reflections, and student artifacts to consider the aspects of teaching that novice teachers attend to, analyzed by a lens influenced by activity theory (Kelly, 2014).
Using a similar sociocultural perspective, Gyllenpalm and Wickman (2011) consider how the meaning a word or concept takes on depends on its use and consequences. They conducted focus groups aimed to situate conversations similar to the types pre-service teachers have with their professors. Analyses of these focus groups led to interesting findings in the conflation of the terms “experiment” and “laboratory task” and emphasized the situated nature of talk in that science education context (Gyllenpalm and Wickman, 2011). Wickman was also involved in a study that looked closely at how students with misconceptions about electrochemical cells talk and act when they are taking part in an activity (Hamza & Wickman, 2008). Interestingly, despite the repeated interview studies and surveys that demonstrate naïve conceptions and how they act as a barrier to learning (for example, see Driver et al, 1994), the authors showed that these misconceptions were not barriers to learning and in some cases were actually generative in students’ reasoning (Hamza & Wickman, 2008).
These studies represent work that view situated sociocultural nature of classroom work to be so important that analysis in more controlled or contrived ways leads to claims that do not represent reality, and in some cases demonstrate a reality that is much different from commonly accepted results (Hamza & Wickman, 2008).
As I will outline in my theoretical framework, I consider the experience of (and reaction to) failure as a social phenomenon that is inseparable from the situated nature of the classroom 31 environment. Participation in groups (small collaborative group, classroom, sports team, etc.) has embedded certain norms for interactions, and participation can be studied using sociolinguistic analyses of discourse practices in situ. For these reasons, my methodology will use methods of phenomenology and ethnography and analysis based on interactional sociolinguistics (Gumperz,
1982).
Theoretical Framework
I have a sociocultural perspective of the classroom environment and school activities.
Lave and Wenger (1996) describe this as being an “integral and inseparable aspect of social practice” (p. 31). Learning is not an individual process, and any attempt to isolate science/engineering learning from the classroom and “into the head” of a student or teacher is potentially oversimplifying a very complex system. The investigation of the experience of learning from failure in classroom engineering design should be undertaken within the context and culture in which it occurs naturally (Brown, Collins, & Duguid, 1989). Humans typically belong to many different social groups, such as classmate, teacher, or bowling team member.
Participation in those groups involves particular ways of talking, thinking, acting, and interacting
(Kelly, 2014). These cultural practices are typified by the interactions among the members, and one way of investigating these interactions is using interactional sociolinguistics (Gumperz,
2001).
Students engaged in engineering activities only fail when the experience is co- constructed as a failure within the context of that group in that classroom. An important distinction I wish to make with this study is the wide range of failure experiences and reactions to it; however, I can think of no design failure that cannot be used as a learning experience if approached in thoughtful ways by teachers and students. For example, at the most extreme end 32 of failure, the Challenger space shuttle mission was forensically investigated to find technical and procedural failures (Rodgers Commission, 1986). At the other end of the continuum would be a student realizing during initial prototyping that the uneven legs of a structure will make her structure unstable. While one was a national tragedy and the other will go unnoticed by almost everyone, learning from each should occur.
Since little research exists about failure in elementary engineering contexts, a useful beginning point is the consideration of other analogous failure types. In study of failure in team sports, the ways in which athletes explain the causes of failure are predictors of resilient responses (Martin & Krumm, 2003). The ways in which student groups talk about failure and proceed to improve from it may be insightful, as will the overall group dynamics because sociocultural influences have been shown to be important in athletes’ abilities to bounce back from failure (Galli & Vealey, 2008). Attributions are also important to consider when listening to students and teachers respond to failure. Cues can be make suggesting personal, interpersonal
(other persons’ behavior), and group attributions (Allen, 2010), and can be analyzed along the axes of internal/external, stable/variable, controllable/uncontrollable, and specific/general causes.
Gender differences have been found in attributions of failure related to engineering (Goodman, et al., 2002) and should be considered while studying students doing engineering.
Teacher feedback is an important aspect to consider because it constitutes part of the sociocultural context for learning. Teachers may take an authoritative standpoint, or may take student ideas into consideration (dialogic) and may vary the extent to which others’ participation is welcome (Mortimer & Scott, 2003). These perspectives may be useful in analysis of teacher feedback after designs fail. Teacher talk has also been shown as a way to manage classroom activity so it stays within the teacher’s control (Carlsen, 1991), and has been shown to affect 33 students’ self-talk and academic self-concept (Burnett, 1999). Due to the nature of the engineering activities, the interactions between teachers and students will likely be different from the I-R-E triadic structure (Lemke, 1990) because evaluation is easily discernable and the teacher will not have to provide feedback of “right or wrong.”
This chapter has described relevant literature about engineering education, about failure and responses to it, and about the role and types of teacher feedback. Through this review, I tried to establish a clear description about: 1) how engineering is different from traditional science education and how these differences provide opportunities for learning a unique set of skills and knowledge; 2) how failure, although a professional norm in engineering, is not typically considered as a norm in K-12 schools; 3) how positive reactions to adversities (resilience) has been studied in several fields, though infrequently in engineering education, and by utilizing significantly different methodologies; and 4) that the role of teacher feedback is a potentially important aspect of promoting resilient reactions. Then, I outlined the theoretical framework with which I enter this dissertation project. Alignment of this framework with the methodology for investigating failure is essential, so Chapter Three will outline my methodological choices made to attempt to answer my four research questions.
34
Chapter 3 - Design and Methodology
The purpose of this study is to carefully examine and describe the experiences of failure that occur in elementary engineering projects, the ways students and teachers react to them, and the ways in which the curriculum, teachers, and students promote or constrain the ability of the failure to lead to learning. Previously, I have described the opportunity to learn more about the experience of and reactions to failure in the classroom, have reviewed literature relevant to this problem, and have clarified my theoretical framework. This chapter describes the context for the study and the data sources that will be used. I also describe the methodological choices and the rationale for using these methods of analyses to answer the research questions, and how these choices align with my theoretical framework.
Choice of Research Tradition
Creswell (1998) outlines four reasons for choosing a qualitative research methodology: 1) the research questions begin with “how” or “what” instead of “why;” 2) the need for the topic to be explored – that is, variables are not easily identified, and theories need to be developed; 3) the need for a detailed account because either the cursory description is not sufficient and/or the detailed account does not exist; and 4) the need to study individuals in natural settings. The study
I will describe attempts to fits all four criteria. After establishing the appropriateness of a qualitative study for this work, I justify the tradition of phenomenology by exploring the history of and philosophical underpinnings of the methodology in order to align it to my own ontological and epistemological understandings.
Failure is a phenomenon that most humans experience. Phenomenology attempts to understand human experience by exposing aspects that are taken for granted (Starks & Trinidad,
2007) and illuminating details that may be considered trivial (Laverty, 2003). van Manen (1990) 35 opines that a study using this method should elicit the “phenomenological nod” from the readers as an outward recognition of experiences they have had. However, phenomenology is both a philosophy and a methodology that needs to be further explicated to justify why this aligns with my theoretical framework, because there are many views that have formed about phenomenology and many versions of it that vary greatly in both the philosophy behind it and the method of doing research using it.
The origins of phenomenology are typically credited to Husserl. His goals were epistemological, because his understanding held experience as the primary source of knowledge building (Racher & Robinson, 2003). Husserl thought a good phenomenological study should be rigorous and should avoid any researcher bias in order to study an experience as it appears and to distill it to essences (Valle et al, 1989). This attempt to avoid bias in order to allow the essences to emerge is termed phenomenological reduction (Racher & Robinson, 2003). Further, Husserl
(1970) argues that “lifeworld” is what individuals experience without interpretation and free from cultural context.
Heidegger agreed with Husserl in many respects, except in the assertion that description should be valued over understanding (Racher & Robinson, 2003). His version of hermeneutic phenomenology is focused on understanding subjective experience of individuals and groups through their stories (Kafle, 2013). He considers the researcher to be a signpost that should point toward the essential understanding of the approach as well as the essential understandings of the particular phenomenon of interest (Kafle, 2013). Another departure from Husserl is in
Heidegger’s view that understanding cannot be mutually exclusive from one’s previous experiences and culture (Laverty, 2003). Further, he suggested the interpretive process of analyzing text is a cycle that zooms in and out (Kelly, 2014b) from parts of the experience to the 36 whole experience, called the hermeneutic circle (Polkinghorne, 1983). In this process, the meaning of social action must take into account the linguistic community is occurs in, while linguistic action must be considered within the meaningful behavior of the actors in the social situation (Darity, 2008).
Gadamer (1989) further altered classical views on Husserlerian and Heideggerian phenomenology by positing that prejudgment (preconceptions) is a part of our linguistic experience that makes our understanding possible (Dowling, 2006). Thus, understanding occurs by interpretation through personal involvement of the researcher in the reciprocal process of interpretation that occurs because the researcher has common experiences enabling that understanding (Gadamer, 1989). He also suggests the researcher should engage in the hermeneutic circle, and that engagement should include further discussion with the study participants in a dialogic method where the interpreter and the phenomenon are studied together
(Dowling, 2006).
These versions of phenomenology have arisen out of differences in theoretical approaches to epistemology and ontology. However, these differences have caused tension in fields whose researchers claim to use phenomenology as a methodology, such as nursing.
Authors like Crotty (1996) accuses researchers of misinterpreting Husserl and Heidegger, and
Paley (1998) claims that misinterpretation of these philosophers’ ontology creates an unacceptable Cartesian duality of reality and lived experience. In response, Caelli (2000) set out to elucidate further the splits and philosophical issues to justify current research criticized by
Crotty (1996) and Paley (1998) and cited difference that split the field into American and
European phenomenology (Silverman, 1987). Caelli (2000) cites the influence of American philosophers on the American phenomenological movement and describes the motive of the 37
American phenomenologists to be the understanding of the reality of an individual’s experience as he engages with the phenomenon rather than an objective reality of the nature of the phenomenon itself (Caelli, 2000). She also suggests that these differences are rooted in the types of phenomena of interest; Husserl and his colleagues were interested in more abstract phenomena such as consciousness, while Caelli (2000) and her colleagues are interested in lived experiences of health care workers and their patients.
The version of phenomenology that aligns with my theoretical framework and analysis is an American/Gadmerian hermeneutic phenomenology that views the phenomenon of failure and the responses to failure as culturally embedded. This lived experience should not be described decontextualized from the sociocultural context in which it is experienced because the experience of the phenomenon and reactions to failure are co-constructed. The goal of interpretive phenomenology of this type is to look for commonalities in a culturally grounded meaning (Benner, 2000), and phenomenological reduction or suspension of researcher bias is impossible (Gadamer, 1989). Thus admission of bias is an important step, but places a great responsibility on the researcher to establish trustworthiness of the results (Fleming, Gaidys &
Robb, 2003), which I will address later in this chapter.
Educational Context
Data that will be analyzed comes from a larger efficacy study of an engineering curriculum developed by the Museum of Science, Boston, called Engineering is Elementary
(EiE). This NSF-funded study is primarily concerned with outcomes in comparison with a control curriculum developed without many of the aspects thought to be critical in to the success of EiE. Some examples of these critical components are: 38
The engineering problem is placed in a narrative context meant to be relevant and interesting to children Scaffolded instruction in the engineering design process The problems have defined criteria and constraints, but multiple solutions The use of math and science in designing solutions The opportunity to collect data, evaluate designs, use failure constructively, and reflect on what was learned to improve designs The lessons are heavily scaffolded for teachers with little background content and/or pedagogical practices.
Description of Engineering is Elementary There are twenty EiE units in all, and they have certain aspects in common. They all include science content commonly taught in elementary classrooms, and they also are related to a specific field of engineering meant to introduce students to a focused view rather than a generic characterization of engineering as a whole. The first lesson is always a story about an elementary-aged boy or girl from a unique geographic or cultural area that encounters a problem that needs to be solved with engineering. The protagonist usually has a friend or family member that is an engineer and helps introduce the design process by including the child in the problem solving. The problem in the story is directly related to the design challenge the students will participate in later in the unit. The second lesson is meant to provide a broader view on the field of engineering through hands-on activities to help students understand the type of work engineers do and the technologies they produce. The third lesson provides students with the opportunity to explore material properties and to collect evaluative data that will inform the design challenge in lesson four. This lesson is also an opportunity to learn science content relevant to the challenge. The fourth lesson is an engineering design challenge that has specific criteria and constraints, and encourages students to use the engineering design process to design, construct, test, and improve a solution. 39
Description of Engineering 4 Children The Engineering 4 Children (E4C) curriculum was designed by EiE staff as a control curriculum to compare in randomized, controlled experiments on the efficacy of EiE for an NSF-
DRK12 grant. The E4C units are paired with a corresponding EiE unit for comparison, so each pair is about the same field of engineering and each should take the same amount of classroom time to complete. However, the E4C embeds the hypothesized critical components to a much lesser extent if at all. The E4C units are not intentionally poor to set up a “straw man” style of comparison; rather, they represent typical engineering curriculum that is available and is taught in schools around the world. In health studies, this would be akin to testing an intervention compared to the current standard of care.
The study reported in this dissertation uses purposeful sampling (Patton, 1990) to investigate the phenomenon of failure. As described in Chapter One, failure can take on many forms. This study is meant to better understand failure by looking closely at instances that are easily recognizable as failing. In contrast, many engineering designs cannot be viewed as binary
“pass/fail” solutions. For example, EiE has a unit on environmental engineering in which students develop a process to clean up an oil spill. They are evaluated on an “environmental impact” based on a rubric developed by EiE. Failure in this sense is on a continuum and success must be determined relative to a scale; however, I chose to investigate the EiE Designing Bridges and E4C Civil Engineering units because: 1) these units were designed to be the same duration;
2) they both deal with the same field of engineering and thus the same science content, and 3) failure is easily recognizable. 40
Description of EiE’s To Get to the Other Side: Designing Bridges (Table 3.1)
The first lesson of this unit begins with a story about Javier, a Latino boy in Texas. A bridge that he must cross to get to his fort collapses under Javi and his sister, and his mother and stepfather will not let Javi return because the bridge is unsafe. The stepfather is an engineer, though, and helps Javi through the engineering design process of building a safer bridge. The story also highlights civil engineers and the work they do and introduces several common bridge types.
The second lesson engages students in activities meant to teach students that engineering of this kind requires balanced forces, and that forces can only be balanced in structures that look straight, level, even, or symmetrical. Through this activity, science concepts like force and equilibrium are investigated by pushing and pulling on different areas of a one-story structure and a tower made of paper. They test both strength and stability of the structures and consider civil engineering solutions to the problems that arise.
The third lesson is meant to provide experience with materials and evaluation of various bridge types. Through controlled experiments of different bridge types, students analyze results of tests to make informed decisions in the design challenge. The curriculum aims in this lesson to have students recognize that beam, arch, and deep beam bridges have their own strengths and the criteria and constraints will determine the appropriate design because there is no “best” design—
– the bridge type must be selected in response to the criteria and constraints. This series of activities will also provide students experience with the materials that will inform their decisions, experience that not all students have prior to this lesson. 41
The final lesson is a design challenge in which students are asked to design a bridge to span a fifteen inch space with six inch abutments. The final structure must be able to support toy cars rolling across it. Their bridge is constrained by the requirement that a “barge” (a small block of wood) must be able to fit under the bridge without hitting the structure, and they are also constrained by the materials available to them.
Table 3.1 - Description of EiE: Bridges
Lesson To Get to the Other Side: Designing Bridges (Civil Engineering)
Students read the storybook Javier Builds a Bridge. After falling off an unstable bridge leading to his fort, Javi helps design a new bridge with the 1 – Javier help of his stepfather, Joe. Civil engineers and their work are highlighted in Builds a Bridge this story, and the characters use the engineering design process throughout the process. Several types of bridges are also introduced. Students examine several different structures and consider how they are affected by force. They also evaluate some solutions aimed to prevent forces 2 – Pushes and from causing structures to fall. A discussion of civil engineers and their work Pulls to counteract unbalanced forces and increase stability and strength concludes this lesson. Students create and test three bridge types (beam, arch, deep beam) and 3 – Bridging evaluate the amount of weight each can support. Students also examine the Understanding materials available to them to consider ways in which they can be used to increase the strength and stability of a bridge. Students apply the Engineering Design Process as they work in small groups 4 – Designing a to design, build, and test a bridge that is able to cross a 15 inch span, will Bridge support toy cars crossing, and is able to allow a “barge” to roll under it without touching the bridge or the abutments. Students are then given the opportunity to improve their design.
Description of E4C Civil Engineering (see Table 3.2) The first lesson of E4C Civil Engineering has students learn about the measurement of critical load of a structure. Students begin by reading about the work of civil engineers, then work individually to design structures with playing cards and tape, and then calculate the critical load. Next, students are grouped and collaboratively design a structure using the same materials. 42
Critical loads are measured with weights, recorded, graphed, and the teacher has a group discussion about the reasons why the strongest structures performed well. Students complete this activity by reflecting on questions posed in their engineering journals.
The second lesson begins with a reading assignment about civil engineers. Then student groups design a structure out of raw spaghetti and marshmallows to hold as much weight as possible. Students test their structures with washers, report their results to their class, and reflect on the experience in their journals.
The third lesson is called the “Tall Tower Challenge” and implores students to design the tallest structure strong enough to support a golf ball and stable enough to withstand winds from a fan. The first class period begins with a reading about why tall buildings are strong and stable.
Groups then design a structure that will be constructed with drinking straws, pipe cleaners, and paper clips. The second period has students build and test their structures in front of the class.
The lesson concludes with reflections in their journal.
The goal of the final lesson is to design a bridge that can hold a five pound weight.
Students read passages about bridge types and how civil engineers design them, then design their structures using popsicle sticks and tape. This activity takes two days, so the testing occurs on day two as a whole group so the class can discuss their observations. 43
Table 3.2 - Description of E4C: Civil Engineering
Lesson E4C: Civil Engineering
Students: Read about the work of civil engineers 1 – Critical Load Design, build, test structures (individually and in teams) Measure critical load of their structures Students: 2 – Leaning Tower Work to build the tallest, strongest structure using spaghetti of Pasta and marshmallows Test their structures and discuss observations
Students: 3 – Tall Tower Read about famous strong, stable, and tall structures Challenge Design their own strong, stable towers out of pipe cleaners and drinking straws
Students: 4 – Popsicle Bridge Read about different bridge types and parts of bridges Design and build a bridge out of popsicle sticks and tape to
support a 5 pound weight
Classroom Context and Data Sources
Data were collected from eight classrooms involved in the efficacy study. Four of the classes used the E4C Civil Engineering unit, and four classes used the EiE, To Get to the Other
Side: Designing Bridges unit. Participating schools were located in both Northeast and Mid-
Atlantic States and contained ethnic, racial, and socioeconomic diversity. As a part of the study, all teachers were provided 30 hours of professional development (PD) to promote a common understanding of their respective curriculum by engaging teachers in a shared definition of engineering and technology, instruction modeling unit implementation, and access to and discussion of resources available to them. Teachers in the larger study were randomly assigned to either the EiE or E4C curriculum; teachers were not made aware of the two curriculum that 44 they were not using or that the goal of the research that was being done was to compare the effects of two curricula. PD providers did their best to prepare the teachers to use the curriculum to its fullest effect.
Transcripts of classroom video and student artifacts were primary data sources for answering my four research questions. In each classroom, at least two cameras were used with at least one permanently fixed on a student group, with a tabletop microphone on their desk and another camera and microphone to capture all teacher actions and talk. Student engineering journals were also used as a data source. Students from both treatment groups used journals for data collection, planning, and reflecting on the lessons. All student journals were retained and digitally scanned. IRB approval was received prior to collection of these data. In all cases, only students that have assented and have signed parental consent were used in the data analyses, and data were be de-identified wherever possible.
Data Analyses
With a sociocultural understanding of learning, analyses of interactions that occur among students, between students and teachers, between curriculum and students and between curriculum and teachers must account for the co-construction of the experiences. The meaning- making during these interactions occurs through discourse (Kelly, 2014). Interactional sociolinguistics (Gumperz, 2001) is a technique meant to interpret discourse events, both verbal and non-verbal. However, there are several challenges to using video data, and researchers must be transparent in epistemological decisions about how to systematically examine how moment- to-moment interactions are indicative of what members are jointly constructing (Green &
Stewart, 2005). 45
The first stage of data analysis with both control and experimental classes was the development of an event maps (Kelly & Chen, 1999) to better understand the overall classroom culture. I was not present during the recording, so I could not consider myself a participant observer (Spradley, 1980), but I used some of the same techniques by developing an understanding of the overall class structure. Micro-analysis of failure events irrespective of the broader picture of the classroom culture and norms would be a mistake, and the hermeneutic circle (Polkinghorne, 1983) requires an iterative process of micro- and macro- analysis.
The definitions of sociolinguistic units that were used come from Kelly (2004). An event is an activity around a particular topic or purpose. Events are marked by a change in purpose, type, or activity and are often discursively marked by the teacher (Kelly, 2004). The purpose of the event map is to analyze “how time was spent, with whom, on what, for what purpose, when, where, under what conditions, and with what outcomes” (Kelly, 2004). For the initial stages of this study, I wanted to know how the teacher conducted the classroom on a macro-scale, and it also enabled me to identify interaction units (Green & Wallat, 1981) around failure events.
The second stage of analysis required verbatim transcription of failure events.
Transcription of discourse is not an objective process and requires consideration. It reflects theoretical biases (Ochs, 1979), but transcription of discourse surrounding failure events must be salient to the reader in terms of format and what speech, actions, and additional information is included. Since a basis of discourse analysis is comparison (Lemke, 1990), all failure events that were identified in the event maps from the eight participants were included, rather than only collecting cases confirming preconceptions. The events that were micro-analyzed required careful consideration with respect to the larger context, giving another reason for structuration maps (Kelly, 2004). The difficulty with studying short events under an analytic microscope is 46 that the brevity can distort the event by making it appear to be more significant than it actually was (Lemke, 2007). Thus, any analysis required iterative cycles of analysis of events at the micro-scale as well as contextualizing the events within the scope of the class (the hermeneutic circle).
Three specific analytic methods for discourse were used. First, interactional sociolinguistics (IS) (Gumperz, 2001) served as a tool to help systematically interpret the reactions to failure in this study. IS utilizes contextualization cues (Gumperz, 1982), which are signal shifts in talk that are significant across linguistic levels, such as phonetic, syntactic, lexical, pragmatic, and paralinguistic features of communication (Gumperz, 1999). These verbal signs are processed in conjunction with another signs that leads to a situated interpretation that constitutes how a message in understood (Gumperz, 2001). Events selected to micro-analyze using IS were scrutinized for verbal and nonverbal contextualization cues, so transcripts include non-verbal cues in addition to utterances. Second, for analyses of student engineering journals, I utilized content analysis (Bazerman, 2006), which looks for what is written and can be used both qualitatively and quantitatively.
The final phase of analysis was the construction of a thick description (Geertz, 1994) of the collective data into a narrative. This description must be representative of the data collected.
It must also be salient so the reader can interact with the data as the researcher makes claims about it.
Preventing Researcher Bias/Increasing Validity Creswell and Miller (2000) identify techniques that can be used to address researcher bias. As previously described, the version of phenomenology I chose rejects the idea that a researcher can bracket out all biases in the pursuit of the essences of a pure phenomenon. This 47 admission of the potential for bias in the analyses puts the responsibility squarely on the researcher to actively assure the reader through the precaution he takes that the study is trustworthy. One of these techniques is researcher reflexivity (Creswell & Miller, 2000). I have self-disclosed my role in this research as well as my assumptions, beliefs, and biases in Chapter
One. Another method is to maintain an audit trail because it is impossible to report on all the data, but by diligently organizing and retaining it to allow an external auditor to verify the process and product of the work and to judge its veracity (Creswell & Miller, 2000).
Additionally, collaboration and peer review of all aspects from planning through reporting can increase the trustworthiness of the research (Creswell & Miller, 2000). Since this work is my dissertation research, my committee acted to peer review my work with a skeptical eye, while collaboration with my dissertation advisor lends more credibility to my work.
48
Chapter 4 - Analysis of the Control Curriculum (E4C)
The purpose of this chapter is to develop and apply a model of failure to construct a way for researchers and classroom teachers to consider engineering design failures. The model was designed from a theoretical perspective but it was also borne out of the data, and was used initially in the analysis of the control curriculum (E4C) to improve my methodology prior to analyzing the experimental curriculum (EiE). Spending the time to conduct an in-depth analysis of a control curriculum is unorthodox; but, I chose this route for two reasons. First, to date there has been little research done by scholars interested in exploring issues of engineering failure in classrooms, so a baseline understanding of how failure manifests itself is useful for comparing to newer engineering education interventions. Second, it was important to pilot test my analysis of failure on E4C with the intention of using it in other contexts, both within this study and in the future.
Context of the Study For this chapter, four classrooms engaged in the E4C civil engineering units were theoretically sampled (Patton, 1990). Civil engineering was chosen because failure of structures like buildings and bridges may be easier to identify than failures in many other engineering tasks.
More specifically, two lessons were selected because both had examples of high and low stakes failure; lesson one yielded instances of intended failure while failures in lesson four were mostly unintended. Understanding the dynamics of students designing structures and failing and the discourse moves students and teachers use in these situations is important baseline work that should be done prior to analyzing EiE classrooms. The control curricula were designed to take the same amount of time to complete as the EiE units, typically between 6-8 hours of class time.
As a result, the civil engineering lessons selected have five separate design challenges, providing many instances to analyze students’ designs failing. The four teachers had the same E4C 49 professional development, were all in their first year of teaching this unit, taught roughly the same age students (4th & 5th grade), and were all held accountable to teach the curriculum as written (documented by observation logs and fidelity of implementation journals).
Table 4.1- E4C Classes (*Mr. Tanner and Ms. Houseman taught in the same school)
Demographics % Free/Reduced (% Pseudonym Grade S:T ratio lunch prices underrepresented minority) Ms. Lyle 4th 15% 16% 15:1 Ms. Flemming 5th 35% 97% 18:1 Mr. Tanner* 5th 60% 30% 15:1 Ms. Houseman* 5th
Analysis of Classroom Video A total of 42 hours of E4C classroom video were analyzed. In three of the classes (Mr.
Tanner, Ms. Houseman, and Ms. Lyle), one camera recorded the full (F) view of the classroom and two cameras were fixed on individual student groups (T1 and T2). The student groups’ voices were audio recorded with a portable device and synched with the video using Transana
Multiuser 3.0. In the Ms. Flemming’s class, two cameras were fixed on individual students groups (T and F), however, portable voice recorders were not used and no camera recorded the whole class.
Video data were first viewed at regular speed (Erickson, 1992) and an event map was created (Kelly & Chen, 1999) to better understand the culture of each class and for macro- analysis in the hermeneutic circle (Polkinghorne, 1983) (see Appendix A for an example).
Immediately after creating the event map, I wrote one to two paragraph summaries of each episode, and about salient points or about ways my thought about concepts related to failure had changed during the video. This was done for two reasons. First, it is an easier way to navigate the events when trying to find particular instances within that amount of video data. Second, it 50 serves as an audit trail (Cresswell & Miller, 2000) to increase trustworthiness; my goal was to provide “friendly skeptics” a way to follow the progression of thinking throughout the analysis if they so choose.
My definitions of sociolinguistic units come from Kelly (2004). An event is an activity around a particular topic or purpose, in this case a lesson. Events were represented on these maps as coordinated phases and sequences of activity. The phases of activity were determined by the units used by the curriculum developers and used in observation logs; sequences were developed post hoc through semantic and context clues (Kelly, 2004). I also coded for interaction units
(Green & Wallat, 1981) surrounding failure to consider in future analyses. Each event map was entered into a Microsoft Excel spreadsheet such that each row represents one minute. In this way, direct comparisons could be made between classes enacting the same written curricula.
The next step of analysis was to code the event maps in four categories: failure type, failure cause, teacher reaction, and an open coding category for concepts that seemed to be relevant in which new themes could emerge. Each coded instance was re-analyzed to verify initial coding, and in some cases the events were transcribed verbatim. In order to eliminate redundancies, high stakes failures were only coded in “F” view recordings (the camera focused on the whole class), and low stakes failures were only coded in “T” view recordings (cameras aimed at individual student groups) unless otherwise specified. My coding scheme is found in
Table 4.2. It is important to note that not all instances of failure could be coded for each category, because some instances were either unobservable or uninterpretable. However, since the goal of this work is to develop a theoretical model of failure types and causes and reactions to it, coding and subsequent descriptive statistics are primarily to show the relative frequency and to increase the reliability of the work. 51
Table 4.2– Coding Scheme for Failure type High Low Stakes Failure happens in front of the class Failure occurs in front of an individual as a part of a public presentation student or small group Unintended Intended Intent The solution is intended to meet The solution is meant to be tested until it criteria within constraints and is fails or is designed to fail in a certain way unable to Objective Subjective Referent The solution fails in reference to The solution fails in reference to other stated criteria, constraints or groups’ performance or other previously expectations of function undefined expectations Relatively few clear examples of subjective failure were observed in the data, so for the sake of parsimony, Figure 4.2 classifies the observed failures only by stakes and intent; however, reference will play an important role in some of the observed interactions (described later in this chapter) and should still be considered.
Failure Types Stakes. As described in Table 4.2 (pg. 51), high stakes failures occur as a part of public presentation or when the performance is a final evaluation for the design. Low stakes failures occur when the audience is only a student or student group and there is time for improvement before formal evaluation. To take this continuum to the extreme, the Challenger disaster would be a failure of very high stakes.
Intent. Another important aspect of classifying failure is the intent. When a design is tested until it fails or if it fails in a certain (planned) way, it is intended failure. However, if the design is unable to achieve the required criteria or to remain within the given constraints, it is unintended (Table 4.2). A circuit breaker would be a version of intended failure, because the circuit fails in a planned way. A short circuit caused by a lack of grounding would be a parallel 52 example of an unintended failure.
Referent. When a design fails with respect to the given criteria, the failure is objective. In some cases, though, a design could be construed as a failure compared with others’ designs and/or it is judged to be “not good enough.” (Table 4.2) In those cases, the failures are subjective.
Despite the inclusion of subjective failure in my model for failure type (Figure 1.2, pg.
9), it is difficult to accurately code for this solely through video analysis of classroom discourse.
The referent axis is an important consideration for teachers diagnosing failure type, and I will discuss it in depth in Chapter Seven. Here, I categorized failure type on two axes, stakes and intent.
50
45
40
35
30
25
20 CodedResponses 15
10
5
0 Ms. Lyle Ms. Flemming Mr. Tanner Ms. Houseman Teacher Pseudonym
Low stakes, Intended Low stakes, Unintended High Stakes Intended High Stakes, Unintended
Figure 4.1- Failure types by class In the E4C curriculum, Lesson One had the majority of intended failure, while Lesson
Four had the majority of unintended failure due to the goal of the activity. Lesson One asked students to build “tall structures” that hold as much weight as possible; that is, they are tested 53 until failure. Bridges built during Lesson Four must hold a five-pound weight for thirty seconds.
If it is unable to, it is considered unintended failure. There were some exceptions. Structures built in Lesson One that could not hold any weight or fell down without adding a load were classified as unintended failure, even though the structures were meant to be tested until failure. Sometimes during testing of bridge prototypes, students tested them by pushing on them until they deformed; this was considered an intended failure.
It is important to note that lessons that contain intended failure as seen in Lesson One and are also observed in several K-12 engineering design challenges like balsa wood towers. These types of challenges are unlike most engineering tasks. A civil engineering firm is never asked to build a structure “as strong as possible” because it would waste resources. Rather, a bridge is designed to withstand a certain amount of load, a specified amount of lateral resistance to wind, and then the job of the engineers is to address other criteria, such as making it inexpensive and aesthetic. The activities in this unit could easily be modified to more accurately reflect this aspect of engineering. For example, the card structures could have to be 6 inches tall and hold at least 6 weights: but each card and each inch of tape could have a cost associated with it. In order to systematically design these structures, though, a series of testing to failure would help students determine how strong their structure was. This type of incremental failure with the intention of improvement is called (in this study), low stakes failure.
All of the high stakes failures coded for in these data occurred as a public presentation.
However, in some of the classes, the teacher was the only audience other than the group members. While I considered coding these instances as low stakes, testing in front of the teacher for purposes of evaluation (success or failure) was coded as high stakes.
Summary of failure types. The first part of this analysis considers the types and 54 frequency of failures in this unit. It is important to consider the ways in which these designs failed to better understand the contexts of the events that are analyzed. A better understanding of the failure types will also account for the reasons why designs fail and the ways in which teachers and student react to the failures.
Teacher Reactions to Failure The types of feedback teachers have for students who experience failure can support or constrain the progress toward improvement. Teacher reactions likely reflect their beliefs about students, their own understanding of the material, the culture of the class, and their views of the broader community. I identified three classes of reactions that teachers demonstrated reacting to failure in the E4C curriculum: the manager, the cheerleader, and the strategic partner. While it is important to recognize that none of the teachers demonstrate these responses the same in every situation and may move between roles depending on the situations, each falls into one of these archetypal categories when looking at the data broadly.
Table 4.3 – Coding scheme for reaction type
Reaction Characteristics of reaction type Representative Quotes name Cheerleader The reaction seems to intend to praise “No one should be sad if their bridge or otherwise verbally reward the falls right over…It’s just to learn and student. It may be done to protect a have fun” student from disappointment. Manager The reaction seems to be a result of “Ok, quickly, quickly, let’s get to the running an efficient class, and not next group!” related to actually responding to the “I know you want to do it again, but failure. we’re out of time.” Strategic The reaction seems to be aimed at “What part of the structure was Partner helping the students think about weakest? How could you make that improving or to consider what went part stronger?” wrong and why.
55
Manager. The manager reaction seems to be characteristic of teachers who want students to complete the activity in an efficient manner. This is also likely the type of reaction you would expect from teachers with little content expertise, as these teachers likely constrain conversations to objective questions and answers (Carlsen, 1991). An example of this was demonstrated by Ms. Flemming during the public testing of the card structures in lesson one
(Table 4.4).
Table 4.4 - The Manager Reaction Type – from Transcript T13811CvL12d1TV
Time Line Class (in unison) Ms. Flemming Context Clues 45:55 1
Another example of this reaction type occurred when Mr. Tanner was circulating around the room while groups were designing their structures for lesson one (Table 4.5). 56
Table 4.5– The Manager Reaction Type – From Transcript T14117CvL12d1FV
Time Line Lucy Maggie Mr. Tanner Action Unit 25:16 1 All right Teacher approaches 2 group 3 Two, four, six, eight. We’ve got We came up with a Maggie directs talk 4 eight cards left. solution. to teacher 5 We’re gonna take Motions to their 6 this and turn it into base structure 7 a circle and put it on 8 top of that 9 You’re gonna turn it Hands in pockets 10 into a circle? 11 Don’t we need the cup inside? Yeah, but what we 12 mean… 13 The cup sits on top 14 But we should put Then it will make it 15 the card on top of it a tall structure 16 and then ### 17 We can make it like a box, but 18 not put the top on and put the 19 cup inside and all the bolts inside 20 Will that work? Ask the teacher 21 I don’t know, but Ends conversation 22 you have seven 25:58 23 minutes to figure it 24 out
In each case, the teacher’s talk and actions was not explicitly helping students learn or improve their designs. Ms. Flemming was trying to finish all the testing in one class period; that’s why she was in such a rush to test. Mr. Tanner provided little guidance for the group that was asking for his opinion. Instead, he reminded them how long they had until they must be finished, a time that was counting down on the class’ projector screen in the front of the room.
Cheerleader. The next form of teacher feedback was also demonstrated by Mr. Tanner.
In this case, the first public test of the card structures ended when the structure collapsed after holding thirty-five weights (Table 4.6).
Table 4.6– The Cheerleader Reaction Type – From Transcript T14117CvL12d1FV
Time Line Class Mr. Tanner Contextual Clues 38:00 1 Structure collapses 2 Thirty-five! Thirty-five. Boy in the group holds his arms 3 in the air, celebrating 4 Nice job, all right. Give them a round Teacher starts clapping, joined by 38:05 5 of applause the rest of the class
57
Because this was the first structure tested, and the standards for success were unclear, no one really knew how strong this structure was. I consider this reaction to be praise, an unhelpful form of feedback (Mueller & Dweck, 1998). Public testing events like the one provide a context for students to collectively analyze and talk about the ways that these structures fail, including specific weak points. Collectively, the failure could become a topic for conversation to synthesize aspects that appear to make structures strong and/or weak. Instead, activities involving intended failure like these often fail to look at the bigger ideas, such as mechanisms for supporting loads, and the critical load becomes a score, rather than an indication of the strength provided by design features. Interestingly, a structure tested later in the class held more than seventy; it did not receive applause.
An additional example of the cheerleader reaction can be found in Table 4.7:
Table 4.7– The Cheerleader Reaction Type – From Transcript T13811CvL12TV
Time Line Ms. Flemming Isaiah Contextual Clues (student) 45:16 1 ..they have to be able to go on top… Notices Isaiah’s structure 2 Oooooooooh! Yeah, everyone give Isaiah a Donna starts clapping, class 3 hand! follows 4 I knew you could do it, you’ve got “Obama Directed to Isaiah 5 bucks” from me coming. I told you you can do 6 it! 45:25 7 He figured it out, he figured it out Directed to class
Because this event occurred off screen, it was difficult to tell how many weights the structure held, and was obviously linked to an earlier conversation between the teacher and student. However, the praise was again very broad. Ms. Flemming does not tell the class what design features he “figured out” or have the students consider the aspects that made his structure strong. I call this type of non-specific praise of either a structure or the students’ attempts, the cheerleader reaction.
Strategic partner. The third reaction is the strategic partner. She treats failure as an 58 opportunity from which to learn. She takes time to have students consider the failure analytically and not only what went wrong, but how to improve it. Again, a teacher’s treatment of failure does not only happen after the failure but can be foregrounded prior to the event. Ms. Lyle was heard consistently referring to a mantra of hers that “We learn most from when me make mistakes,” or some variation of that sentiment. 59
Table 4.8– The Strategic Partner Reaction Type – From Transcript T14093CvL12D1T2
Time Line Ms. Lyle Ivan Kaylie Contextual Clues 55:28 1 That structure didn’t Referring to an earlier 2 work. Why? version they had 3 improved in their group 4 The one we couldn’t Wait, our last 5 get to stay up at all 6 Ok Oh, because I 7 think it was too 8 tall, because, um, 9 the second box 10 kept caving in and 11 falling. Like the 12 cards kept coming 13 untaped and stuff, 14 Yep, Kaylie? like they were 15 caving in 16 Um, I think it was 17 because it was too 18 hollow because if you 19 had a card in the 20 middle, it could 21 evenly place it all out 22 and it wouldn’t fall if 23 you taped it in 24 What could even place Asking for 25 it all out? clarification of her 26 point 27 All the cards on the Kaylie uses her hands 28 outside because every to help describe what 29 time we taped them, she means with the 30 they would fall, but cards 31 when if you put a 32 card in the middle 33 and taped it to the 34 cards, you’d be able 35 to stick em together 36 and they won’t fall. 37 They pretty much 38 would be pretty 39 steady and that 40 could’ve worked 41 Yeah, so we changed 42 our design a little bit. 43 So why was this design 44 more successful than 56:23 45 the other one? In this example, Ms. Lyle paid little attention to the number of weights their structure held, but instead took the opportunity to talk about the improvements they made to their initial design. Specifically, she had them focus on the fact their initial design did not have a card supporting the load in the middle of a cube-shaped box. She even marked this as important enough to ask Kaylie for clarification. Brenda also uses the term, “we” (line 41), suggesting she 60 viewed herself as a partner in the group that designed this structure. This reaction stands in contrast to the manager that students look to for judgement because the goals are too unclear.
Ms. Flemming was consistent with her manager-like responses until the end of the popsicle stick bridge tests. After no bridges successfully held up the weight, not only did she add another day on to the unit to allow for redesign, she also took a lead role in talking each group
(publicly) through the aspects that made their bridge unsuccessful and how to make improvements. Prior to that, her talk and actions seemed to be about controlling the pace of the activity and she provided little strategic help. But she seemed to take the failures personally and took steps to help students achieve success. The managerial tendencies were noticeable by the way Ms. Flemming led the class —it was a very time-efficient discussion— but the type of feedback she provided was more like that of a strategic partner.
The discourse observed from the strategic partner can be viewed from an attributional feedback framework (Le Foll, Rascle & Higgins, 2008). The roots of the failure can be attributed to internal or external causes, can be portrayed as static or changeable, and can be specific or general in nature. Table 4.11 is used to demonstrate another aspect of the failure model; however, it also shows Ms. Flemming attributing a failed bridge design to the lack of a pier. This kind of attribution is specific, changeable, and external. During the next iteration of design, students added a pier, an obvious improvement from their first iteration.
Interestingly, attribution theory (Weiner, 1985) from a psychological perspective promotes internal, specific, controllable attribution as the most productive style. From analysis of several failed designs, attributing failure is most productive when it focuses on the external locus of control. For example, “The attachment of the pier to the deck was the weakest spot,” 61 gives a specific and controllable target to improve. This differs with Weiner’s psychological model (1985) which holds that internally attributed failure is preferable.
Figure 4.2 shows the variety of reactions the four teachers expressed in response to failures. You will notice that none of the teachers always react as the same style; even so,
55
50
45
40
35
30
25
20
NumberReactions of Coded 15
10
5
0 Ms. Lyle Ms. Flemming Mr. Tanner Ms. Houseman Teacher Pseudonym
Manager Cheerleader Strategic Partner
Figure 4.2 - Failure Types by Class these descriptive statistics can be further misleading. For example, Ms. Flemming’s reactions were coded as “strategic partner” thirteen times. Nine of those thirteen reactions were in the final class period, when she added an additional day for redesign because there were originally no successful bridges.
Summary of reactions to failure. Categorizing teachers is not useful, because as Figure
4.2 demonstrates, each teacher reacts in more than one way to these failed designs. However, it is useful to consider classifying the reactions when considering the roles of the teacher and the 62 potential effects of the reactions on the students. Conceivably, these reactions send social cues regarding the ways we should think about failure in engineering. For example, modeling failure analysis using the strategic partner response may well support students learning from failure in ways that the cheerleader response may not.
Although I had planned only to consider failure types and teachers’ reactions to them, it became evident during analysis that there was another important aspect of failure to consider: the reason why the design failed. The next section of this study considers causes of failure.
Table 4.9– Coding Scheme for causes of failure
Causes of Failure Cause Characteristics of causes Specific Examples
Lack of knowledge of The students do not understand a key Lesson 1 – Cards need to support the area science/technology science or technology concept. where the load is applied. Lesson 4 – Piers or rigid deck are necessary to support the 5 pound load Lack of knowledge of The students do not understand the Using a metal pipe for a water system materials characteristics of materials they are that will corrode and pollute and effluent using. Poor craftsmanship The reaction seems to be aimed at Designing a bridge that is uneven so the helping the students think about load falls off improving or to consider what went wrong and why. Limitations of materials Eventually, even a well-designed Lesson 1 – even a tower that is designed solution fails due to constraints using software to optimize its strength within the material or the activity. will eventually break if the goal is testing until failure After coding several teacher reactions, it became evident that teacher reactions might vary depending on the reason why the design failed. This led to theoretical discussions with my advisor and with professional engineers to determine the reasons why designs fail. I identified four causes for failure in elementary engineering design projects (and theorize they also hold true for engineering failure outside of school as well). The four causes are: 1) a lack of understanding of science and/or technology, 2) a lack of understanding of the materials, 3) poor craftsmanship, and 4) limitations of the materials used. Table 4.9 describes the coding scheme I used to identify 63 these causes. Due to the nature of video analysis, it is not always possible to determine the exact cause of a design failure, and in many cases there are more than one cause. I increased the probability that these codes are as valid as possible by 1) collaborating with my advisor and colleagues in analyzing video data, and 2) by presenting a thick description (Cresswell & Miller,
2000), and 3) by including several transcripts describing these constructs. My goal is to help teachers and professional development providers use failure as a learning tool, thus the failure causes will be more easily determined when teachers are given the opportunity to discuss the design with the student groups. The actual causes can be ascertained easily by teachers with direct questions to the students, which would enable better feedback. Also, the basis for discourse analysis is comparison (Lemke, 1990), and the intention is to compare the failure causes in the E4C curriculum with those observed in EiE.
Lack of understanding science/technology. The first cause is demonstrated in the episode below. In it, Laura, Max, and Rachel are designing a structure made out of 25 playing cards and 18 inches of tape to hold as much weight as possible (Figure 4.10).
Table 4.10 – Lack of Understanding of Science/Technology – From Transcript T14093CvL12d1T2
Time Line Laura Max Rachel Contextual Cues 17:48 1 We need to put more Puts cards down with 2 support on it. This is triangle on its side 3 the triangle base 4 No, I was thinking Places two cards on top 5 something like… 6 Ok, let’s make another 7 triangle base 8 We have all these Holds up stack of other 9 cards unused cards 10 Ok, do another 18:03 11 triangle Rachel insisted three times that they should use a triangle-shaped structure. Probably, she has heard of the structural benefits of using triangles to distribute a load; however, by placing the triangle on its side (i.e., placing all vertices of the triangle in a horizontal plane, with the load exerting its force perpendicular to that plane), it eliminates the benefits. This misconception of 64 the use of triangular shaped structures to support weight is a misconception frequently demonstrated by groups designing similar structures in a design challenge called, “Power
Tower” in the EiE curriculum (Cunningham, personal communication).
Another example of students’ structures failing due to this lack of understanding scientific principles can be seen in Ms. Flemming’s class during Lesson Four – Popsicle Stick bridges. The class was testing their re-designed structures publicly. Lisa and Rebecca’s bridge was unable to support the weight and collapses, so Ms. Flemming asked the class about it
Table 4.11– Lack of Understanding of Science/Technology – From Transcript T13811CvL42d2FV
Time Line Ms. Flemming Paula Class Contextual Clues (researcher) 38:13 1 Bridge collapses and girls attempt to 2 retest, unsuccessfully 3 That’s it, sweetie Shakes head 4 What did this (directed to class) 5 group need? 6 Raise your hand! [###] Piers! 7 And tell us what’s 8 missing from their 9 bridge. Brianna? 10 Piers. Answered by Brianna 11 Their piers are Widens eyes, sounds frustrated 12 missing. We talked 13 about piers 14 [Yes] [It’s nice and 15 rigid, though, 16 right?] 17 It’s definitely not a Directed towards Lisa and Rebecca. 18 bendy bridge, but it Donna puts fist under bridge to 38:58 19 needed a pier, ok? represent a pier This interaction demonstrates a common reason why many students’ bridges failed to hold the weight. It lacked a support to balance the downward force of the load. The idea of balancing forces is a primary science concept that can be applied in this application. When prompted, the class suggested that adding piers to the bottom would have led to a more successful outcome— a feature Ms. Flemming had emphasized earlier that class.
This episode also serves as an example of another cause of failure in these engineering solutions, a distinct but related reason: a lack of understanding of the materials. In their first 65 iteration, Lisa and Rebecca’s bridge was made of one layer of popsicle sticks and failed. During the redesign, they added rigidity to their bridge by adding another layer. A rigid beam design could have successfully held up the weight if it was constructed of a solid piece of wood of the same thickness, but the students did not recognize that this strategy would not be successful with popsicle sticks.
Lack of understanding of materials. A similar misunderstanding of the materials used to add strength to structures was noted in the following episode. The group of Rosie, Michael, and Larry were trying to finish their card structure and were confused how to continue (Table
4.12).
Table 4.12– Lack of Understanding of Materials – From Transcript T13811CvL12d1FV
Time Line Larry Michael Rosie Contextual Clues 26:14 1 We don’t have 2 enough tape 3 It don’t mat:ter Grabbing more cards 4 We just used
Poor craftsmanship. Many of the designs were unsuccessful because the students apparently lacked the coordination or physical dexterity to construct solutions, and student often failed to accurately measure their bridges so they would be long enough to span the abutments of the bridge. This type of failure is likely to be more common in younger grades, but is a reality at all levels; a well-grounded and elegant design that cannot be executed will fail just like a poorly- 66 conceived design.
Limitation of materials. The limitations of materials are also a reality. Even a theoretically perfect design will eventually fail due to the physical limitations of the materials used. Also, materials can become worn out (e.g., fifty year-old cement) and will fail after years of proper functioning. The types of interactions that could facilitate improvement in response to failures will likely be based in part on the type of failure students have. In the case of students not using piers in their design, feedback could be based around the main idea of bridge design— dispersing the load to the ground. However, when students misunderstood the ability of tape to add strength, students would likely benefit more from experiences around the capabilities of the materials involved in the design.
50
40
30
20
NumberFailure of Causes Coded 10
0 Ms. Lyle Ms. Flemming Mr. Tanner Ms. Houseman
Knowledge of Science/Tech Knowledge of Materials Poor Craftsmanship Limitation of Materials
Figure 4.3- Failure causes by class Summary of failure causes. The causes of failure are important to consider because of the reason why a design fails is important to understand or else improvement is unlikely. And since failure is of little benefit to students (or society) unless it leads to improvement or learning, 67
I next considered the features necessary for improvement after engineering design failure. By better understanding the things that prevent improvement, recommendations may be generated to better enable students to improve, either in curriculum development or in the modification of existing curricula.
Obstacles to Improvement
Regardless of the type of failure or the reason(s) for it, improvement is not inevitable.
Through my analysis of video, coding for obstacles I noticed, and refining the categories, I identified three general obstacles that hinder improvement and/or success. These obstacles are: unfair comparisons, insufficient opportunities, and unproductive strategies. This section describes these categories using examples from the data.
Unfair comparison. The goals in engineering design tasks such as the ones described here are often unclear, presenting challenges both in the initial design and subsequent iterations.
For example, Lesson 1.2 directs student groups to build a structure that is as tall as possible and that can hold as many weights as possible. While the constraints are clear, the task lacks clarity in the definitions of both minimum height and minimum weight the structure should hold.
Classroom tasks like this become confusing to students and teachers, and makes it challenging to understand what success or failure entails. Is a structure that holds thirty weights successful? The students are more likely to express subjective failure in situations where the only measures to compare their structure to are other students’ designs. In one class, thirty weights might be the strongest structure, but in another it could be the weakest. A salient example of this was seen in Mr. Tanner’s class after the public testing of the structures described above. 68
Table 4.13 – Unfair comparison – From Transcript T14117CvL12D1F
Time Line Owen Mr. Tanner 53:35 1 Did we get first, second 2 or third place? 3 First, off, it was not a competition. Because if it was a competition, we 4 would have had a lot more variables, such as height, because directions 5 were a ta:ll structure, right? So it’s not a competition, it’s about how 6 did your group do compared to what you predicted. So don’t worry 7 about first, second, third place. There was no prize. Two interesting and related points arise in this teacher’s description of the results. First,
Mr. Tanner recognizes the flaw in the curriculum directions. In this context, a four-inch high structure that holds thirty weights cannot be judged “better” than an eight-inch structure that holds twenty, and therefore comparing one group’s structure to another is difficult. By the emphasis he placed on the word tall, it seems he thought some groups did not build “tall” structures (line 5). However, the teacher suggested that success would be achieved if their structure held the amount of weight they predicted it would. (These predictions were guesses because they did no testing prior to forming these predictions.) While this example happened at the end of the class when it was too late for any of the groups to improve their structure, any group that would build another prototype would struggle to evaluate improvement because the directions call for the structure to be both as tall as possible and as strong as possible.
Unproductive strategies. Another challenge to improvement is the lack of productive strategies to improve. Like many of the categories in my analysis, it is related to other categories, but distinct enough to warrant discussion. Ideally, the specific weaknesses of a design can be identified and future iterations take action to make improvements. In fact, several professional engineering journals have published case studies that analyze key elements that led to the failure
(for example, the Rodgers Commission (1986) analyzed the Challenger explosion). Explication of weaknesses lead to strategies for improvement. However, watching several group designs in this study, I found that many groups lacked observable, productive strategies. 69
An example of unproductive strategies might be repeating a failed design over and over with no improvement. Although there is something to be said for persistence, some students seemed to exhibit this pattern because they lacked a better strategy. This was evidenced in an episode in Ms. Flemming’s classroom in Lesson 1.1, an individual student activity where the students design a “tall structure” with eight playing cards and a foot of tape. Ida quietly built and rebuilt a structure that was so weak, it could not stand independently. She was very animated each time it fell and was clearly pleased when she added a second layer. She also looked around the room (presumably to see the strategies other students are using) when she was at a point in which she seemed to be stuck. Although she seemed to have a strategy (persistence and learning from peers), she did not appear to recognize the need to connect the cards with tape or by notching them, to add stability and ultimately strength to the structure. (Event map
T13811CvL11d1T).
Insufficient opportunity. Even if students have a clear goal and a productive strategy, improvement cannot occur without the opportunity. Teachers have to make curricular decisions several times a day between covering content versus slowing down or remediating to improve student learning. Similar decisions are required in teaching units of engineering design. Given enough time, students could all build bridges that support five pounds for thirty seconds; but, it would be at the expense of other units of study. The time devoted to the design projects in E4C varies among teachers, attributable to these decisions. Ms. Lyle’s class completes lesson four in one hundred twenty seven minutes. Mr. Tanner’s class takes one hundred twelve. Ms. Houseman completes lesson four in eighty-three minutes, Ms. Flemming in eighty-seven. Ms. Lyle makes the decision aloud that the students needed more time to construct their bridges, while Ms. 70
Houseman and Mr. Tanner do not make that same choice. In fact, Mr. Tanner is heard on the video say to the videographer,
(14:25 – Transcript T14117CvL12FV) I’m wondering if they had two hours to build this, what the success rate would be versus ####. Because they’re all working and they’re all trying. So #### give them more time building, or… just going through the process for a little bit and then learning from it. But I think we’re going to have a higher success rate than last class, because we only had one successful bridge.
In contrast, although Ms. Flemming class completes Lesson Four in only eight-seven minutes, she chooses to extend the lesson an additional day because no group built a successful bridge. In the next class period, not only does she give the groups a chance to improve their bridges and re-test, she has public conversations with each group about the improvements they should make, and talks about the benefits of a rigid deck and adding piers to support the load. As a result, each group shows improvement, even though no bridges were successful (event map
T13811CvL42D2F).
Although Ms. Flemming’s decision to extend her lesson by a day to let student redesign was interesting, especially since the E4C curriculum was designed to have no improvement steps. Generally speaking the process of redesigning after high stakes failure is prohibitive from the perspective of cost in time and materials, because teachers have to have at least twice the amount of materials and use more class time. However, improvement from low-stakes failure during the design process is an important opportunity that was lost in many of these classes.
Testing of designs in student groups was not allowed by the teachers. The following two episodes happened during lesson one in Brenda’s class. Laura, Max, and Rachel were designing their structure as Brenda sat down to check their progress (Table 4.14) 71
Table 4.14– Denying the opportunity for low-stakes failure – From Transcript T14093CvL12D1T2V
Time Line Laura Max Rachel Ms. Lyle Contextual Clues 21:15 1 Now is this going to Waves her hand over the 2 be attached, or just structure 3 free-standing? 4 [Just kinda [Ok] 5 free- 6 standing] 7 ºWe can just 8 see what it’s 9 gonna doº 10 11 12 We already Points to some school 13 tested with supplies 14 putting that 15 on 16 No testing, Max. 17 Remember, no 18 testing. 19 Well, we just 20 put it on 21:35 21 No, no testing Max. Ms. Lyle laughs
Then, ten minutes later, when the teacher was no longer with the group (Table 4.15):
Table 4.15– Denying Opportunity for Low-stakes Failure
Time Line Laura Max Rachel Contextual Clues 31:00 1 Wait, let me see Grabs a journal from the desk 2 something. I want 3 to see if it can hold 4 something 5 [No! No testing!] [No testing!] Max backs away, crossing his arms 6 I really wanna test 7 it 8 Like, after we’re 9 done, I really want 31:20 10 to see will ### Prior to Ms. Lyle arriving at their group, the students (understandably) want to test the strength of their structure, so they use school supplies they have at their desk as the load. This information would give them insight into the weak points of their design. Ms. Lyle’s understanding of the curriculum lead her to believe that this type of testing is not allowed and forbids it, emphasizing her point by repeating it (lines 16-18, 21, Table 4.14). Then, when Ms.
Lyle is not there to enforce the rule, Max again wants to test their structure, but the “no testing” 72 rule is upheld by his group. Thus, any improvements students attempted to make would not be evidence-based, but based on speculation.
These examples contrast differences in the types of opportunities the enactment of this curriculum could provide for student improvement after failure. These opportunities support productive strategies by allowing students to gain a better understanding of the material properties and potentially the science concepts of balanced forces by giving them events to analyze, both in their group and by observing the designs of other students. More interestingly, all four teachers in this study outlawed testing at the group level, even though this point was not mentioned specifically in the teacher guide.
Summary of obstacles to improvement. Despite stories of failure eventually causing success, improvement is non-trivial. Through analysis of approximately fifty hours of video of
E4C students doing hands-on designing, constructing, and testing structures, no groups improved due to failure. Even in Ms. Flemming’s class where she discusses every student’s design publicly to talk about what improvements to make, no group’s bridge is able to hold the weight for thirty seconds. This E4C curriculum was designed as a control curriculum to compare with
Engineering is Elementary, so it does not allow for redesign and evaluation after high stakes failure—which is why Ms. Flemming’s case was so interesting. And teachers’ denying low stakes testing prevented improvement even within the groups prior to public testing. But opportunity alone does not guarantee improvement, and requires a curriculum that is set up with fair tests to compare subsequent prototypes, and students need to use productive strategies or their designs will not improve. 73
Summary of Analysis of Classroom Video
In this first phase of developing my analysis methods, I chose to study a control curriculum like E4C for several reasons. The civil engineering unit provides a comparison for the bridges unit from EiE, and it also features several design challenges in which to observe students’ designs failing. Lessons One and Four were selected due to the nature of the failures.
There are opportunities to analyze high and low stakes failures as well as watching failures that were both intended and unintended. Through this study, I developed a methodology for systematically analyzing the classroom discourse of students and teachers engaging in engineering and developed a model of improvement with which to apply in my analysis of the
EiE: Bridges unit in Chapter Five. The model developed includes the constructs presented in this chapter: failure types, failure causes, and obstacles to improvement. Teacher reactions support or constraint the improvement process by contributing (or not!) to the students’ ability to develop productive strategies (Figure 4.4). 74
Figure 4.4 - Model of Improvement Analysis of Engineering Journals
Engineers use journals as a tool to design and improve solutions. In them, you will find problem specification, criteria, constraints, initial ideas and calculations, results from pilot tests; the journal serves as a record of the evolution of a design. My goal is not to critique a control curriculum because it is intentionally lacking components thought to contribute to high-quality engineering instruction. However, in Chapter Five, I will use the student engineering journals to evaluate the process of improvement. This relies on an assumption that what students write in journals adequately represents the events that occurred in the group work. This section serves to 75 directly compare the types of information recorded by students in the E4C classrooms with the event map data I generated.
Comparison of student journals with event maps
Tables 4.16 and 4.17 compare what each student on camera wrote with evidence from event maps I constructed during the initial analyses of the video data. Since E4C is a control curriculum and was designed to leave out some theorized critical components of learning, the
E4C journals are mostly for definitions, sketches of designs, and record keeping. I chose to include student groups’ critical load from Lesson 1.2 (pg. 16 of their journals), and whether their bridge was able to hold the weight in Lesson 4.2 (pg. 42 of their journals). This is to not only compare what they write and what I observed, but also to consider the consistency within the student groups. While this approach is admittedly unhelpful in understanding student learning during these design challenges, this bolsters the analysis in Chapter Five, where I take a systematic approach to scoring learning through their journals. This first step is to build the case that students generally write in their journals true accounts of what happens during group work.
Of the 52 responses recorded from student journals, only five fields were blank, and in no case did the blank responses account for more than one-third of the group responses (i.e., at least two out of three students responded to every query). When comparing within groups, I found responses to rarely be in conflict. The only example found in which students reported data in conflict with my observations was in group one of Ms. Flemming’s class. Two of the three students claimed that their bridge was able to hold the weight, while one claimed it did not. My event map for that test recorded, “Group 2 tests— fails. Ms. Flemming ‘gives them credit for about three seconds’ then says thank you and sends them away” (line 42,
T13811CvL41_L42D1F). While I cannot account for this discrepancy, it seems reasonable that 76 the students misinterpreted failure due to Ms. Flemming’s assertion that they were given partial credit. Aside from this one outlier, every response in Tables 4.16 and 4.17 match with other group members’ responses as well as with the observable data from the video recordings. 77
Table 4.16– Comparison of student journals with event maps (Ms. Lyle & Ms. Flemming)
Teacher Lesson Student Journals Event Map evidence Table
Critical Load 1.2 74527 49 “Table 1 tests – breaks at 49” (line 53, “How many T14093CvL12d1t2V) 74521 49 1 weights did you structure hold?” 74513 49 (pg. 16 of journal) 74512 Blank Popsicle Bridges 74527 Yes “Next group tests (t1). Piers directly under 4.2 ends of weight. Holds the weight.” (line 54, 74521 Yes T14093CvL42d1FVa) 1 Did the popsicle bridge hold? 74513 Yes (pg. 42 of journal) 74512 Yes Critical Load 1.2 74531 33 “Structure collapses, also at 33. Teacher asks Ms. Lyle for the next group to go - no acknowledgment” (pg. 16 of journal) 74528 Blank 2 (line 42, T14093CvL12d1T2V) 74522 33 74515 33 Popsicle Bridges 74531 No “When J places the weight on, the connection 4.2 of the left pier gives way, and the bridge 74528 No collapses. They re-align the pier, take a book Did the popsicle off the abutments, and try again. It holds bridge hold? 74522 No 2 momentarily until the same left pier comes (pg. 42 of journal) 74515 No untaped and gives way.” (line 50, T14093CvL42d1T2V)
Critical Load 1.2 74950 1 “Group 1 tests – Prediction was 5 – broke on the first one.” (line 33, T13811CvL12d1TV) 1 (pg. 16 of journal) 74948 1 74940 Blank Popsicle Bridges 74950 No “Ms. Flemming can’t test it because it’s not 4.2 long enough. She also points out it wouldn’t 74948 No be strong enough.” (line 40, 1 Did the popsicle T13811CvL41_42d1TV) bridge hold? 74940 Blank
Ms. (pg. 42 of journal) Flemming Critical Load 1.2 74952 13 “Prediction is 11 – Structure much taller, holds 13, Ms. Flemming quickly moves to them on (pg. 16 of journal) 74956 13 2 to the next group.” (line 45, T13811CvL12FV) 74949 13 Popsicle Bridges 74952 Yes “Group 2 tests – fails. Ms. Flemming gives 4.2 them credit “for about 3 seconds” then says 74956 Yes “thank you as she sends them away.” (line 42, 2 Did the popsicle T13811CvL41_L42d1F) bridge hold? 74949 No (pg. 42 of journal) 78
Table 4.17– Comparison of student journals with event maps (Mr. Tanner & Ms. Houseman)
Teacher Table Lesson Student Journals Event Map evidence
Critical Load 1.2 74821 1 “Anxiety from T1 about their structure. “How many weights Collapsed on first washer.” (lines 46-47, 74819 1 1 did you structure T14117CvL12D1T1V) hold?” 74615 1 (pg. 16 of journal) Popsicle Bridges 4.2 74821 No “Piers are too long, so deck is a few inches above abutments. It collapses under the Did the popsicle 74819 No 1 weight immediately” (line 30, bridge hold? 74615 No T14117CvL12D1T1V) Mr. (pg. 42 of journal) Tanner Critical Load 1.2 74624 Blank “Cup starts to fall off. Mr. Tanner starts to “How many weights 74619 35 call that as the end but lets them persist until 2 did you structure it breaks at 35. Calls for applause. (line 38, hold?” 74617 35 T14117CvL12d1T2V) (pg. 16 of journal) Popsicle Bridges 4.2 74624 No “Weight falls off immediately. Mr. Tanner asks what they needed to do, they decide they Did the popsicle 74619 No needed another pier.” (line 39, 2 bridge hold? T14117CvL42D1T2V) (pg. 42 of journal) 74617 No 1 Critical Load 1.2 74567 41 “Gives way on one side @1:59. Jaime leaves “How many weights the shot, Michah counts the weights, and did you structure 74565 41 other boy plays with the broken structure and hold?” weights. Michah says they got 41.” (line 33, 74563 41 T14116CvL12D1T1VaVb) (pg. 16 of journal)
1 Popsicle Bridges 4.2 74567 No “They take it to the front to test. It breaks.” (line 26, T14116L42D1T1V) Did the popsicle 74565 No bridge hold? 74563 No (pg. 42 of journal) Ms. 2 Critical Load 1.2 74566 27 “At 0:46, the cup tips over. The group yells Houseman “How many weights 74561 27 out, ’27,’ but I’m not sure if collapses or just did you structure got off balance.” (line 35, hold?” 74555 27 T14116CvL12D1T2) (pg. 16 of journal) 2 Popsicle Bridges 4.2 74566 Yes T2 is next to test. Obstructed view. It bends, but must hold. Ms. Houseman asks them Did the popsicle 74561 Yes what they might do. They say add legs, she bridge hold? makes them call them piers. She then tells (pg. 42 of journal) 74555 Yes them to go back and reflect on that, and tells them good job. (line 27, T14116CvL42D1T2V)
79
Summary of Analysis of Engineering Journals
The analysis of the student journals is a basic comparison of what students wrote and the reality they socially constructed as a group. As I will propose in Chapter Five, student journals may be a useful tool in analyzing student learning during the process of engineering design that may supplement traditional forms of assessment. However, student journals cannot serve as a proxy for actual events if students do not write about actual events. The data from this analysis suggests that: 1) students accurately capture the results or what they have done in their journals, and 2) results from one student journal can serve as a proxy for the group.
Summary of the Analysis of the Control Curriculum
In this chapter, I have refined existing research methods (Kelly & Chen, 1999) to address how to understand engineering failure through interactional analysis. I also developed a model to consider the aspects of the complex process of improvement after failure. The model consists of two primary aspects: causes of failure and obstacles to improvement. However, equally important are the classification of the failure type as well as the discourse that occurs in response to it. The interactions between the students, the teachers, and the enacted curriculum will support or constrain the progress toward improvement, and the types of feedback provided will also vary based on the type of failure. See Figure 4.4.
Not all failures in engineering are equal. The collapse of the walkway at the Hyatt
Regency hotel in Kansas City (Petroski, 1985) was significantly different than a failed popsicle stick bridge. Expanding on this point, I categorized failures into three continua: stakes, intention, and reference. This categorization allowed me to theoretically sample (Patton, 1990) two types of lessons; one that contained opportunities for intended failure and the other for unintended failure. Both had examples of high and low stakes failures. 80
These data suggest there are four reasons students’ designs failed: students lacked understanding in materials or science concepts, they were limited by the materials provided, their craftsmanship was lacking, or some combination thereof. The types of discourse moves the teachers and students use in each of those cases can differ greatly in their content. However, the discourse in many of the situations constrained improvement because they failed to address certain obstacles.
Having a productive strategy is essential to improve. Using an engineer’s approach, students could analyze failure by identifying weaknesses and focus their improvement efforts on them. Additionally, they could learn from other students’ failures, called vicarious failure by
Kapur (2014); however, two of the teachers in this study (Ms. Houseman & Mr. Tanner) chose to conduct the formal bridge tests while others continued to construct their bridges. Teachers may use attributional feedback (Le Foll, Rascle & Higgins, 2008) to help students find these areas to improve, and by scaffolding students to carefully consider these external, changeable, and controllable factors, they can foster the process of productive failure.
Without fair comparisons, improvement is difficult because students are unable to compare the second iteration to the first. For example, even if the E4C allowed for an improvement cycle in Lesson One, the second prototype will likely not be comparable with the first because height was not an evaluated criterion (although it was in the directions). While this can be addressed by curriculum developers, teachers with an understanding of this aspect may reflect these goals in the way they introduce, implement, and evaluate these types of lessons. An example from our data misrepresenting the nature of engineering design was demonstrated by
Mr. Tanner that considered the activity successful if the predicted critical load (a guess) matched the actual critical load (line 53, T14117CvL12D1FV). This approach is strikingly similar to way 81 the traditional scientific method is sometimes operationalized to consider only the hypothesis and conclusions (Windschitl, 2004), but is equally as troubling from the perspective of promoting students’ engagement in engineering practices.
Improvement cannot occur without the opportunity. Teachers sometimes make the curricular decision to give students more time to design. Ms. Flemming added a day at the end of the unit to allow her students an opportunity to redesign their bridges. But the opportunity that was lost most frequently was the chance to learn from low-stakes failure because that kind of failure was not explicit in the curriculum and was not encouraged by teachers. Given the chance, thoughtful strategies, and clear goals, students are more likely to improve upon their design in each iteration.
Teachers’ and students’ enactment of curriculum varies significantly, even in a study like this in which EiE researchers attempt to control for fidelity of implementation. It is my understanding that teachers were not coached on how to react to failure, so there were a variety of discourse moves teacher used. Two of the reaction types (the manager and the cheerleader) provide little feedback to enhance learning (Hattie & Timperly, 2007). Sometimes teachers used discourse to manage the classroom and to complete their lessons efficiently. In other situations, teachers gave praise, which has been shown to do little good for students’ progress (Dweck,
1986). Arguably, engineering design projects with clear goals shift some of the burden of evaluation from the teacher and provide opportunities for teachers to help students strategize ways to improve. Modeling the practices of engineering, a teacher may demonstrate strategies for failure analysis and redesign. This may become a “habit-of-mind” (Katehi, Pearson & Feder,
2009) and a transferable skill that will work in many contexts in and out of the classroom. 82
This work serves as a baseline of comparison for my analysis of the Engineering is
Elementary curriculum on designing bridges because I studied teachers that all: 1) enacted the engineering curriculum of a similar engineering field, 2) completed a fidelity of implementation log, and 3) attended a similar duration in professional development training in the teaching of their curriculum. However, these teachers taught a curriculum in which failure was not mentioned in the curriculum as an opportunity for students to learn, so comparisons can be made.
Further, three angles were video recorded for each classroom, enabling me to get a wide view of teacher discourse and actions in addition to the interactions of the students that were not teacher- mediated during the design and construction phases. Chapter Five will use the model developed here to consider failure that occurs while students engage in the Engineering is Elementary: To
Get to the Other Side unit.
83
Chapter 5 - Analysis of the Experimental Curriculum (EiE)
The purpose of Chapter Four was to develop and apply a model to better understand failure in elementary-level engineering design projects. In Chapter Five, I begin with a parallel analysis of the civil engineering unit of a commercially available curriculum, Engineering is
Elementary (EIE), to determine the generalizability of the model. EiE considers improvement from failure as a critical component of its curriculum, and includes improvement cycles in every unit, so the second part of this chapter examines specific cases in which student groups systematically improve from failed designs. This kind of improvement did not occur in the control curriculum because the lessons did not afford the opportunity. I then propose a method of documenting evidence of learning from design failure through evaluation of student engineering journals. The parallel analyses in Chapters Four and Five will be compared in Chapter Six in an attempt to make more general claims about failure and subsequent improvement.
Context of the Study
The classrooms analyzed were theoretically sampled (Patton, 1990). I chose the civil engineering unit of the Engineering is Elementary series entitled, To Get to the Other Side:
Designing Bridges to compare to the civil engineering unit of E4C. Lessons Three and Four were chosen for analysis because they include opportunities for failures in both stakes (high and low) and intent (unintended and intended) (see Figure 1.1, pg. 9). Students evaluate three different bridge designs and explore the material properties of the building supplies in Lesson Three.
Lesson Four is centered on a challenge in which students design, construct, test, and improve bridges. The four teachers in this sample received the same EiE professional development, teach students of similar ages, and were required to teach the curriculum uniformly with other teachers in the study, enforced as documented by observations and required fidelity of implementation 84 logs. The teachers in this data set all taught EiE: Bridges as their first EiE unit as a part of a larger efficacy study. It was also the first time any of the teachers had taught an EiE unit.
Table 5.1– EiE Classes (*Ms. Maddux and Ms. Clay taught in the same school)
Demographics (% % Free/Reduced Pseudonym Grade underrepresented S:T ratio lunch prices minority) Ms. Thomas 3rd 2.7% 19% 18:1 Ms. James 4th 6.6% 22.6% 16:1 Ms. Maddux* 3rd 19% 16% 19:1 Ms. Clay* 3rd
Analysis of Classroom Video
A total of 97 hours of EiE: Bridges classroom video was analyzed. In each class, one camera was fixed on a wide view of the classroom and the video was synched with an audio file recorded using a lanyard microphone worn by the teachers. Two additional cameras were fixed on individual student groups, and audio of student group talk was recorded using a portable microphone wired directly to the camera.
The methods I used were similar to those used in Chapter Four. To briefly reiterate, video data were first viewed using Transana 3.0 at regular speed (Erickson, 1992) and an event map was created (Kelly and Chen, 1999) in Microsoft Excel to better understand the culture of the classroom, to identify patterns of activity, and to “zoom out” to be better able to contextualize failures within the hermeneutic circle (Polkinghorne, 1983). Refer to Appendix A for an example of these event maps. Each row of the event map represents one minute of footage so comparisons of time units can be made between teachers. In total, 43 event maps were used to describe the general activity of the class, to highlight incidents to return to for additional analyses, and to break the activities into phase and sequence units (Kelly, 2004). Phase units are considered to be coordinated action among participants with a common focus of the group (Green & Wallat, 85
1981; Kelly & Brown, 2003). These units were aligned using broad activity types from the observation logs completed by EiE research staff. An example of a phase unit was “Construct and Test Bridges.” Sequence units are identified post hoc by using context clues to separate phase units into thematically concerted interactions (Green & Wallat, 1981; Kelly, Crawford, &
Green, 2001). Example of a sequence units were, “Construct deck” or “Low-stakes testing of stability.”
From a sociolinguistic perspective, background information and context are essential aspects of analysis and a researcher must “zoom in and zoom out” to make sense of the data
(Kelly & Chen, 1999). He does this by looking across spans of time and then looking at micro- level interactions, and back at the longer timeframes. The volume of data restricts others from devoting that amount of time to verify my assertions, so I am the only researcher to have evaluated these data with my research questions. Nevertheless, a study gains validity with collaboration (Cresswell & Miller, 2000). A graduate student colleague using this data set also created 20 event maps in this way. We compared sequence units and worked to establish a shared understanding for what counted as a new sequence, and what types of discourse moves signal these shifts. My purpose was not to formally establish interrater reliability; however, it was useful to gain another researcher’s perspective on classroom discourse. These event maps should also be useful to other researchers interested in using this data set.
Immediately upon mapping each class period, I wrote a one to two paragraph summary of the episode. These summaries facilitated finding events easier than reading through several event maps, and they serve as an audit trail (Cresswell & Miller, 2000) for keeping track of the order in which I viewed videos and how my thinking developed and changed over time. I often wrote about new concepts I thought about while viewing each episode. Sometimes these concepts did 86 not develop into an important part of my research; however, I also found it useful to periodically review how my ideas about failure type and reactions to it developed over the course of the 15- month analysis.
I then coded the event maps using the same constructs for failure type, teacher reactions and causes of failure that were used in Chapter Four. I also used an open coding category for events I thought were interesting and could potentially become important in my analyses. To prevent coding the same event multiple times (due to multiple camera angles), I cross checked event maps for overlap. Again, to increase validity (Cresswell & Miller, 2000), my colleague also coded for failure types, causes, and reactions for three episodes. In cases where we disagreed, we worked to come to an agreement and I made changes when appropriate.
Failure Types
Table 5.2 – Coding scheme for failure type
High Low
Stakes Failure happens in front of the class as a part of Failure occurs in front of an individual student a public presentation, or when the performance or small group, and time exists for is a final evaluation for the design improvement Unintended Intended Intent The solution is intended to meet criteria within The solution is meant to be tested until it fails constraints and is unable to or is designed to fail in a certain way Objective Subjective
Reference The solution fails in reference to stated criteria, The solution fails in reference to other groups’ constraints or expectations of function performance or other previously undefined expectations As described originally in Figure 1.1 (pg. 9), I coded the failures into three categories: stakes (high or low), intent (unintended or intended), and referent (objective or subjective).
Stakes and intent are directly attributed to the curriculum and/or teacher. For example, a public test in front of the class leads to high stakes failures. Failures in the referent category can be attributed to students and/or teachers, and are much more difficult to determine unless students 87 verbalize their comparison to other designs. My coding scheme for failure type can be found in
Table 5.2, and is identical to Table 4.2 (pg. 51).
50 45 40 35 30 25 20 15
Number Failures of Coded 10 5 0 Ms. Thomas Ms. Maddux Ms. James Ms. Clay Teacher Pseudonym
Low stakes, Intended Low stakes, Unintended High Stakes Intended High Stakes, Unintended
Figure 5.1 - Failure Types by Class Summary of Failure Types. In this section, I described the different failure types the students encountered in these classes. The EiE: Bridges unit is described in greater detail in
Chapter Three, but to summarize, all of the high-stakes failures occurred during the fourth lesson, during public testing of the design challenge. The designed bridges had four criteria on which to be evaluated (using a rubric developed by EiE). The first criterion was cost. Each building material was assigned a value, and total cost of each design corresponded to a score.
Designs over ten dollars earned a score of one; designs under four dollars earned the highest score of five. A “barge” also had to be able to pass under the bridge. The barge was represented by a block of wood, a pencil box, or a three-hole punch. Third, bridge stability was evaluated on its ability to allow a self-propelled toy car to safely drive across. The design earned a point for each of the four attempts traversing successfully. Last, bridge strength was measured by adding weight to the deck until it collapsed. A perfect score of five was achieved for holding more than 88 one hundred weights; the minimum score was earned for all bridges able to hold less than twenty five weights. The EiE: Bridges unit provides for three public tests (one initial design and two iterations of improvement), so there are many opportunities for high stakes, intended failure
(strength test) and high stakes, unintended (span length, barge test, and stability test).
In the third lesson, students compare relative strengths of three bridge types: arch, beam, and deep beam. During these tests, students are directed to closely watch how the bridge and abutments behave when the load was applied until the bridge failed. These tests account for several of the low-stakes, intended failures noted in each teacher’s class. Figure 5.1 shows that there were several more instances of low stakes, intended failures in Ms. Thomas’ and Ms.
Clay’s classes. In Ms. Thomas’ class, each table of four was broken into two groups of two, so twice the trials were coded compared to groups of four. Also, some groups tested designs more than once, while others did not.
Requiring designs to achieve multiple criteria within given constraints creates the opportunity for instances of high and low stakes, and intended and unintended failure. It also creates the opportunity to improve in multiple criteria, which I will speak about later in this chapter and provides several opportunities for teacher feedback.
Teacher Reactions to Failure
In these data, I identified every instance in which teachers reacted (verbally or nonverbally) to engineering design failure. This coding was done using the same coding scheme as in Chapter Four (see Table 4.3, pg. 54). By coding and counting these reactions, my intent was not to classify each teacher as a manager, cheerleader, or strategic partner. As in Chapter Four,
I coded actions; I did not develop categories to classify teachers. As Figure 5.2 (pg. 97) 89 demonstrates, most teachers (with the exception of Ms. James) used more than one style of response to design failures.
Manager. The manager reaction is when teachers are concerned about time constraints and efficiently completing the unit. This is understandable from teachers during the first time teaching a complicated unit like EiE: Bridges, and I suspect the frequency of this reaction would decrease in future years, as teachers gain experiential knowledge in the types of mistakes students typically make while designing these bridges. An example from Ms. Thomas, exemplifies this type of reaction. This incident occurs during the first public testing phase of
Lesson 4.2 (Table 5.3).
Table 5.3– Ms. Thomas reacts as a manager – from Transcript T14142BrL42d1T1B
Time Line Mrs. Thomas Lenny Context Clues 25:50 1 Lenny, did you guys finish 2 counting yet? 3 Yes, seventy eight Turns from bridge toward Ms. 4 Thomas 5 Wow, seventy eight? Ok, can we 6 have those weights please? 7 Yeah Hands Mrs. Thomas bowl full of 8 weights 9 All right, guys, we need to do this Hands weights to Ronnie 10 quickly. We have five more 11 bridges to test and we don’t have 26:15 12 much time
In the eight hours of Ms. Thomas’ class, she emphasizes her expectation that the class be orderly. In this situation, she is trying to manage a third-grade class’s design and public demonstration of bridges, and she is so behind schedule, she enlists the help of a researcher to help facilitate public tests, suggesting a motive for this reaction style. In another example, Ms.
Maddux reacts to a group’s high stakes, intended failure during a strength test (Table 5.4): 90
Table 5.4– Ms. Maddux reacts as a manager - From Transcript T14143BrL42d1T1B
Time Line Mrs. Maddux Leah Farrah Rory Context Clues 9:37 1
Cheerleader. Only five examples of the cheerleader reaction type, representing only about 14% of the responses were observed. This was surprising since many more of these were observed in E4C classes. The first example is of Ms. Clay, reacting to a design failure in the strength test on the fifth class period of Lesson 4.2. Upon a bridge collapse, Ms. Clay responds to
Rena’s comment that their strength actually got worse than the previous design (Table 5.5). 91
Table 5.5– Ms. Clay reacts as a cheerleader – From transcript T14098BrL42d5T1B
Time Line Mrs. Clay EiE Researcher Lacey Rena Context Clues 46:53 1 Thirteen Places a weight 2 Fourteen Places a weight 3 Fifteen Places a weight, 4 some weights 5 fall off the side 6 That’s it, right? That’s it. Oop. Good job 7 Fifteen↑ 8 Fifteen Fifteen? That Puzzled look on 9 was worse than Rena’s face 10 the first one 11 Oh, dear, you’ll have to 12 figure out what went 13 wrong. That’s part of your 14 job. 15 We need more Quietly to Rena 16 tape right here 16 But we had fun, didn’t 17 we? 18 Yeah 19 And you did a good job, 47:18 20 didn’t you. Yes, you did.
During my initial screening of this episode, lines 11-14 suggested that Ms. Clay would go on to help her students analyze what went wrong and why. Instead, she asks if they had fun and tells them they did a good job. In this case, the girls did not seem to understand that the deck of a bridge needs to be rigid, and their weights fell off the side. Lacey still tries to determine what went wrong (lines 15-16) when she quietly tells Rena they needed more tape. But it is likely that
Ms. Clay realizes that they will not be able to redesign and does not want them to be discouraged, accounting for this type of reaction.
The next transcript demonstrates an example of the challenges in coding. Although it could be initially construed as a manager, I classify it as a cheerleader reaction because the intent seems to be more motivational than attending to the efficiency of the lesson (Table 5.6) 92
Table 5.6– Ms. Maddux reacts as a cheerleader – From transcript T14143BrL42d2T1A
Time Line Class Farrah Stefan Ms. Maddux Context Clues 33:48 1 Seven, eight Seven, eight Farrah adds weights 2 Nine Is it tipping yet? 3 It’s out, it’s out Points at bridge 4 sagging below first 5 abutment block 6 No it’s not It’s not under the 7 first abutment yet, 8 is it? 9 Yeah, yeah it is Another student 10 points to the 11 sagging bridge 12 So nine? So 13 Farrah, when you 14 were touching it, 15 it looked like it 16 was going lower. 17 All right, so nine? 18 Yeah Counts weights 19 Is that better than 20 yesterday? 21 Yeah Shakes head, smiles 22 Yea↑ Good job↑ Gives Farrah a 23 Awesome. Ok. “high five” 24 All right, go fill it 25 in. That’s 26 awesome. That 27 means you did it 28 even better. Great 29 job↑ That’s 30 exactly what you 34:19 31 wanted to do.
Ms. Maddux reacts in this way after the first redesign for Table 1, and the students had one more class period in which to improve it. According to the EiE scoring system, this bridge received the lowest strength score possible because it held fewer than twenty five weights. While it is important to acknowledge progress, this type of praise (Mueller & Dweck, 1998) may not help students attribute a cause of failure (Weiner, 1985), arguably necessary if they are to develop a productive strategy for improving their design.
Strategic partner. A third category of teacher reaction to failure is the strategic partner.
The goal of this type of response is to help students improve a new design, by helping them consider what went wrong and/or why. There were at least three coded instances of this type of 93 reaction for each of the four teachers. One example of this kind of reaction is from Ms. James, speaking with Reggie, Madden, and Lara after the group had spent almost two full class periods constructing a bridge with little success. She notices the group struggling to come up with another strategy, and sits down with the students (Table 5.7).
Table 5.7– Ms. James reacts as a strategic partner – From transcript T14164BrL42D2T2A
Time Line Madden Reggie Ms. James Context Clues 46:32 1 Lara, Madden and Reggie 2 are looking at the bridge 3 We have so many ideas Mrs. James sits down. 4 Ok, so you’re shooting and Reggie replies inaudibly 5 brainstorming ideas with each 6 other? 7 Ok, so you need counter forces 8 because you have a weakness in 9 the center. 10 Yes 11 All right, failure in the center. 12 What are some options that you 13 can have, even looking around 14 to see what other are doing 15 16 We were trying to put a Madden demonstrates to 17 popsicle stick (across but it Mrs. James 18 didn’t work) 19 Ok, so did you think about 20 doing a deep beam? 21 Uh, no (inaudible) 22 Ok, so you may have to make an 23 improvement. You may have to 24 go open your wallet and buy 25 something else because you do 26 need to make it stable and 27 strong, so…We do know that 28 the deep beam is stronger than 29 the beam. And it may be an 30 adjustment you can make, Madden nods 31 especially since you have to get 32 the barge through and you 33 already have piers built, ok? 34 All three students stare at 35 bridge 36 Uh, what other ideas are people Students look around at 37 doing? other groups. Madden points 38 at one but his words are 39 inaudible 40 Ok (6 seconds). Mrs. James gets up and 47:58 41 All right leaves. Lara sighs.
Ms. James is direct in her comments to the group. This reaction is understandable in this situation because she realizes they do not have much time to execute the redesign and wants to give them a strategy that will work, so she tells them what is causing their failure in strength 94
(lines 7-9). First, she suggests they consider other students’ designs (lines 13-14), then she suggests a design (lines 19-20) and marshals evidence from their lesson comparing bridge designs (line 28-29).
Another example, though less direct than Ms. James, occurs at the end of the third design in Ms. Clay’s class. The group had just completed their public test in which their strength decreased. The weights fell off a flimsy deck in all three strength tests. Part of this problem is related to the way Ms. Clay’s class tested strength—all other classes put the weights in a cup in the middle of the span, but her class added weights directly on the deck, and students could place them anywhere, leading to inconsistent measures of strength. If one weight slid off the side, it was declared a failure, and this was the case in the following transcript (Table 5.8)
Table 5.8– Ms. Clay reacts as a strategic partner – From transcript T14098BrL42d3T2B
Time Line Liam Ms. Clay Context Clues 23:01 1 Ok, so you’ve done your third testing, Tahlia cleans up weights 2 correct? 3 Yeah 4 Ok, what I want you to think about, is Libby and Tahlia keep testing with 5 “barge” 6 Your lowest score was the weight, 7 right? 8 Yeah Liam, Tahlia, and Libby continue to 9 work on bridge 10 So I want you to think about how you 11 could’ve made it stronger 12 What could you have done to make it No reply to first question 13 stronger? How could you reinforce it? 14 Added more columns 15 Um, the columns didn’t seem to make Acknowledges Liam’s raised hand 16 the difference. The columns don’t seem 17 to be, I mean you put a lot of columns in 18 there, but what else could you have 19 done? Yes? 20 We could have put more support where Uses hand gestures to show he meant 21 the columns are gonna hit under the decking 22 They are underneath, right? 23 We put a couple straws underneath I think that if I… 24 Well, what’s the problem? What would Hand motions back and forth signaling 25 you have used? What do we know about the answer she’s looking for that 26 spans? What do you want to have across support goes across the whole span 27 the span 28 Um, the strongest material. 29 What would the strongest material be? 30 What’s it called, uh, craft sticks. 95
31 Or the straw. Either or. But what’s Girls talking to each other recording 32 wrong with what you’ve done? results in their journal. Mrs. Clay only 33 Something’s wrong. What were you talking to Liam, points to bridge 34 thinkin’ when you were doing this? 35 What is it with the weight? What was it 36 that needed to be underneath there? 37 What’s gonna help the span?
38 What would you have done with the Rephrases question when Liam doesn’t 39 craft sticks? respond 40 Taped it, under the… 41 Where? Where under the paper, explain Gestures to Liam to respond 42 to me. How would it have been 43 stronger? What would have made that 44 stronger? How did Javier make it 45 stronger? What did he put across? Yes 46 He put across rope 47 Underneath? And how did he, what did 48 he do with that rope? Did it just go in 49 the middle, did it go at the end? Did it 50 go the whole way? What did it do? 51 The whole length 52 Ok, so look at your design, what do you 53 suppose you could have improved it by 54 with that thought in mind? 55 Two straws on each side, and string Mrs. Clay nods 56 through the straws 57 Yes, you could have done, excellent. So 58 when you redo your thinking, those are 25:28 59 the kinds of things I’m looking for Ms. Clay spends two and a half minutes speaking with the group about why their strength score was low. She asks several leading questions, and references the storybook from the beginning of the unit (Lines 44-50) often without allowing enough time for Liam or anyone else in the group to respond. I consider this style of interaction to be attributional feedback (Le Foll,
Rascle, and Higgins, 2007), because her intention seems to be to focus their attention on the reason why their strength score was low. To characterize this attribution in Weiner’s framework
(1985), the failure was due to an external, specific, and changeable feature, and her goal was to help them realize this.
One version of the strategic partner response was unexpected. All of the teachers except
Ms. James were observed providing the strategic partner response pre-emptively. That is, they talked to the class about how to observe or respond to a failure that might happen during their design. An interesting example of this was in Ms. Thomas’ class. She added an additional day onto Lesson 3.1 for a whole-class comparison of the strength of beam, deep beam, and arch 96 bridge models. Even though each group had tested the bridge types and compared data, Ms.
Thomas wanted to ensure that the students realized that the bridges responded to a load in the same way, even when the span increases, so she recreated the test with poster board (rather than index cards) and a larger span. During the demonstration, she directs the class’ attention to the way the deck and abutments react as the load increases (see Transcript T14142BrL32D2F).
Table 5.9– Ms. Thomas acts as a strategic partner pre-emptively
Time Line Class Ms. Thomas Context Clues 27:01 1 Ok, which way did the index cards Class sitting on floor, Mrs. Thomas at table with 2 move when you added weights, beam bridge 3 Phillip? Points to Philip 4 Philip: Down 5 Ok, here we go. So here’s my cup in Places cup for weights on the bridge and gets 6 the middle weights 7 One, two, three, four. Do you see it To class 8 change? 9 Yes↑ 10 So the weight I’m putting on is doing 11 what? What kind of force is it adding? Points to Carlos 12 Carlos-push Single student answers when called on 13 Keeps adding weights until it collapses 14 Ok, there we go As it collapses 15 Ok, so that was one, two, one, two, 16 three, four, five, six, seven, and the last 17 one was the eighth, so it supported 18 what? 19 Seven 20 Seven. Did it support it in a strong 21 way? That if you were this person you 22 would want to walk on this beam 23 bridge? D.J. can you fill out the beam DJ gets up to complete the data sheet on the smart 24 bridge on the board for us? board 25 So when I add these weights, the force 26 that was added to the beam what a 27 what? 28 D.J.-A push 29 Yes, a push down. Um, did the index Air quotes 30 card or poster board move the same 31 way it did when you tested it? It did, Class nods in agreement 28:29 32 didn’t it? All right.
Ms. Thomas repeated this same process with the deep beam and arch bridge, and extended the duration of this unit by almost a whole class period. This indicates she felt the students might struggle with the transition from testing in small scale (Lesson Three) to the design challenge (Lesson Four). 97
Figure 5.2 (pg. 97) shows a distribution of the reaction types across the four teachers in
this part of the study. Given that the duration of these units is approximately eight hours, the
number of reactions was surprisingly few. However, the teachers spent little of the design and
construction phases providing feedback to small groups because they were busy at the materials
“store” providing students with requested supplies. In fact, two teachers (Ms. Thomas and Ms.
James) that did not participate in all the public tests, decreasing the number of teacher reactions.
This is likely because they were busy trying to keep all the students on task and productive, but
fewer teacher reactions were coded than I expected.
Again, it is not my intention to categorize teachers, but to categorize their responses. I
also attempted to investigate a connection between the failure type and reaction type, which I
describe after the section “Causes of Failure.”
12
11
10
9
8
7
6
5
4 Number Incidences of Coded Number Incidences of Coded 3
2
1
0 Ms. Thomas Ms. Maddux Ms. James Ms. Clay Teacher Pseudonym
Manager Cheerleader Strategic Partner
Figure 5.2 – Descriptive statistics of teacher reaction types by class 98
Summary of Teacher Reactions. So far, I have characterized both the types of failures that can occur in elementary engineering design projects, and have also described three ways in which teachers respond when students’ designs fail. It is also plausible for teachers to use multiple reactions because they attend to multiple role in sequence. For example, I can imagine a teacher reacting first as a cheerleader to lessen the disappointment of a failed high-stakes, unintended failure, then transitioning into a strategic partner to help students’ learn from the failure. However, the reactions types are also likely to depend on another factor—the causes for the failure. In the next section, I consider the reasons for failure to better understand the complex interplay of failure and subsequent improvement.
Causes of Failure
Next, I coded for the reasons why students’ designs failed with four categories: 1) lack of knowledge of science and/or technology, 2) lack of knowledge of material properties, 3) poor craftsmanship, and 4) limitation of materials. To do this, I coded using the same constructs developed in Chapter Four (Table 4.9, pg. 62)
Knowledge of science/technology. When students design a solution without understanding the mechanics of it, the ensuing failure is caused by the lack of understanding of science/technology. For example, Mark, Lexi and Rose attempt to construct a truss bridge in
Table 5.10 and Figure 5.3: 99
Figure 5.3– Mark, Lexi, and Rose attempt to build a truss bridge
Table 5.10– Lack of understanding of Science/Technology
Time Line Mark Lexi Rose Context clues 47:26 1 How are we supposed Rose holds up truss 2 to keep this up? 3 Uh Lexi puts her hand on 4 truss and helps hold it 5 up 5 Yeah, cuz they’re Mark puts his hand on 6 gonna the truss to hold it up 7 Yeah, we could 8 probably use a 9 popsicle stick to keep 10 it up 11 Ok Rose leaves to get a 47:37 12 popsicle stick Truss bridges work to prevent a deck from deforming by a combination of components that simultaneously resist tension forces (pulling apart) and compressive forces (squeezing together). This group has clearly seen this bridge design before, but they do not understand how trusses work in order to support weight. As a result, they decide to prop up the truss with a craft stick, resulting only in an increased load on the bridge that must be supported by the piers. 100
The first table group from Ms. Maddux’s class demonstrated a variation of this failure cause when they attempted to design an arch bridge (T14143BrL42D1T2A). The increased span from Lesson Three to Lesson Four caused several groups to struggle with incorporating the arch design (the strongest among those designs tested). In Lesson Three, one index card fit perfectly between the abutments such that the top of the arch was level with the abutments. To achieve a
Figure 5.4 - A schematic of a double arch bridge Figure 5.5 - A double-arch bridge similar result, this group used two arches. Figure 5.4 is diagram demonstrating the misunderstanding and Figure 5.5 is a screenshot from the video.
An arch works by transferring the force from the applied load to abutments or piers that are well connected to the ground. In this case, the students had to test their bridge strength by adding weights into a cup directly in the center of the span. As you can see in Figure 5.5, the load is directly between the arches, negating some of the strength provided by the arch. A group from Ms. Thomas’ class avoided this issue by using three arches so the load was applied directly onto the middle arch. Alternatively, they could have anchored the deck to the piers to use tension to transfer the load to the piers rather than the arches.
Knowledge of materials . The transcript in Table 5.11 illustrates a group that did not understand the material properties of the copy paper they used as a deck in their bridge design.
The group’s strategy for design and for improvement relied on increasing the number of piers, 101
Table 5.11– Lack of knowledge of material properties – From transcript T14098BrL42T2
Time Line Liam Ms. Clay Context Clues 23:53 24 Well, what’s the problem? Hand motions back and forth 25 What would you have used? signaling the answer she’s 26 What do we know about spans? looking for that support goes 27 What do you want to have across the whole span 28 across the span? 29 Um, the strongest material. 30 What would the strongest 31 material be? 24:08 32 What’s it called, uh, craft 33 sticks. but they would have greatly benefited from a more rigid material for the decking, or by using multiple layers of paper and/or index cards. Table 5.11 is an excerpt from Table 5.8 (pg. 94).
Group Two in Ms. Maddux’s room also tried to use an arch bridge. However, they used one arch made of copy paper, which was very weak. Figure 5.6 is a screenshot of this design, and it shows the arch sagging when the cup (with no weights) is first placed on the deck (line 63,
T14143BrL42D1T2). Just prior to this public testing, Robyn notices the sagging arch (line 59,
T14143BrL42D1T2) and pitches the idea of adding an arch. It is not taken up by the group; thus the group’s design failed, in part, due to a collective misunderstanding of material properties.
Poor craftsmanship. Poor craftsmanship is another common cause for failure that is likely to be common in all elementary classrooms. Physical Figure 5.6– Lack of understanding of material dexterity capable of transforming an idea into reality properties is non-trivial, and it often plays a role in causing failure. Ronnie’s group’s design technically
fails before it is even tested. During the strength test, the
failure point was established to be when the deck falls
below the top block of the abutments. In Figure 5.7, notice
Figure 5.7 – Poor craftsmanship that the left side of the deep beam deck falls below that 102 point. In Table 5.12, a brief interaction makes this point relevant in a classroom public testing environment. Zack points out that technically the bridge failed before it was tested. Ronnie dismisses him (successfully), although it could have easily been determined by the group that the testing should not continue until the group fixed their design.
Table 5.12– Poor craftsmanship – From Transcript 14142BrL42d1T1
Time Line Zack Ronnie Context Clues 25:35 1 Yeah, but it’s already on the Points to an area sagging below 2 second line there. See? Look at the agreed upon “failure point” 3 that bend in there 4 Dude, it was already like that Speaks softly, doesn’t look at 5 Zack, puts cup on the bridge to 25:50 6 proceed with test Another common occurrence leading to failure caused by poor craftsmanship was students’ cutting piers without measuring their length. None of the students used a ruler to measure the height of the abutments and cut the piers all to the same length. They all cut either without any measurement or they “eyeballed” it, often requiring them to recut until the pier was either the right length, or as in the case with Riley, who cut the pier too short, they needed to make another pier (Table 5.13).
Table 5.13– Poor Craftsmanship – From Transcript T14098BrL42T2
Time Line Liam Libby Riley Context Clues 26:25 1 We need a whole new piece of Crumples a pier in her 2 paper hand 3 Dang it. Huh? What? Both look up from the 4 bridge 5 A whole new piece of paper. I Picks up other piers and 6 screwed up. I screwed up on all drops them on the desk. 7 of them and made them too 26:35 8 small. Limitation of Materials. The last category of failure causes is the limitation of the materials. In other words, the design is based on a strategy that takes into account scientific concepts and the best available materials were used. This failure cause is only associated with intended failures, such as loading weight onto a structure until it collapses. This code was used often in the EiE: Bridges classrooms because Lesson 3.1 involved comparing strengths of bridge designs in small scale by adding weight until they collapse. Table 5.14 is a brief interaction 103 where two pairs of students are testing the beam bridge design.
Table 5.14– Limitation of Materials
Time Line Louisa Ronnie Rebecca Context Clues 56:02 1 We got five Both groups’ beam 2 bridges fail at the same 3 time 4 We got three 5 No, we got, we got four She realizes it collapsed 6 on the fifth weight, 7 meaning it held four. 8 Let’s try that again To Lenny 9 Yeah, let’s try that To Rebecca 10 again 11 Do we try again, or not? To all students at the 12 Are we supposed to try it table. Both groups 13 again? proceed with another 56:22 14 trial
Students did not have a say in the design of these prototypes, and they were to be tested until they failed. The strength of each bridge type should be very similar to other groups in the class. The two groups in Table 5.14 appear to recognize that they should get the same critical load for the bridge, so when their results differ they instinctively think to retest and verify their measurement. In the final design challenge, a perfect score for strength was achieved if the bridge could hold one hundred weights. Figure 5.9 (pg.104) shows a design from the second table group in Ms. Maddux’s room. They reinforced their deck with drinking straws to make it rigid, and used piers of craft sticks to balance the force applied by the weights. It never fully
collapsed, but it shifted enough to cause them to stop testing
at 147 weights.
Figure 5.9 is a stacked bar graph with descriptive
statistics showing the relative number of causes of failure
coded within the four classes using EiE: Bridges. To
summarize, the majority of failures from misunderstanding
science concepts were related to students failing to use piers Figure 5.8 - Limitation of materials 104 or other supportive structures or for not constructing a level deck (which often caused weights to fall off or the car to drive off the side). The majority of the failure due to misunderstanding material properties were related to rigidity. Students sometimes did not realize the structural weakness of copy paper and used single sheets for arches and/or decks. Poor craftsmanship was typically noted in failure to measure properly, and failures due to limitation of materials were exclusively in intended failures.
Figure 5.9 - Causes of failure by class Summary of the causes of failure. As I will discuss in further depth in Chapter Seven, the combination of failure type, the cause of the failure and teacher reaction to it are all important in considering the classroom discourse that surrounds the failure and attempts at improvement.
For researchers, these categories should help to understand the context of why and how the designs may have failed. For teachers, a better understanding of the nature of the failure will better enable her to engage the students in discussions or activities to help them in learning about engineering and toward improvement in their design. 105
Obstacles to Improvement
Improvement from failure is non-trivial. The next portion of this analysis uses the same constructs developed in Chapter Four and applies them to the data from Chapter Five. While all components of this complex theoretical system are important to consider, it is likely that this is the most important because without an understanding of the conditions necessary for improvement, it is less likely students will be supported by teachers and/or curricula in improving.
Lack of opportunity. In Chapter Four, part of the improvement model (Figure 4.4, pg.
74Error! Bookmark not defined.) considers obstacles to improvement. One of these obstacles is that often students lack the opportunity to improve after failure. This was not as much of an obstacle in these data. All classes were given an opportunity to construct for at least two hours, including at least one improvement cycle. Ms. Maddux and Ms. Clay allowed their students to improve their initial design twice, and Ms. Thomas allowed for one formal testing and redesign.
Ms. James instructed her students to build, test, and improve over two class periods, but did not hold any public (high-stakes) tests. Her class was the only instances of lack of opportunity to improve I observed in these data. Although she directed them to test and improve, the two groups
I observed did not take up those instructions.
Unfair comparisons. The lack of fair comparisons did present itself as an obstacle for improvement in some cases. In Ms. Maddux’s class, they did not use cost as a criterion for evaluating a design, even though they calculated costs. In the second public test, T2’s design earned perfect scores for stability and strength. Rather than improving their bridge by making is cost less, they added more materials to it to make it stronger (Transcript T14143BrL42d2T2).
Similarly, Ms. Thomas’ class did not use cost as a criterion for evaluation. Leo and Lola 106 designed a bridge that had several craft sticks, three arches, and many index cards. It was successful in strength and stability, but cost 16 dollars. According to EiE’s scoring system, any bridge costing over 10 dollars earned the lowest score. Rather than improving on cost, they bought more materials and made it even stronger. This contradicts an important practice in engineering, attending to multiple criteria. Many professional civil engineers are employed by construction companies primarily to take plans from engineering design firms and work with them to decrease costs while maintaining their strength, stability, and aesthetics.
Unproductive strategies. Only one of the groups in this treatment group had both the opportunity for improvement and a fair comparison, but whose ineffective strategy prevented improvement. The “T1” group in Ms. Thomas’ class seemed not to recognize the fact that the piers that supported their deep beam bridge design were unevenly cut and unevenly spaced (i.e., caused by poor craftsmanship). In their journal, they wrote they planned to make their bridge
“lower and stronger” (pg. 34, journals 75993 & 75998). By adding additional piers that were unevenly spaced and cut, the result was an arch too steep for a car to pass. Student 75993 wrote,
“There was a hill on both sides so the car couldn’t go up” (journal pg. 38). The changes also caused the strength to decrease from a score of two to a score of one (28 weights to 18 weights).
The other group whose strategy proved ineffective in leading to improved scores was
“T2” from Ms. Clay’s class. As described above, Ms. Clay suggested the students to place the weights directly on the deck instead of into a cup in the center of the span. This caused an uneven comparison in strength from one prototype to the next, but the students also misappropriated the cause of failure to the lack of piers rather than the pliable deck. The group attempted to add extra supports (pg. 34, journal 80958), but their scores for both strength and stability decreased (pg. 46, journal 80958). 107
Summary of obstacles to improvement. Despite efforts by EiE to include improving from failure as a key feature of their unit, EiE: Bridges, many of the student groups I studied were unable to improve their designs in their formal evaluations. Some of them were constrained by the manner in which the teacher directed the activity (Table 5.18), and some of them failed to recognize weaknesses in their design, and chose ineffective strategies (Tables 5.17 and 5.18). In contrast with the control curriculum, three of the eight groups did systematically improve their final designs after a phase of redesign.
Patterns of interaction between constructs
I attempted to determine if there was a pattern between the failure causes, failure type, and ways in which teachers reacted. To do this, I counted every teacher reaction code and classified it based on the failure type and failure cause. In situations with multiple codes (i.e., the failure was caused by both poor craftsmanship and a lack of understanding of materials), I counted it as two reactions. This table can be found in Appendix C. This analysis showed no discernable pattern of the ways these three constructs interact. I expected, for example, a distinct pattern of cheerleader responses to high stakes, unintended failures because teachers might be tempted to try to prevent students from being disappointed by the failure. However, this type of pattern was not found in these data, and may be very difficult to find in any of these classes due to the complexity of the teacher’s perceived role, and the circumstances of the failed designs.
However, as I will discuss in Chapter Seven, students’ learning from failure might be bolstered by a better understanding of these constructs by the curriculum developers, professional development providers, and teachers.
Before moving on to an analysis of groups that achieved improvement through systematically redesigning bridges using productive strategies, Figure 4.4 (pg. 74) is a 108 representation of the model for failure and improvement that considers failure type, cause, and obstacles to improvement. Applying it to the EiE curriculum did not require any modifications.
Systematic Improvement
The purpose of this section is to make some claims about improvement in classroom engineer design projects. Tables 5.16 and 5.17 (pgs. 112 & 113) summarize the attempts by the eight groups to construct, test, and improve bridges that were evaluated (and thus could fail) in any of four criteria. Spans had to be 15 inches, a “barge” had to be able to pass underneath the deck, and they had to be stable (tested by a toy car being able to pass across the deck) and strong
(tested by adding weights in a cup in the middle of the span until the structure collapsed). For each group in the Tables 5.16 and 5.17, I listed the instances of tools they used for low stakes testing and examples of improvement I observed. Based on the obstacles to improvement from the model (Figure 4.4, pg.74), I cite evidence that teachers provided both the opportunity for improvement and conducted tests that allowed a fair comparison for gauging improvement. I also recorded strategies students employed for improvement and evidence from video data and/or student journals of improvement.
All groups I observed improved in some ways throughout the course of their design, but not all improved in the high stakes evaluations or public tests, which are the scores reported in the student journals. Each of these student groups used tools to test these criteria during their bridge design and improvement phases. Some groups frequently used the “barge” to test their bridge any time they altered the piers or arch. Others used the toy car, both as a weight and as a test of the bridge’s stability. Some groups also tested the bridges’ strength using school supplies like rulers or with water bottles they had at their desk as weights. 109
Although there was evidence of improvement for every student group, only four of the eight groups demonstrated increases in their design’s final scores. This can be attributed to the collective actions of the students and the teachers. Opportunity and fair comparison are primarily the responsibility of the curriculum developers and the teachers. The third category, productive strategies, falls largely on the student groups and discursive work of the teachers.
Opportunity. The EiE: Bridges unit provides for at least one cycle of improvement. Ms.
Maddux and Ms. Clay allowed their students two improvement cycles, while Ms. Thomas only permitted one improvement. However, the groups in Ms. James’ class misunderstood the directions they were given. Ms. James told the class to test their structures and then improve them during the second day of Lesson 4.2 (lines 67-69, T14164BrL42T1), but neither of the groups in this class formally tested their bridges, and there was no structured public testing— and therefore no examples of high stakes failures in her class (Figure 5.1, pg. 87). Although both student groups could be observed making improvements from low stakes failures, they lacked the opportunity to improve from high stakes evaluations of their designs (Table 5.16, pg.113).
Ms. Clay allowed for two cycles of improvement, but did not provide the opportunity to improve on their cost score. She characterized the improvements as additions to the existing structure that they had already paid for [Line 143, T14098BrL42T2] rather than allowing students to improve by more efficient use of materials and designs.
Fair comparisons. The EiE teacher guide has specific directions for testing and scoring of the bridges according to a rubric. Each bridge is scored on cost, strength, and stability. In order to improve in a particular criterion, the comparison before and after redesign must be evaluated in the same way. A specific example from these data that demonstrably impeded improvement was the method used to test strength of the bridges in Ms. Clay’s class. Strength 110 testing in the other three EiE classes used a plastic cup placed in the middle of the span that was filled with weights one by one until the structure collapsed. However, Ms. Clay’s class was encouraged to place the individual weights directly on the span, and to place them “strategically” to get a higher score in strength [lines 172-173, T14098BrL42T1]. In both groups, failure was declared when a weight (or weights) fell off the side of the bridge [line 173, T14098BrL42T1; line 136, T14098BrL42T2]. This is actually a measure of stability rather than strength, and it also prevents a fair comparison of one design to the next due to this testing procedure. It is important to note in Ms. Clay’s class in Table 5.16, that although there was not a fair comparison for the strength test, and not an opportunity for improvement in cost, group “T1” improved in stability because there was a fair comparison, they did have the opportunity, and they had an effective strategy. The rest of the teachers in this treatment group allowed for fair comparisons for all the criteria by following the EiE directions for bridge scoring (see Tables 5.16 and 5.17).
Productive strategies . Table 5.17 shows that due to the collective actions of teachers and students, three groups systematically improved by using several iterations of low stakes tests and improvements, fair comparisons, productive strategies, and the opportunity to improve their prototypes. The case of “T2” in Ms. Maddux’ class was particularly interesting (Table 5.17).
Their initial prototype only held two weights (line 65, T14143BrL42T2), so they aimed to improve the strength score by “putting straws on the bottom” (pg. 29, journal 76571). They also included piers that used craft sticks to reinforce the rigid deck (line 102, T14143BrL42T2). The resulting design yielded a perfect score of four for stability and five for strength. Perhaps more interestingly, the students seemed to work on, test, and improve on criteria serially in this order
(Table 5.15): width of span, space for the barge to pass, a deck that can support the car’s weight, a deck stable enough for the car to pass, and then a deck can bear weight. This activity can be 111 represented with sequence units from my event map (T14144BrL42T2):
Table 5.15 – Sequential analysis and changes aimed at improvement in T2 (T14143BrL42T2)
Phase Build and Test Bridge Designs Sequence First design- Test Test car Improve arch and Calculate Test with Add Test-car Ends (32:01) arch Barge as weight then cost car as piers driving (55:30) redesign weight This pattern is likely to occur in other situations and was observed in several situations in these data. It makes sense for the students to systematically increase their complexity from relatively easy and static measures of criterion (e.g., that the span is 15 inches), and work on increasingly complex criteria. For example, I noted the use of the car as a test of weight in multiple groups. It stands to reason that if the bridge cannot support the car’s weight, it would not be able to score well on the stability test! 112
Table 5.16 Improvement in Ms. Thomas’ and Ms. Maddux’ Classes
Teacher Group Tools used for Low Low Stakes Fair Comparison Opportunity Strategies for High Stakes High Stakes Number Stakes Testing Improvement Improvement Improvement
Car [lines 30, 34, 41, 42, Yes – stability improved Yes – classes tested as Yes – groups were given Ineffective – “We will try No – Scores for both 75, 115, 118, 120, 142, from line 37-41 described in EiE teacher the opportunity to to make the bridge lower strength and stability 148] Barge – [lines 33, T14142BrL42D1T1] manual [lines 83-88; 152- redesign once [lines 88- and stronger” [pg 34, decreased. [pg 46 Journal 48, 140, 155,T14142BrL42D1T1] 152, journals 75993 and for S75993] T14142BrL42D1T1] T14142BrL42D1T1] 75998]. They did not Table 1 recognize their piers were uneven and the deck was too arched to allow the car to pass [T14142BrL42D1T1] Ms. Thomas Car – [lines 34, 40, 41, Yes - Leo recognizes the Yes – classes tested as Yes – groups were given Effective – “[our score Yes – Scores for both 49, 52, 56, 60, 145, 146] pier is too long and cuts described in EiE teacher the opportunity to was highest in strength] strength and stability Barge- [lines 18, 40, 52, it. Barge can still pass. manual [lines 64-66 redesign once [lines 67- because of the column in increased due to 60, 64, 112, 120, 144, Car can pass because it T14142BrL42D1T2] 163, the middle” [pg 38, increasing the number of Table 2 T14142BrL42D1T2] is flat and has three piers T14142BrL42D1T2] Journal for S76002]; piers [pg 46 Journal for [lines 40-41, “Bridge still inverted S76002] T14142BrL42D1T2] showing 4 unevenly placed piers” [line 120, T14142BrL42D1T2]
Ms. Table 1 Car – [lines 95, 11] Yes – “Farrah cuts the Yes – Ms. Maddux conducts Yes – Ms. Maddux gave Effective – “they use Yes – their strength score Maddux arches to make them three public tests according to groups an opportunity to drinking straws to increased from 1-2 [pg Barge – [lines 29, 30, shorter….she re-installs the EiE teacher manual [lines improve twice [lines 70- strengthen the deck” [line 46, Journal 76571]; It 48, 52, 54, 67, 96, 111, and it works [line 54, 67-69; 112-113; 157-160 111; 130-154 137]; “we will put straws went from holding 8 153 T14143BrL42T1] T14143BrL42T1] T14143BrL42T1] T14143BrL42T1] on the bottom”[pg weights [line 69] to 46 39,journal 76571] [line160, T14143BrL42T1]
Table 2 Car – [lines 39, 40, 43, Yes – “They put 4 Yes – Ms. Maddux conducts Yes – Ms. Maddux gave Effective – They added Yes – Their bridge went 48, 50, 51, 68, 110, 130, straws under the arch as three public tests according to groups an opportunity to straws to reinforce the from holding only 2 171] Barge – [lines 37, piers. G tests their new the EiE teacher manual [lines improve [lines 66-131 deck [lines90-96 weights [line 65] to over 96, 103, 109, 130] design with the car as a 64-65; 131-135; 173-177 T14143BrL42T2] T14143BrL42T2]; “[We 100 [line 135, Weights – [lines 107, weight and it holds. T14143BrL42T2] will improve strength] by T14143BrL42T2] 108, 110, 130, 171] Someone remarks, "We putting straws on it” [pg got it!" [line 48 34, Journal 76560] T14143BrL42T2]
113
Table 5.17– Improvement in Ms. Clay’s and Ms. James’ Classes
Teacher Table Tools used for Low Low Stakes Fair Comparison Opportunity Strategies for High High Stakes Stakes Testing Improvement Stakes Improvement Improvement
Ms. Clay Table 1 Barge- [lines 34, 145, Yes – Group tests load No – Ms. Clay allows them Yes – Ms. Clay Effective – [to improve Yes – Stability increased 166, 293] Ruler [lines bearing by using to “strategically” place allowed the group to stability] “We will from a score of 2 to a 19, 20, 84, 87, 95, 102, someone’s hand, then weights on the deck and improve their design flatten our paper so the score of 4. [pg 46, 112, 270, 277] Hand – they add a pier and test weights that fall off are twice [line 175-295; car goes over” [pg 34, Journal 80961] [line 80, 117, 123, 125, again and are satisfied considered failure in strength 298-364] No: does not Journal 80961] 293, 295, 362 [lines 125-129, [lines 172-173; 298; 366 allow improvement on T14098BrL42T1] T14908BrL42T1] costs [line143, T14098BrL42T1]
Table 2 Barge [lines 29, 56, 60, Yes – Low stakes No – Ms. Clay allows them Yes – Ms. Clay Ineffective – [to improve No – the strength and 125, 157]; Ruler [line testing with ruler [as to “strategically” place allowed the group to strength] “We will add stability scores remained 127, 128]; Notebook weight] Riley is weights on the deck and improve their design extra supports” [pg 34, the same, and the cost [line 128]; Water bottle surprised it holds [line weights that fall off are twice [line 66-134; Journal 80958] score decreased [pg 46, [line 134] Weights 127, T14098BrL42T2] considered failure in strength 140-159] No: does not Journal 80958] [line 169, [lines 65, 136, 162, allow improvement on T14098BrL42T2] T14098BrL42T2] costs [line143 T14098BrL42T2]
Ms. James Table 1 Car – [lines 38, 43, 63, Yes – Car able to pass Yes – Ms. James describes No – Groups never No opportunity No – No opportunity 77, 81, 82, 89, 105, over after improving EiE testing procedures to the formally tested their 114, 119] Barge – piers [lines105-115, class [lines 7-8, bridges – “[strength [lines 12, 36, 78, 83, T14164BrL42T1] T14164BrL42T1] was our lowest score] 92, T14164BrL42T1] because we never got to test it [pg 33, Journal S74850]
Table 2 Car – [lines 48, 49, 77, Yes – “… although it Yes – Ms. James describes No – Groups never No opportunity No – No opportunity 79, 90, 92, 93, 94, 98, sags, it can hold the EiE testing procedures to the formally tested their 100, 102, 105, 116,] car” [line 90, class [lines 7-8, bridges, so they never Barge – [lines 73, 91. T14164BrL42T2] T14164BrL42T1] were able to improve 97, 99 [T14164BrL42T2] T14164BrL42T2] 114
Summary of Systematic Improvement. Through this data analysis, I developed a conceptual framework of a progression of steps involved in systematic improvement after failure
(Figure 5.10). Although I consider this a typical sequence, I acknowledge the sequence may differ depending on the context. I propose this model to consider student actions and potential opportunities for helpful teacher feedback, and in the next section of this chapter I use a rubric based on this framework to evaluate learning that occurs during improvement from failure.
Upon observing (and socioculturally acknowledging) failure, groups must attribute the cause of failure to some specific, changeable aspect of the design (Weiner, 1985). This
Figure 5.10– Systematic Improvement attribution could come from data collected (measured or observational) at the point of failure or might require an investigation (e.g., Rodgers Commission, 1986). The group must next decide to improve one or more criteria that failed to achieve success and then use a strategy that is coherent with both the chosen criteria and the attribution. For a more concrete example, consider a hypothetical bridge design from this unit. The bridge tipped every time the toy car was driven across, leading to a stability score of zero. The attribution might be “the deck was not stable because it only had one pier in the middle, causing the car to drive off the side.” The group would then choose to improve the stability of the deck, and would increase rigidity by increasing the number of layers of index cards and use a series of drinking straws as a frame. Very likely this series of events would lead to improvement. A counterexample that might be useful to consider is improvement that occurs due to luck, or a combination of other choices that would 115 not be considered systematic or following any logic. For example, a group whose bridge only held 4 weights that chose to improve cost would not be following a logical process (Figure 5.10).
Summary: Analysis of Classroom Video
By systematically analyzing classroom video of sixteen student groups from eight classrooms engaged in one of two civil engineering curriculum, I derived a model for understanding the complex phenomenon of failure. I felt it was important to carefully consider the various types, causes, and teacher reactions to failure, and to compare student groups that systematically improve after failure to groups that do not. Then, I planned to uncover patterns of how these constructs interact. For example, I expected to see more cheerleader reactions to high stakes, intended failures. I was unable to detect such patterns, likely due to the small sample size; but, understanding these features may help both curriculum developers and teachers to better facilitate improvement after failure because failure does not necessarily lead to improvement and students will need to be supported as they redesign.
Analysis of classroom discourse using video evidence is time intensive, expensive, and not plausible in most situations; however, better understanding the learning students experience during the process of the activity is important. For that reason, I then set out to determine if student learning from failure could be observed in the things students write about in their engineering journals. The next section of my analysis uses the framework for systematic improvement (Figure 5.10, pg.114), and applied it to the EiE: Bridges student journals collected as a part of the research study by Museum of Science, Boston, to assess student learning during the process of improving after a failed design.
Analysis of Student Journals
116
One way to measure the effectiveness of curricular interventions is to administer pre- and post-tests to measure increases in test scores and compare results from randomly assigned control and experimental groups. However, attempting to find causal relationships between interventions in education can be challenging (for example, see Maxwell, 2004). In fact, the data
I studied were collected during a study comparing a control curriculum (E4C) and the experimental curriculum (EiE), and data collected from pre- and post-tests is being analyzed for statistically significant differences in learning that can be ascribed to the intervention. However, we might consider other ways of measuring learning in elementary engineering design projects.
This content analysis follows the recommendations of Bazerman (pg. 79, 2006).
I set out to determine the feasibility of assessing learning through analysis of student engineering journals. For this study, I developed a rubric for scoring the learning that occurs during systematic improvement from failure (Figure 5.10, pg.114), using the five elements of this progression to determine a score (Appendix D).
In this section, I describe this scoring system, and demonstrate that it is reasonable to infer learning that occurred in group work through analysis of these student engineering journals.
To do this, I must first demonstrate that there is a correlation between the documentation of events in the journals with the observable events from the digital video recordings, with a greater level of detail then was done in Chapter Four (Table 4.17, pg. 78). Second, I must show that the rubric is able to be used by multiple scorers and scores would be comparable. Last, I must show that there is not an inadvertent observer effect (Brown, 1992). I do this by comparing journals from student both on and off camera. Analysis of both student video recordings, student journals, and follow-up interviews would yield much more accurate data, but this type of analysis is both cost and time prohibitive.
117
Comparison of Video data to Journals
The rubric used for scoring student journals is based on Figure 5.10 (pg. 114) and can be seen in its entirety in Appendix B. Briefly, the scorer looks for evidence of the following features of improvement: 1) acknowledgement of failure; 2) attribution of failure; 3) choosing to improve a criterion that failed; 4) choosing a strategy aimed at improving the criterion while addressing the attributed shortcoming, and 5) evidence of an improved design. Each cycle of improvement is evaluated on a scale from one to five. Zero signifies there was no failure, and thus no improvement. To score the journals, I first looked for evidence of acknowledging failure. This
5
4
3
2 Journal Score
1 0 0 0 42T1 42T2 43T1 43T1B 43T2 43T2B 64T1 64T2 98T1 98T1B 98T2 98T2B Group Number
Figure 5.11– Group Journal Scores by Improvement Cycle could be found in the text of the journals, or in the admission of scores lower than 4 or 5 in stability and strength, respectively. Journals with evidence supporting this experience of failure received one point. Next, I looked for evidence of an attribution of failure (why the bridge failed). I then looked for answers to questions on page 34 of their journal that showed evidence
118 that the group aimed to improve at least one criterion that failed, and that it was aligned with the attribution (if one was cited). I then looked for evidence of a strategy coherent with the failure, attribution, and criterion, then looked for evidence of improvement in the criterion the students tried to improve. In classes that had two cycles of improvement, I scored each iteration as a separate event. The results can be found in Figure 5.11. The category in which students rarely provided enough evidence to earn a point was in attributing failure. This is likely in part caused by the wording of the question, “Which of your scores was the lowest? Why do you think so?”
(pg. 33, emphasis in original). While this question apparently seeks a cause for the low score, a majority of students answered this question literally. Many students answered that their strength was the lowest because it got the lowest score, or as student 76560 wrote, “I think [strength was our lowest score] because we only got two weights on.” (pg. 33). I chose to use the highest journal score per category per group because learning is a sociocultural process and these groups worked together. While I anticipate some contention with this decision because it does not evaluate students individually, I respond to that objection with the fact that some students may have done more of the constructing and thus may have less detailed answers in their journals despite being equally intellectually engaged in the engineering design.
I then scored the event maps using the same rubric I used for the journals. It is possible students may not write in their journal about a rich conversation about the strategy they chose, but it is equally possible they might write about the attribution of failure in strength but not say anything on the video because the cause was so obvious. Figure 5.12 (pg. 119) compares the scores earned from event map scoring to the collective high score of each group’s journal.
119
5
4
3 Score 2
1
0 0 0 0 0 0 42T1 42T2 43T1 43T1B 43T2 43T2B 64T1 64T2 98T1 98T1B 98T2 98T2B Group Number
Group Score Video Score
Figure 5.12– Comparison of improvement scores of journals and video event maps *B indicates a group's second improvement cycle In ten of the 12 cycles evaluated, the scores were within one point. For finer grained
analyses of the differences, refer to Appendix B. Of this sample, the two groups in which there
were greater than one point differences were 42T2 and 98T2 (second improvement cycle).
However, in each of these cases, the journal score earned a higher score than the classroom video
analysis, suggesting video (or participant observation) evidence alone is less sensitive measuring
these factors. One explanation that could account for this difference in 42T2 is that one student
in the pair got sick and went to the nurse’s office (line 107, T14142BrL42d1T2), so there was
little reason for Leo to verbalize his thoughts about attribution, goals, or strategies. Only two of
these improvement cycles earned a higher score in the video analysis, (42T1 and 98T2),
120 suggesting that in the absence of access to both video and journal evidence, journals may provide more evidence of learning from the improvement process than from video analysis alone.
Despite the small sample size, I compared the improvement scores from the journals and event maps using a paired t-test. The mean of the scores on the event maps (M=2.92, SD=1.78) was not significantly different from the scored journals (M=2.58, SD=1.83, p=0.266).
This comparison is bolstered by the agreement that exists within the detailed comparison
(Appendix B). Even with a small sample size, and the fact that in all but two of 12 cases the group journal scores either matched or exceeded the observed scores from event maps of video data suggest that scoring of journals in this way can serve as a reasonable documentation of learning through the process of improving from failure.
Reliability of the Rubric
I developed the rubric based on the theoretical model of improvement (Figure 5.10, pg.114). The rubric itself can be found in Appendix D. After using it to score all the journals myself, I met with my research group to describe the scoring system. Each of the three participants scored two journals and we discussed issues they encountered or confusion they had using the system. I revised the rubric, and included the typical page numbers on which to find evidence, and I included examples and counterexamples of sufficient responses. I then sent the revised rubric to a fellow researcher from the group and asked her to score eight journals, two from each experimental class. A Cohen’s kappa test of the twelve scored improvement cycles and found an interrater reliability quotient of 0.51, considered to be in moderate agreement
(Viera & Garrett, 2005).
I then sent the rubric to two researchers from EiE, who are most likely to be interested in this type of scoring system. No training or discussion occurred prior to their scoring, and they
121 scored the same eight journals as my research group colleague. I calculated the kappa statistics to be 0.52 (moderate agreement) and 0.67 (substantial agreement) (Viera & Garrett, 2005).
This measure is limited due to the small sample size. Although there was moderate to substantial agreement, these researchers do not have the same research questions as I do. I envision this rubric being used either by a research team at EiE or by individual teachers grading the work students did during an engineering design unit. It is likely that multiple raters would be in a research group in which they would be able to develop a shared understanding of the threshold of evidence for each category, and a single teacher does not have to be concerned about consistency between raters.
Considering the Hawthorne Effect
I initially chose to score only the journals of student on camera in an effort to better understand what they write about as it relates to what they say and do during the engineering design process. But being video recorded during school is an atypical experience for a student and could potentially lead to an undesirable phenomenon known as the Hawthorne Effect in which effects of an intervention are observed solely because the subject knows she/he is being researched (Brown, 1992). To evaluate this, I compared average scores of journals of students who were video recorded during the design and eight randomly selected journals from students in the same classes that did not appear on camera.
First, I calculated average journal scores for each student who appears on camera. EiE staff had supplied me with de-identified student journal numbers that corresponded those students on camera, and I was able to deduce the group the journals came from by cross- checking data found in the journals with data found in my event maps (e.g., critical loads measured for bridge types in Lesson 3.1). I then randomly chose four student journals from each
122 class that came from students off camera and calculated an average score. The average journal score on camera (M=2.78, SD 1.32) was not significantly different from the average journal score from students off camera (M=3.04, SD 0.935, p=0.68).
I further investigated differences between classes by performing t-tests on average journal scores on camera and off camera. Two classes (Ms. James and Ms. Maddux) showed statistically different average scores; journal scores were higher in the off-camera journals
(p=0.03) in Ms. James’ class, but were significantly lower off camera in Ms. Maddux’ class
(p=0.002). Statistical power in this comparison is limited by low sample size as with previous quantitative analyses, however there is no evidence in these data that suggest that student journals from groups that know they are being researched should be expected to be different from other student journals (Table 5.18)
Table 5.18– Differences between on- and off-camera journal scores
Class On camera N Off camera N P-value average average Ms. Thomas 3.63 6 3.75 4 0.762 Ms. James 0.67 5 1.75 4 0.032 Ms. Maddux 3.5 12 2.5 8 0.002 Ms. Clay 2.69 14 2.63 7 1.0 Total 2.78 37 3.04 23 0.68 In this section, I suggest that student journals can be scored for learning during the process of improving after failure. To make this case, I first developed a rubric based on a theoretical model (Figure 5.10, pg. 114) in which students must first identify the cause of the failure, and then follow a logical progression in which they aim to improve on a criterion that failed, and use a strategy that responds to the cause of the failure. Next, I applied this scoring system to both student journals and digital video recordings and compared them both with a measure of interrater reliability. To increase trustworthiness (Cresswell & Miller, 2000) of my
123 analyses, I also included a table (Appendix B) with direct evidence for the comparisons. These data suggest that in the absence of a combination of video and journal data, journal responses alone can be used to make reasonable inferences about group work by considering a composite score from the group’s collective journals.
Then, I consulted three independent researchers familiar with this data set. Two of them scored a total of 12 cycles of improvement with no training or discussion, and one helped to refine my rubric prior to her analysis. Comparisons of journal scores with each of these researchers yielded moderate to significant agreement using the weighted Cohen’s kappa measure. Last, I addressed concerns that engineering journals from students being researched are not representative. A statistical analysis suggests there is no significant difference between the average journal scores, and my analysis of over 95 hours of digital video recordings support that finding; students in this study rarely acted differently because they were on camera.
Summary: Analysis of Student Journals
This chapter began with a parallel analysis of the EiE: Bridges unit that I used in Chapter
Four. The affordances of this unit for students to engage in the process of systematic improvement led to two additional findings. First, students with productive strategies that are given the opportunity to improve and a fair comparison to realize improvement will use tools to frequently engage in cycles of low stakes testing and modifications that leads to improvement in their final design. Second, journals can be considered as a way to measure learning during the engineering design process, particularly as it pertains to systematic improvement. The next chapter will be a cross-case analysis of Chapter Four and Chapter Five, and I will return to my research questions to frame the comparison.
124
Chapter 6 - Comparison of the EiE and E4C
In Chapters Four and Five, I analyzed separately two engineering curricula used in the
Evaluating the Efficacy of Engineering is Elementary (E4) project conducted by the Museum of
Science, Boston. E4C: Civil Engineering was designed as a control curriculum specifically for the E4 study, and was never intended to be used in classrooms after the conclusion of the study.
For that reason, I have tried to avoid criticism of the curriculum or teachers from this group because it is the control curriculum. It does, however, resemble hands-on engineering lesson plans found on the internet, making it reasonable to analyze how students and teachers engage with a hands-on engineering curriculum like E4C: Civil Engineering to better understand the nature of engineering design failure. I developed models of failure types and of the causes, teacher reactions, and obstacles to improvement. Then, I applied these models to the EiE:
Bridges unit to consider their generalizability, and to compare these constructs in the contexts of both curricular unit. Since EiE: Bridges allows for improvement, I expanded my analysis to consider that process in Chapter Five, and then proposed a way to document learning that occurs during the process of improvement.
In Chapter Six, I return to my research questions to frame the comparison of the E4C:
Civil and EiE: Bridges classrooms I analyzed in this study:
1. What is the nature of engineering design failure in elementary school settings? a. How do designs fail? b. Why do designs fail? 2. How do teachers react to failure during engineering design? 3. How do the collective actions of students and teachers support or constrain the students’ ability to use failure to improve?
125
Research Question 1a. How Do Designs Fail?
Similarities
Figure 6.1 is the model first describe in Chapter One describing the three axes of failure types: intent (intended to unintended); stakes (high to low); and referent (subjective to objective).
The classes I chose to study (Lessons One and Four in E4C: Civil; Lessons Three and Four in
EiE: Bridges) were selected because they had several opportunities for both high and low stakes failures and failures that were both intended and unintended.
Intent. Lessons One (E4C) and Three and
Four (EiE) ask students to test the strength of their
structures by adding weight until it collapses. In
E4C: Civil, the critical load was recorded and
compared to a prediction students made prior to
testing (e.g., line 53, T14117L12d1F) while in EiE: Figure 6.1 - Failure types Bridges, the critical load is scored on a scale from one to five on a curriculum-provided rubric (e.g., line 187, T14143BrL42T2). Intended failure is common to the field of engineering, but not in the way it is presented in these classroom activities. Household circuits and highway signs are designed to fail in intended ways to increase safety. And cement is destroyed in tests in civil engineering applications to verify its composition. Not all technologies are tested to failure in their final form, for example, bridges are not constructed and then loaded with weight until they collapse. However, strength testing in these ways in classroom applications provides a tangible opportunity for improvement, and engages students in the suspense of waiting for the bridge to collapse.
126
Lessons Four from both curricula provide the opportunity for unintended failure. A popsicle stick bridge in E4C fails if it is unable to support a five-pound weight for thirty seconds.
In EiE: Bridges, a bridge fails in this way if a toy car is unable to pass across its deck. The
Challenger explosion and the Tacoma Narrows Bridge were unintended as are virtually every engineering failure described in engineering classes and books by authors like Petroski (1985).
Stakes. Both curricula include public testing of structures. When these final designs failed, they are considered to be high stakes. Depending on the class, the stakes ranged from a formal presentation at the front of a quiet room (e.g., line 48, T14093CVL42F) to an independent test in which the teacher moves on to the next group part way through the test (e.g., line 85,
T14142BrL42T1). Interestingly, none of the classes used these public testing sessions to point out specific features of students’ designs as a way for students to improve in their next iteration.
At the other end of the stakes continuum, low stakes failures occurred in both sets of classrooms during the design and construction phases. These instances often involved the students recognizing, either through low-stakes testing or through their judgement that improvements had to be made. The stakes are generally related to the time the student has to improve the design before the public testing.
Referent. I did not report on differences between subjective and objective failure in the previous chapters. Subjective failure refers to situations where students view their design as inferior to other designs in the class, even in cases where it did not fail to meet the criteria of the lesson. An example of subjective failure was seen in EiE: Bridges when Lenny told Ronnie that
“Yours [bridge strength] was second best” (line 89, T14142BrL42T1) and in E4C: Civil when
Lucy thought their structure “took third place” (line 49, T14117CvL12d1T2). I chose not to
127 present descriptive statistics for subjective failures because they are dependent on explicit verbal comparisons from the students on video. Subjective failure is a reality of engineering when a client chooses one design over another or when an engineer decides to improve upon an existing solution, but comparing designs with other students’—especially on only one criterion like a structure’s critical load—might be reconsidered to view the design as a whole instead.
The classification system developed for Chapter Four was sufficient to categorize all the failures observed in both units studied, and so far I have been unable to think of an engineering design failure in any context that requires additional distinctions. Among the many pithy quotes about failure, their message often depends on the failure type. For example, Thomas Edison is often cited for having said, “I haven’t failed, I’ve just found 10,000 ways that won’t work,” but the title of former NASA flight director Gene Krantz titled his book, “Failure is Not an Option,” referring to the Apollo 13 mission in which three astronauts were nearly lost in space due to an explosion in the fuel tank (Krantz, 2001). Edison is referring to the many times he encountered low-stakes failure and had the opportunity to improve and re-test, but the failure Krantz refers to is among the highest of stakes—preventing the death of three men.
Differences
The designs in E4C: Civil differed with the designs in EiE: Bridges in how they failed due to differences in the number of criteria required. Lesson One has students build a tall bridge that can hold as much weight as possible, but the height is never measured when testing the designs. The popsicle bridges of Lesson Four are only required to support a weight for thirty seconds. The EiE: Bridges unit is evaluated on cost, stability, strength, and whether a barge can pass under the deck. This difference is significant because it eliminates an important aspect of
128 engineering design—tradeoffs. All other conditions equal, the strength and height of a tower in
E4C: Civil are inversely proportional, so decisions have to be made to balance these criteria if height were part of the evaluation. Cost is another important factor and tradeoffs often must be made between performance and the amount of money to produce (i.e., “You get what you pay for”).
Further, the increased number of criteria creates more opportunities for failure—but also more opportunities for improvement. Only one EiE group in the sample earned the maximum possible points (pg. 42, journal 76571), and they only achieved that score because cost was not included in the final evaluations. Five groups in E4C achieved “perfect” outcomes in the bridge design of Lesson Four (lines 54, 61, 64 T14093CvL42d1F; line 37 T14116d1F; line 32
T14117CvL42d1F), because there was only one criterion to accomplish.
Summary of Research Question 1a
The categories of failure type were developed to account for the myriad of ways in which designs fail. In his recent book, Firestein suggests, “failure is much too simple for the class of things it represents” (2015, pg. 7). Rather than trying to better define failures in the context of science (the title of Chapter One of his book), he stops short suggesting that a “lengthy polemic trying to define failure would surely fail” (Firestein, 2015, pg. 8). This dissertation is about engineering design failure only, and excludes failures Firestein also considers in academics, business, and marriage (2015). A better understanding of the nature of design failures should enable us to better analyze failures within all types of engineering and engineering education contexts.
129
The categories of failure types were developed prior to analysis, but it quickly became clear that another important consideration is the reason why designs fail. For that reason, I also developed through analysis four categories of reasons for failure in these engineering designs.
The second part of this research question addresses them.
Research Question 1b. Why Do Designs Fail?
Similarities
In the model developed in Chapter Four, I identified four reasons why engineering designs fail: 1) lacking knowledge in science and/or technology, 2) lacking an understanding of materials and their properties, 3) poor craftsmanship, and 4) an inherent limitation of the materials. These causes are not isolated to classroom engineering designs. The Tacoma Narrows
Bridge was an example in which the designers did not account for the “cumulative effect of undampened rhythmic forces… [producing] intense resonant oscillation” (Washington DOT,
2005). The explosion of the Challenger space shuttle has been attributed to two rubber O-rings that were not designed to withstand the cold temperatures on that January morning (Rodgers
Commission, 1986), a version of misunderstanding the material properties. The Kinzua Bridge was a 301 foot-tall railroad viaduct in northwestern Pennsylvania that collapsed in 2003 after standing for over one hundred years. It collapsed during a tornado while undergoing repairs because its steel struts, legs, and lacings were severely corroded (Eck, 2010), an example of the limitation of the materials used at the turn of the 20th century. And the Hubble Space Telescope famously failed due to a spherical aberration caused by mismeasurements during the polishing of the mirror (Wilson, 1990).
130
Of these failure causes, the two that can be challenging to distinguish are those related to materials. The Kinzua Viaduct collapse (Eck, 2010) would be considered a limitation of materials, even though the designers used a material that corroded and allowed the failure.
Similarly, the World Trade Center was designed to withstand the impact of the largest jet that existed at the time (Bazant & Zhou, 2001), but failed when impacted on September 11, 2001. In each of these cases, the materials used at the time were appropriate but failed later for reasons that cannot be attributed to the designers. In classroom engineering projects, students are typically limited in the materials that are available to them. Particularly in situations of intended failures, the students may utilize the materials to near-optimal design but the solution may fail due to the nature of the materials. The main distinguishing factor is whether the failure was primarily caused by the designer (due to a lack of understanding) or primarily caused by the shortcomings of the materials used.
All failures coded from both E4C and EiE all fit into the categories established in Chapter
Four. Throughout this project, I have attempted to compare these failure causes with failures in engineering design and I have not found any causes that requires an additional category. An example I have considered is the design of a circuit in electrical engineering. It may fail because it is not properly grounded (lack understanding of science), because an insulator was used rather than a conductor (lack of understanding materials), because one of the connections was soldered incorrectly (poor craftsmanship), or because after years in the elements the metal eventually corroded (limitation of materials). However, there is high likelihood that examples will present themselves when this work is reviewed by others. It is also important to note that these failure causes have only been applied to engineering design and not to other engineering activities. As I
131 will describe in Chapter Seven, further research in other fields of engineering will test this model and may also require modification.
Differences
Frequency. Although the reasons why designs fail could be classified into four categories, the frequency for each cause varied drastically between the E4C: Civil Engineering and EiE: Bridges. Table 6.1 compares the percentage of each failure code in each curriculum.
Table 6.1 – Comparison of Percentage of Failure Cause Codes
Curriculum Knowledge of Knowledge of Poor Craftsmanship Limitation of Science/Technology Materials Materials
E4C: Civil 49% 30% 17% 3% EiE: Bridges 14% 23% 18% 46%
A drastic difference is in the high percentage of failures in the EiE unit caused by a limitation of materials. This was related to the structure of the curricula. Students in Lesson 3.1 of the EiE unit test three bridge designs in small scale to compare the relative strength of beam, deep beam, and arch bridges. The comparisons were made by testing until the designs collapsed
(low stakes, intended failure). Additionally, Lesson 2.2 involved a teacher demonstration of methods to counteract unbalanced forces. Students observed the effect of adding a pier to support a weight added to the top of a one-story structure. The effects of this difference were observed in differences between the bridge designs from E4C Lesson Four and EiE Lesson Four. All of the
EiE groups used piers at some point during their designs, even when they utilized an arch design; some of the groups in the E4C classes did not. This accounts in part for the difference in the percentage of failures I attributed to a lack of knowledge in science/technology.
Difficulties transitioning. Another difference observed that led to failures in EiE was
132 the difficulty in transitioning from small scale testing to the final designs. This did not occur in the E4C units because they did not test small scale prototypes (EiE Lesson 3.1) or explore material properties (EiE Lesson 3.2). One of the reasons for testing different bridge types is to enable students to make evidence-based decisions in their final designs. The arch bridge, under the conditions of Lesson 3.1 is the strongest. Not surprisingly, five of the eight groups first attempted to construct an arch bridge. However, the increased span length required in Lesson
Four caused difficulties for them because single index cards (used in Lesson 3.1 as the arch) would not cover the fifteen-inch span. Two groups overcame this challenge by using multiple arches. Leo and Lola from Ms. Thomas’ class used three arches, while group two in Ms.
Maddux’ class used two (Figure 5.5, pg. 100). The other groups either abandoned the arch altogether in favor of a beam and pier design, or simply added piers to the arch design, in which the arch added little benefit in terms of strength.
The transition from exploring material properties in Lesson 3.2 to the design challenge of
Lesson Four also proved to be difficult for most of the groups in EiE classes. Each group had the opportunity to handle the building materials and consider their properties, the type of material it is made from, ways to change its shape, and ways it could be used in a bridge design. Despite this opportunity, none of the groups were able to use this time as it was intended. Table 6.2 summarizes the class discussion about the string. It is a representative example of every class I observed.
133
Table 6.2– Summary of Class discussion about the String – From Transcript T14142BrL32D1T1 (lines 56-59)
Question posed by Ms. Thomas Class discussion What are its properties? It’s pink. You can tie it. It’s floppy. Twists itself. Make it any shape. How can you change its shape? Peel it apart. Cut it. Tie a knot into it. Unravel it. Braid it. Tie it to things. You can make a bow with it. How can you use it in a bridge You can tie it to things. You can hold things. design?
From this example, it is clear to see why none of the students in this class used the string in their design. While it is likely the curriculum designers envisioned this activity as a way for students to consider specific ways these materials can be incorporated into their designs based on their characteristics, the classes I observed did not apply these ideas to the specific features of a bridge: that they must transfer the weight of a load to something attached to abutments or ground
(strength) and they must account for unbalanced lateral forces (stability).
The EiE: Bridges unit is designed to help alleviate two of the causes of failure: lack of understanding of science/technology and lack of understanding of materials. Lesson Two is a science lesson about balanced forces, and Lesson Three has students test prototypes of three bridge designs in small scale, and gives them opportunities to investigate the properties of building materials. Despite these affordances, the translation from these lessons to the final design challenge is non-trivial.
Low stakes testing. Another difference noted between the curricula that led to failure in designs was the use of low stakes testing. Each group observed in EiE classes utilized low-stakes testing to improve in some ways (Tables 5.16 & 5.17, pgs. 112 & 113). However, as demonstrated in Table 4.14 (pg.71), Ms. Lyle (E4C) specifically outlawed low-stakes testing. In fact, none of the E4C teachers provided students with materials to test their designs prior to the
134 public demonstration. Consider an engineering firm presenting a design to clients. The design team would have surely tested their design to ensure it has the highest likelihood of success during the high stakes presentation. Nevertheless, in these high-stakes classroom tests, none of the groups in E4C were able to test their bridges adequately, and thus did not know the likely outcome of their designs.
Summary of Research Question 1b
The bridge designs from EiE and the card structures and bridges from E4C shared similarities in the reasons they failed. Poor craftsmanship, for example, will likely be a challenge for any engineering activity in elementary schools because of the physical dexterity required.
However, differences were noted in the students and teachers’ enactment of the respective units.
These data suggest the importance of low-stakes testing of prototypes as a means of incremental improvement. They also suggest specific challenges in the ability of students to generalize concepts of science/technology and materials from one context to another. This difficulty led to failures in designs in EiE classes caused by misunderstanding science concepts like balanced forces and misuse of materials despite explicit lessons meant to prevent them.
Research question one only deals with failure. The second two consider failure as only the beginning of the process of improvement. Question Two considers the role of teachers in promoting improvement after failure.
Research Question 2. How Do Teachers React to Failure During Engineering Design?
The role of teacher as evaluator during engineering design challenges like those described in this study is less important because success and failure are easily interpreted by the students
(Cunningham and Carlsen, 2014b). Although their feedback is not essential for students to
135 improve, this research question considers the roles teacher can play in supporting students’ learning from failure and in improving. This role is interesting when considering the effects of feedback on design, testing, and re-design, and potentially in supporting students in learning
“habits-of-mind” (Katehi, Pearson & Feder, 2009) and participating in some of the epistemic practices of engineers (Cunningham & Kelly, in review).
Similarities
The categories I used to code teacher reactions were initially developed through conversations with my advisor. They were grounded in my experiences in observing teachers and in my experiences as a classroom teacher. Teachers must wear many hats in the classroom to be effective. They are not only responsible for the intellectual development of students, but also to varying degrees for their social and emotional wellbeing. Additionally, they are responsible for covering certain instructional material in a limited amount of time, and to create and maintain a productive learning environment. The categories of teacher reactions consider these roles. The manager role was created to consider the roles of class and time management, and conducting an efficient lesson. The cheerleader reaction considers the role of teacher in helping students avoid disappointment, and the strategic partner considers the role in helping students learn. While all these roles are important, it is necessary to consider the effects of each reaction. Table 6.3 shows all coded teacher reactions.
136
Table 6.3– Teacher reactions to failure
E4C:Civil Manager Cheerleader Strategic Partner Total Ms. Lyle 4 1 14 19 Ms. Flemming 10 17 13 40 Mr. Tanner 29 13 9 51 Ms. Houseman 6 0 13 19 Total 49 31 49 129 Percentage 38 24 38 100 EiE: Bridges Ms. Thomas 3 0 4 7 Ms. Maddux 3 3 3 9 Ms. James 0 0 7 7 Ms. Clay 5 2 5 12 Total 11 5 19 35 Percentage 31 14 54 100
With the exception of Ms. James, every teacher displayed different roles when reacting to failure. This suggests the dynamic nature of teachers’ perceived roles during instruction and a confirmation of my expectation—teachers understand that they have multiple responsibilities in the classroom. At several points through the analysis, consideration was given to creating new categories or specifying sub-types (e.g., what strategies does the teacher employ when playing the strategic partner?). I ultimately decided that these categories were sufficient for this study because they serve not only to consider the general ways in which teacher react, but also as a lens to consider the potential outcomes of each reaction in supporting improvement and learning.
Future research will likely require more fine-grained analysis of teacher discourse related to their reaction types.
137
Differences
As I described in Chapters Four and Five, there were significant differences in the reaction types within each group of teachers, as might be expected given the variables in teachers’ differences in their emphasis on the different roles. This variability is likely to occur in any group of teachers. I next consider the differences in teacher reaction between the two groups to highlight the ways in which curricula affect the frequency and type of reactions.
Frequency. Table 6.4 compares the total number of failures coded to the number of teacher reactions coded.
Table 6.4– Total failures and teacher responses
Treatment Group Total Failures Coded Total Teacher Reactions Coded E4C 133 129 EiE 137 35
A clear difference between the treatment groups lies in the frequency of responses made by the teachers. There are a few reasons that account for this difference. First, 82 percent of the failures coded in EiE were low stakes (Figure 4.1, pg. 52), while only 54 percent of the E4C failures were low stakes (Figure 5.1, pg. 87). Thus, the vast majority of failures in EiE were done within student groups without a teacher present. Further, many of the low-stakes, intended failures in EiE were done while evaluating the relative strength of bridge types in Lesson 3.1. In the four E4C classes, all of the high-stakes failures were done while the teacher officiated, prompting responses. Among the EiE teachers, Ms. Thomas was not present for all public tests, and even enlisted the help of a researcher to help facilitate the tests, and Ms. James never conducted public bridge tests.
138
Pre-emptive strategic partner response. Another factor that likely led to this difference in frequency of responses is due to the curriculum design. E4C groups are given all materials in advance; EiE students write a list of materials and “purchase” them from the teacher, preventing
EiE teachers from circulating during design and construction. As a result, I noted nine instances of the strategic partner response that was given pre-emptively (Appendix C). These responses were given during class discussions about failure, and might have occurred as a result of teachers understanding they would not be present to help their students during low-stakes design failure.
This parallels a finding of Lottero-Perdue (2015), which she calls, anticipating failure. Only Ms.
Lyle (E4C) used this pre-emptive strategic partner approach, prompting her students to remember what the class knows about failure. Rachel responds, “You learn from your mistakes”
(line 18, T14093CvL11D1T2). No other pre-emptive strategic partner responses were noted in
E4C.
Prospects of improvement. The last major difference found in the lower frequency of strategic partner responses in E4C (38% to 54%) is also likely related to the curriculum design and professional development. Teachers from E4C were coached to respond to all public tests with two questions, 1) What went well, and 2) What would you do differently? While the second question could be construed as a strategic partner response, I did not code it as such unless the teacher engaged with the student more than listening to the responses. By design, E4C provides no opportunity for improvement, so the strategic partner response is less likely because student need no new strategies—the activity is over! Further, for the purpose of this study, teachers were unaware that they were teaching the control curriculum. Most of the teachers liked the fact that it is “hands-on” and are excited the Museum of Science has created a curriculum to help them teach engineering (Cunningham, personal communication). This is not a critique of the teachers
139 or the curriculum; it is a potential explanation of differences I observed.
Summary of RQ 2
The teacher reactions to failure I developed through this study have not been used in any context that I am aware, but more specifically they were not used in the professional development of the teachers in this study. Due to the differences that exist among teachers and their perceived roles, is it not surprising that there is no discernable pattern within each group— the teachers display a varied repertoire of responses based on the feedback they perceive is necessary at that time. Differences were noted between the treatment groups in the frequency and types of their responses due to the structure of the curriculum. EiE teachers responded less frequently to failure compared with E4C teachers because the majority of the failures in EiE were done within the group. Because the teachers would not always be present to support the students’ use of failure, teachers of EiE used the strategic partner response proactively. In addition, due to the affordance of EiE’s improvement cycles, the strategic partner response was also used a higher percentage of the time to support those students with the opportunity to improve.
My final research question also considers failure as a part of improvement, rather than an isolated phenomenon. It compares the ways that the teacher’s and students’ collective actions and enactment of each curriculum support (or constrain) improvement.
Research Question 3. How Do the Collective Actions of Students and Teachers Support/Constrain the Students’ Ability to Use Failure to Improve?
The final research question for my study considers the actions of students and teachers engaged in engineering design activities and the ways in which they support or constrain improvement. Some of these actions of driven by the curricula, while others are a result of the
140 perceived roles of the students and teachers in their respective social groups. The written E4C unit does not afford the opportunity for improvement, however there are some notable exceptions to highlight. And despite the inclusion of at least one improvement cycle in EiE, sometimes the students and teachers constrain improvement by their actions. Rather than arranging this section in which I compare the similarities and differences between the two curricula, I present this section in the ways improvement is supported and constrained.
Support
Related to the perceived teacher roles discussed in question two, teachers and students generally act in ways to fulfil their perceived roles. Participation in those groups involves particular ways of talking, thinking, acting, and interacting (Kelly, 2014), a component of my theoretical framework and justification for the methodology I used. In a study of classes doing projects from curriculum kits, I am most interested in the ways students and teachers deviate from the “written curriculum” because they give insight into the perceived roles. In some cases, those deviations supported improvement after engineering design failure.
Multiple opportunities. In Chapter Five, I showed that every group improved after low- stakes failure because they frequently test their bridges prior to high stakes evaluations (Tables
5.16 & 5.17, pgs.112 & 113), but in no class do the teachers direct them to do so. Low-stakes testing was coded 122 times in the EiE classrooms (Tables 5.16 & 5.17, pg.112 & 113), and students used not only the weights and cars to test the strength and stability, they also used other objects that are available, like a water bottle as a weight (line 134, T14098BrL42T2).
In a TED talk, “Build a tower, build a team,” Wujec claims that “prototyping matters”
(2010). He describes a simple engineering design challenge he uses as a team-building activity.
141
Participants have eighteen minutes to build a tower with raw spaghetti, string, and tape, and the goal is to build the highest tower that can hold a marshmallow. He claims that kindergarten students typically perform better than recent business school graduates due to frequent low stakes testing and suggests the business students spend too much time trying to get it “right” just in time for the final presentation (Wujec, 2010). This observation concurs with my data, that most of the students instinctively use low stakes testing to monitor and adjust their designs.
Wujec’s talk seems to suggest society (or school?) teaches us to reject the idea of prototyping. Table 6.5 was presented earlier, but lends credence to this theory. Minutes after
Max was scolded by Ms. Lyle for low-stakes testing, the group works on their card structure
(Table 6.5)
Table 6.5– Denying Opportunity for Low-stakes Failure
Time Line Laura Max Rachel Contextual Clues 31:00 1 Wait, let me see Picks up a journal from the desk 2 something. I want 3 to see if it can hold 4 something 5 [No! No testing!] [No testing!] Max backs away, crossing his arms 6 I really wanna test 7 it 8 Like, after we’re 9 done, I really want 31:20 10 to see if it will ###
This example shows the girls, Laura and Rachel upholding the rules Ms. Lyle recently enforced: no low-stakes testing. Despite Max’ desire to test and modify, Ms. Lyle, Laura and
Rachel—all trying to be good teachers and students—do not allow him to do so.
In addition to improvement from low-stakes failure, improvement is supported by formal testing and class time in which to improve. Three teachers from the EiE classes allow their students to formally test their bridges and redesign at least once, while only Ms. James does not.
142
The result of this opportunity is that at least one group in those three classes shows improvement after high stakes failure while neither group from Ms. James’ class does (Tables 5.16 & 5.17, pgs. 112 & 113).
Interestingly, Ms. Flemming, an E4C teacher adds an additional class period to her bridge lesson after every group’s bridge fails to hold the weight. As described on pg. 60, she holds a class discussion in which each of the failed designs are publicly analyzed, and suggestions for improvement are made by both Ms. Flemming and the class.
Attribution of failure. The purpose of failure analysis is to determine why a design failed. Students’ understanding of the reason their design failed will often lead to a strategy for improvement. Also, finding aspects of a design that can be improved is found in the elementary level NGSS and can be supported in this way (NGSS Lead States, 2013). Table 6.6 is an excerpt from Table 5.9 (pg. 94) in which Ms. Clay (EiE) speaks with a student group.
Table 6.6– Ms. Clay helps attribute a cause of failure – from Transcript T14098BrL42T2
Line Liam Ms. Clay Contextual Clues 24 Well, what’s the problem? What Hand motions back and forth 25 would you have used? What do we signaling the answer she’s looking 26 know about spans? What do you for that support goes across the 27 want to have across the span whole span 28 Um, the strongest material. 29 What would the strongest material 30 be? 31 What’s it called, uh, craft sticks. In this conversation, she uses leading questions to help the students to recognize the reason for their bridge’s low strength score was due to their use of a decking material that was not rigid. This example was used to demonstrate a strategic partner reaction, but emphasizes the point that attributing a cause of failure (a “weak” material) immediately moves on to a new strategy (craft sticks) that will likely be an improvement.
143
Further supporting this attribution of failure, some teachers prompt students to critically analyze the failures of prototypes to attribute failure and to use effective strategies. Ms. Thomas adds an extra class period to her EiE: Bridges unit. To demonstrate how the bridge types respond to a load on a larger scale then their initial prototypes, she conducts low-stakes, intended comparisons between arch, deep beam, and beam bridges, marking the movement both of the deck and the abutments as weights were added (Table 5.9, pg. 96). This strategic partner reaction to low-stakes, intended failure serves two purposes. First, it helps students to recognize the relative strengths of the bridges do not change with the increased span length. Second, she not only responds to an intended failure, she also models a strategic approach to analyze their bridge designs by considering both the movement of the deck and abutments. This type of failure analysis is supported by the EiE engineering journals for students’ small scale testing, but Ms.
Thomas is the only teacher to alter the unit by adding this activity.
Multiple criteria. The structures that students in E4C classes design are only evaluated on one criterion: critical load in Lesson 1.2 and holding a weight for 30 seconds in Lesson 4.2.
EiE: Bridges is written to evaluate bridges using multiple criteria: span, deck clearance, strength, stability, and cost. Evaluating designs along many criteria serves two purposes, it increases the likelihood of improvement, and it better represents an important characteristic of engineering: there is no “perfect” solution.
In contrast to E4C, five groups in E4C achieved “perfect” outcomes in the bridge design of Lesson Four (lines 54, 61, 64 T14093CvL42d1F; line 37 T14116d1F; line 32
T14117CvL42d1F), because there was only one criterion to achieve. In EiE, only one group perceived their bridge as “perfect” because only strength and stability were counted in the
144 evaluation (line 135, T14143BrL42T2). Multiple criteria means there are more ways designs can fail, but failure can be supported as a part of learning.
As a result of the combination of multiple criteria and the ability to use low-stakes testing, every group in EiE improved after a low-stakes failure. In contrast, no group scored the maximum score when cost, strength, stability were all evaluated. This more closely mimics authentic engineering because there is a constant tension between multiple criteria, including the cost to construct. The concept of trade-offs is a disciplinary idea that is unique to engineering but is not found in the NGSS (NGSS Lead States, 2013).
Summary of support. The most compelling lesson from this study, in my opinion, is the importance of low-stakes testing and prototyping. The students in the study are compelled to test their designs several times. Again, failure of any type is not alone enough to promote improvement. Therefore, students (and teachers) must first learn to pragmatically analyze these failures in ways that promote effective strategies for improvement. Teachers can support this analysis in the feedback they provide both prior to and after failure events to support students’ improvement, but can also model ways to strategically analyze a design to find the reason(s) why it failed. This process is made more complicated, albeit more authentic, when solutions are evaluated on multiple criteria. While this increases complexity, it also increases opportunities for failure. But student whose designs fail but are given opportunities to improve are given more opportunities to learn.
Constraints
Just as there are ways in which students and teachers support improvement, there are ways that teachers and students unwittingly prevent it. I identified four constraints here, using
145 examples from both E4C and EiE. Three align with the obstacles to improvement identified earlier. The other is related to the ways in which public tests are conducted.
Public testing. Two of the E4C teachers (Mr. Tanner and Ms. Houseman) choose to conduct the high-stakes testing while the other students continue to construct their bridges or to clean up their desks after testing. Similarly, students in EiE classes are either given the option to observe groups testing (Ms. Thomas, Ms. Maddux, and Ms. Clay) or students were directed to conduct the high-stakes tests independently (Ms. James). Kolodner (2003) found students improved when given the opportunity to engage in forensic analysis of classroom design failures.
By affording students the opportunity to see other designs and to participate in analysis of its features, they are provided further options to consider during re-design. In contrast to the EiE teachers, Ms. Flemming adds an additional day to her E4C unit after each group’s bridge collapsed. During this episode described on page 60, she spends fifteen minutes discussing each group’s design and leads a group discussion about the reason each design failed and how it can be improved (lines 2-17, T13811CvL42D2F). This enables students to learn both by analyzing their own designs and from learning from others’ failures vicariously (Kapur, 2014). Engineering relies on innovation, and improving on existing solutions, so preventing students’ learning from vicarious failure constrains improvement. Providing students with ideas with which to expand on promotes this innovation; keeping students isolated constrains this opportunity.
Lack of opportunity to improve. With the exception of Ms. Flemming allowing her students to re-design and re-test, none of the E4C classes were given the chance to improve. By definition, that constrains improvement. But two EiE classes also constrained improvement in the ways they collectively acted. Ms. James never held public testing. Although she verbally
146 directed students to test and redesign on their own (line 67, T14164BrL42T1), neither of the groups in my sample followed her directions. As a result, neither group ever got a formal bridge score on which to improve.
Another example of a way that constrained improvement was in the way Ms. Clay scored students’ designs. Her class went through two cycles of improvement, but the costs were not evaluated in the way the curriculum suggests. Any materials used in improving their bridges were added to the cost of the previous design (line 143, T14098BrL42T2). As a result, students did not have the opportunity to improve their cost score, and their overall scores suffered by the characterization that improvement had to be additions to an existing structure.
Unfair comparisons. Prototypes must be comparable in order to observe improvement. I illustrated this point in Chapter Four with a transcript of Mr. Tanner (Table 6.7)
Table 6.7– Unfair comparison – From Transcript T14117CvL12D1F
Time Line Owen Mr. Tanner 53:35 1 Did we get first, second 2 or third place? 3 First, off, it was not a competition. Because if it was a competition, we 4 would have had a lot more variables, such as height, because directions 5 were a ta:ll structure, right? So it’s not a competition, it’s about how 6 did your group do compared to what you predicted. So don’t worry 7 about first, second, third place. There was no prize. E4C did not allow students to improve their card structures from Lesson 1.2, but if it did, the comparison Owen tries to make between their structure and others in the class would run into the same challenge. Confusing instructions: “…design a structure that is as tall as possible and that can hold as many weights as possible,” (pg. 15 of the E4C journals) coupled with only evaluating the designs on their critical load would make it impossible to accurately compare structures of differing heights. This highlights a very tangible trade-off between height and strength, but by evaluating only strength, fair comparisons become impossible.
147
A subtle variation on testing methods created a similar situation in Ms. Clay’s EiE class.
The preferred strength test in EiE: Bridges is to place weights in a cup placed in the center of the deck. Ms. Clay’s groups tested strength by adding individual weights directly to the deck (Figure
5.18, pg.113). Rather than a test of strength, failure was declared when a weight fell off the side
(a measure of stability). This subtle change in methodology prevented fair comparisons between prototypes, leading to one group to get a lower strength score in each subsequent test (Figure
5.16, pg. 113). This inability to recognize there is an unfair comparison between prototypes potentially leads to designs’ improvements are masked by an ineffectively conceived lesson
(Lesson 1.2, E4C) or poor methodology (Ms. Clay’s testing method).
Unproductive strategies. The first two categories in this section of actions that constrain improvement are largely controlled by the curriculum, with a few exceptions noted above. The strategies students use in response to failure can also inhibit improvement and is less curriculum- dependent. I will first highlight one example from EiE in which a group with a fair comparison and the opportunity to improve was unsuccessful, then I will speak more generally about other potential causes for this situation to occur.
The first group of Ms. Thomas’ group redesigned a bridge that scored lower (Table 5.16, pg. 112). They attempt to add piers to increase its strength, but their poor craftsmanship and failure to recognize their piers are of unequal lengths causes it to become too arched in the middle and the toy car is unable to pass across the span. In the same way that attributing the cause of failure can support productive strategies, the inability to accurately assess the performance can lead to subsequent failures, for the same reasons outlined as the causes of failure.
148
Innovation and improvement require a certain amount of risk to novice designers. In addition to understanding the causes of failure, students must be able to develop strategies to account not only for that cause, but to prevent unanticipated outcomes. Systems thinking is a cross-cutting concept outlined in NGSS (NGSS Lead States, 2013), and since most designs can be thought of as systems with interrelated components, often changes in one component can lead to unanticipated effects in another component. This lends further emphasis on low-stakes testing to work out these unanticipated outcomes.
Summary of RQ 3
As previously stated, failure does not necessarily lead to improvement, and this process relies on several necessary components. The first and most obvious is that students must have the opportunity to improve, not only by conducting formal tests and allowing for phases of redesign, but is also supported by the ability to continuously test and improve in low-stakes conditions.
Further, it is essential that students have strategies that help to overcome the shortcomings that led to the failure. These strategies are bolstered by the ability to critically analyze the reason(s) why the design failed and the ability to consider other alternatives through the opportunity to observe others’ designs in public testing and/or “gallery walks” (Kolodner, 2003). All of these conditions can be supported through curricular design and in the collective actions of students and teachers during the design, testing, and redesign phases of activity.
Summary of Comparison of EiE and E4C
This study presents a complex argument about the types of, causes of and reactions to failure, and the obstacles to improvement. To synthesize this complex argument, I will condense two of models I developed (failure model and model of improvement) and attempt to
149 demonstrate the ways in which they interact to support or constrain improvement in elementary engineering design project like those observed in this study. Figure 6.2 uses the concepts developed in Chapters Four and Five and reiterated here to show the ways in which the enactment of the curricula in this study serve to support or constrain the process of improvement from failure. It is based on the model for improvement (Figure 4.4, pg. 74). Supports for each step are shown above the model, and the constraints are shown below.
150
Table 6.8– Observed supports and constraints of improvement
Supports Improvement Constrains Improvement
Opportunity Lack of Opportunity
EiE – Table 5.17, Clay T1 Low stakes E4C - Table 4.4, pg. 74, Ms. Low stakes testing testing outlawed Lyle EiE – Table 5.17, Clay, EiE – Table 5.17, Ms. James Class time for Maddux, Thomas No redesign redesign E4C – Flemming phase E4C – Lyle, Tanner, Houseman (T13811CvL42D2F)
Productive strategies Unproductive strategies
EiE Student journal 76555 EiE Student journal 75998 No attribution/ Attribution of failure E4C-class discussion, Flemming misattribution E4C – Table 4.6, pg. 61 (T13811L42D2F) EiE – Table 5.16, Maddux T1 Poor EiE – Table 5.16, Thomas T1 Understanding of understanding of science/technology E4C – added piers, Flemming E4C – line 38, (13811L32D2F) science T13811CvL42D2F Poor Understanding of EiE – Table 5.16, Maddux T2 EiE – Table 5.17, Clay T1 understanding of materials materials EiE – Table 5.16, Maddux T2 EiE – Table 5.14, Clay T2 Poor Good craftsmanship Craftsmanship E4C – line 45, T13811CvL42D2F
Fair comparisons Unfair comparisons
EiE – Lesson 4 EiE – Figure 5.17, Clay T1 E4C – Lesson 4 E4C – Lesson 1
Teacher Responses Teacher Responses
EiE -Table 5.8, Ms. James EiE – Table 4.4, Ms. Flemming Manager EiE - Table 5.9, Ms. Clay E4C – Table 4.5, Mr. Tanner Strategic Partner EiE – Table 4.6, Mr. Tanner E4C - Table 4.8, Ms. Lyle Cheerleader E4C – Table 4.7, Ms. Flemming
Public Testing/Gallery walks No Public Testing
EiE – Ms. Thomas (T14142BrL42D1T1) EiE – Table 5.17, Ms. James E4C – Ms. Lyle (T14093CvL12D1T2) E4C – Mr. Tanner (T14117CvL42D1T1) 151
This diagram shows how complicated the process of improvement can be. And it is not only curriculum-dependent because it relies on either prior knowledge of students or a concerted series of conditions to support a logical progression that allows groups to acknowledge failure, find the reasons(s) why it failed, and then develop responsive and effective strategies toward improvement. It also is dependent on the discursive work of the teacher. You may notice I have largely refrained from using the word success in this dissertation, because as with all engineering designs, success is relative (and temporary). Improvement is about balancing trade-offs between multiple criteria in ways that are more effective than in prior prototypes. However, improvement suggests that learning has occurred in ways that a “successful” design that was observed in
Lesson Four of E4C does not.
To conclude this study, I will use Chapter Seven to consider the implications of this work and how it fits within other literature from the fields of science and engineering education, engineering, psychology, linguistics, and other studies of classroom discourse. Then, I will consider the limitations of this study in order to propose future research studies in this field that will further enhance our understanding of the role of failure in engineering education and its role in improvement.
152
Chapter 7 - Implications, Study Limitations, and Future Research
This study initially sought to better understand failure in engineering design projects in elementary classrooms and the specific ways teachers guided the students to improve. It is one of many currently progressing research projects studying K-12 students and teachers in engineering contexts and is timely because of the recent inclusion of engineering in K-12 science education reform documents (NRC, 2012; NGSS Lead States, 2013). This chapter serves to: 1) describe the ways in which this research relates to others’ work in engineering, psychology, and science/engineering education; 2) consider limitations of the study; and 3) propose further research projects to better understand this complex phenomenon.
Implications
Failure as an Epistemic Practice
Science education reformers have been promoting the use of practices of researchers to teach content for several years (Duschl, Schweingruber, & Shouse, 2007; Kelly, 2008, 2011;
NRC, 1996; Loucks-Horsley & Olson, 2000). The framers of the NGSS went so far as to feature eight such practices, and to compare the similarities and differences between engineers and scientists (NRC, 2012; NGSS Lead States, 2013). NGSS also promotes teaching engineering in
K-12 settings, following such calls by the National Academy of Engineering (2008) and others
(Katehi, Pearson, & Feder, 2009). Thus, recent reforms can be informed by research done about students engaged in epistemic practices, in engineering design projects, and in the epistemic practices of engineers during classroom activities (Cunningham & Kelly, 2015). Several studies have been conducted about students engaged in epistemic practices in science contexts (Kelly,
153
2008; 2011). Examples include studies on explanation (e.g., Krajcik, McNeill & Reiser, 2008), argumentation (e.g., Kind, et al., 2011) and modeling (e.g., Manz, 2012).
However, Cunningham and Carlsen (2014b) have suggested that the epistemic practices of engineers are not as similar to those of scientists as they are portrayed in the Framework for
K-12 Science Education (NRC, 2012) or the NGSS (NGSS Lead States, 2013). And Cunningham and Kelly (in review) suggest that these unique practices should be considered as ways to frame educational experiences. Although there is some work being done on identifying epistemic practices students and teachers engage in during engineering design projects, like controlling variables, using data for improved designs, and analyzing data as a collective group
(Cunningham & Kelly, 2015), this dissertation study is one of only two studies to date that deeply investigates failure in elementary engineering design (see also, Lottero-Perdue, 2015), and the only to attempt to characterize the process of improvement through sociolinguistic analysis in situ.
A better understanding of classroom discourse around failure will serve to advance the scholarship in a few ways. First, the research methodology examining failure in this study is complementary to work done by others in the field. Researchers like Lottero-Perdue and Parry
(2015) closely look at teacher-reported reactions to engineering failures by both teachers and students. Their work finds a great deal of negative connotation associated with the word, “fail,” and notes a majority of teachers avoid using these negative sounding words when responding to students. Their study benefits in that it includes surveys of over one hundred teachers, while mine took a closer look at eight classrooms engaged in engineering units to consider not only the ways in which teachers respond to engineering design failure, but also the context in which the
154 failures occurred. Moreover, this methodology enabled me to study student discourse around failure events when the teacher was not present, which is somewhat unique.
Among my findings, this study suggests that when given the opportunity, students will use low-stakes testing to improve their designs during the design process. Low stakes testing is one type of epistemic practice that allows students to iteratively test and improve, leading to knowledge of both engineering and the related science concepts and leading to better final designs. This finding parallels the narrative of Kujac (2010), in which he claims that kindergarten students were successful in an engineering challenge called, “The Marshmallow
Challenge” because they frequently tested and revised their designs based on the low-stakes tests. In E4C classrooms that prohibited low-stakes testing, students were inhibited from improving their designs.
Because work on epistemic practices in K-12 engineering is relatively new (Cunningham
& Kelly, 2015; Cunningham & Carlsen, 2014; Cunningham & Kelly, in review), there is much work to be done to better understand the types of practices students and teachers engage in while engineering in classrooms. A coordinated approach with several perspectives will likely shed more light on the affordances that can be made by curriculum developers and teachers to better support students’ engagement in disciplinary practices of engineers, particularly in improvement after design failure.
The work presented in this dissertation also applies to the understanding of failure in engineering education. Petroski claims, “The paradox of engineering design is that successful structural concepts devolve into failures, while the colossal failures contribute to the evolution of innovative and inspiring structures” (Petroski, 1985, pg. 163). In the E4C unit, there was little
155 evidence of learning for groups that successfully constructed a popsicle bridge that was able to hold the weight. Failure in classroom design projects provide opportunities for improvement, mimicking the professional work of engineers, but for a different purpose; engineers’ primary goal is not to learn, it is to design a successful solution. The primary goal in K-12 classrooms is for student learning, so designs that fail may be preferable (from a learning perspective) to those that do not because of the opportunities for learning they provide. Students in EiE classrooms are unable to achieve the maximum scores because there are evaluations on strength, stability, and cost. Thus, improvement is always possible. When given the opportunity to redesign, all students are presented an opportunity to learn through the improvement process. Students with poorly designed structures will be afforded the chance to improve, but even students will well-designed structures will be able to enhance their design in some respects.
Another contribution of this dissertation is the classification system of failures. In education settings, particularly from a sociocultural perspective, it is important to understand context. When describing failure, it is important to know how the design failed (intent), but also under what conditions (stakes), and what it failed in relation to (referent). This is not the first attempt at categorizing failures. Petroski devotes a chapter to intended failure, called, “Designed to Fail” (2012). In it, he describes examples of products that fail on purpose or manage failure in particular ways. An example he uses is the shatterproof glass of a windshield (Petroski, 2012). A better understanding of the conditions of failure precedes analysis of how engineers, students, and teachers respond to it. Lottero-Perdue and Perry (2014) categorize failure as being within the
Engineering Design Process of a failed final design. Fortune and Peters (1995) outline four failure types: 1) objectives not met; 2) undesirable side effects; 3) designed failures; 4) inappropriate objectives. While there is significant overlap with Fortune and Peters’ system
156
(1995) between my referent category (type 1) and my intent category (type 3), my system allows for the categorization of eight failure types. These failure types matter when considering the antecedents and reactions to them. For example, the balsa wood tower is a common classroom engineering challenge in which students design and construct a tower that they test by adding weight until it collapses (e.g., Techtronics, 2016). The discourse surrounding the testing and reactions to failure should be viewed much differently from activity like the bridge design from
E4C in which a bridge can fail to the hold the specified weight. Moreover, a distinction between objective and subjective failure will be important in several contexts. I chose not to report on subjective failure because given my research methods, subjectivity relies on the verbalization of students comparing their structures to another group’s. But there were poignant examples in my data that suggest it is an important aspect to consider. On this same axis, improvement does not have to occur because of a catastrophe or even an observable failure. Innovation often occurs because a designer wants to “build a better mousetrap.” The recognition of a design that can be improved is subjectively a failure.
The field of engineering is full of case studies about failure. As Lawson suggests, we should study disasters and failures because it is “more cost-effective and less stressful than experiencing them oneself” (pg. XX of the introduction, 2005). Learning from others’ failure is termed vicarious failure by Kapur (2014). While Lawson (2005) and Petroski (1985) use case studies of others’ failures as teaching tools, and Kolodner uses case-based reasoning in the same way in classroom projects (2006), significant consideration should be given to helping students develop a strategic response to personal failure (Kapur, 2014). Ross (1995) is intended for metallurgists and describes how to investigate mechanical failures. Books like these are much less common than those using forensic case study analysis. My work suggests a failure analysis
157 by both the student group (supported by the teacher when possible) is the first step toward improvement. Thus, an awareness of failure types and failure causes will better suit teachers in diagnosing failures in an attempt to help students improve upon them.
Improvement
When failure is considered as a part of the improvement process, it becomes more interesting. Matson (1996) advocates frequent failures during the experimental phase of design because earlier failures test the boundaries and enable creativity to emerge. However, these failures are only useful if they reveal some knowledge that can be used in subsequent iterations.
Similarly, several low-stakes failures were observed in the EiE classrooms but were only productive when the cause was able to be interpreted and responded to by the student groups.
This work suggests the encouragement of low-stakes failure and support in students’ analysis of failure causes them to progressively improve designs during low-stakes settings.
Engineers are not the only groups concerned with improvement. Psychologists have been considering success and failure for years, including a concept I modified for this study: attribution theory (Weiner, 1985). Psychologists like Weiner consider perceived causes of failure
(or success) and their locus, stability, and controllability (1985). These psychologists suggest that persistence in the face of failure is greatest in situations where failure is attributed to internal
(personal behaviors), that are unstable (able to be changed), and controllable (Weiner, 1985). In my study, I used the term “attribution” when considering the cause of a failed design. However, the model is modified to consider that the structure failing (as opposed to the person) and thus the locus of control is external to the students, is changeable, and is controllable in optimal conditions.
158
Others are interested in improvement of performance, not in engineering designs.
Attributional feedback has been studied in athletics, and it was found that the study of feedback aligned with internal, controllable, and unstable attributions were most successful at coaching feedback for novice golfers (Le Foll, Rascle, & Higgins, 2007). It has also been studied as a reading intervention (Chan, 1996) and in a recreational therapy context for depressed patients
(Dieser & Ruddel, 2002). This study characterized teacher feedback to failure as the manager, cheerleader, or strategic partner. In many cases, teachers used attributional feedback to help students consider the cause of the design’s failure as a part of their strategic partner reaction.
This allows students to interpret failure in ways similar to engineers. In manager or cheerleader reactions, no support for attributing a cause for failure is provided.
Attribution of failure became an important part of the framework in which I considered the process of improvement. However, determining a cause alone is not sufficient to achieve improvement. Students must also choose and execute a strategy that is aligned with the attribution in able to improve, and it depends on the failure (and attribution) being accurately interpreted. Future studies on students’ attributions of failure will benefit from interviews with students to better understand how they interpret failure and to what they attribute failure.
Most surprising to me in this study were the ways in which the students responded during failures. I reviewed literature on resilience prior to analyzing data because I anticipated students to differentially persist in the face of failure. However, most student groups in this sample persisted, even in the face of frequent design failures. This willingness to continuing to redesign prompted an altered view of Richardson’s resiliency model (1990), but I think it is still relevant to this research. The model of improvement from Chapter 4 (pg. 74) can be easily compared to
159 the resiliency model because there are several ways in which improvement does not happen, just as there are several reasons resilient reintegration is prevented. Their willingness to persist also led me to an understanding of their pragmatic approach to design and why students were not as quick to give up as I had anticipated. Even third graders are capable of intellectually engaging in a design challenge for several hours; a finding I suspect that will be surprising to many.
Many of the psychological constructs to which I refer in this study (e.g., attribution and resilience) are cognitive theories that are considered to occur within an individual’s mind. The students in this study work in collaborative groups; to what the failure is attributed is negotiated as a group, and decisions are socially constructed. Thus, attribution occurs interactionally and the field of STEM education would benefit from studies that look closely into the social construction of attribution and resilience. Investigations of this nature will be challenging due to the complexity of human interactions. Nevertheless, systematic investigation of students engaged in engineering projects in situ should be able to shed some new light on how these phenomena occur through interactions of social group members. For example, I do not perceive the actions and “biopsychosocial protective factors” of Richardson’s resilience model (1990) to occur in the mind—the resilience happens within the social group engaged in the engineering design process.
When a group’s bridge collapsed, students collectively acted in a resilient manner through their words and actions.
The improvement process was documented in the scoring of student engineering journals.
Finding statistically significant causal relationships between increased pre- and post-test scores is often challenging, and is not necessarily congruent with the goals of engaging students in epistemic practices of engineers. Assessing learning that occurs during the process of
160 improvement has implications for the field of K-12 engineering education. This data set enabled me to analyze video from three cameras recording a teacher and two student groups simultaneously. This was helpful in being able to correlate the types of things the students wrote in the journal with the events observed from the videos. The consistency between the journals and the events were similar enough to consider using this rubric (or one like it) to evaluate student artifacts as proxy measurements for video data.
Studies of Engineering-in-the-Making
Last, this work contributes to others’ work in classroom discourse of students engaged in disciplinary work. Similar to this study, Kelly et al. (2000) used ethnographic methods to analyze classroom activities and also collected student artifacts. Sezen, et al (2009) studied novice teachers and their attention to classroom activities by analyzing video recorded interactions.
More recently, studies like this have started to consider student engagement in engineering practices (Cunningham & Kelly, 2015).
The work in this study adds to this field by modifying existing methodologies (Kelly,
2014) to systematically analyze lessons from a large video collection to consider the epistemic practices of improvement from failure in engineering design. Some studies have measured the effect of these engineering curricula on increases in science knowledge (Carlsen & Johnson,
2014), increases in creativity (Hegedus, 2015), and increases in engineering and technology knowledge (Lachapelle, et al., 2010), using relatively large sample sizes. Others, like
Cunningham and Kelly (2015) have studied single classes in great depth to understand their engagement in engineering practices (2015). I chose to expand upon this work by studying eight classes and their experiences with failure during civil engineering units and to include lessons
161 from classrooms using two quite different curricula. By choosing to use video data and student journals only, I sought to avoid the influence of the connotations of the word “failure” in interviews and instead focus on the situated nature of the talk before, during, and after failure events. While in some respects, this is an advantage of this methodology, it also has some shortcomings which I will speak about in the next section. What I found was that contrary to an aversion to failure, students were able to persist when given the opportunity and work with great focus toward improvement. Studies like this serve to complement those choosing to use other methodologies.
Recommendations
This work enabled me to be a silent observer of eight elementary classrooms engaged in engineering design challenges. I have also spent many hours thinking about failure and improvement, and offer a few recommendations to both curriculum developers and teachers that may increase the ability of students to improve from failure.
To curriculum developers. An obvious lesson of this research is that students cannot improve without the opportunity to do so. However, there are two levels of improvement I was able to observe. Improvement from low-stakes failures was noted in each group in the EiE treatment group, but was constrained in E4C classrooms that outlawed low-stakes testing.
Improvement from high-stakes failures was non-trivial, because it required students to overcome at least three obstacles, including enacting a productive strategy. To better support students’ use of productive strategies, high-stakes testing can be done as public presentations and features that are effective (or ineffective) can be the subject of class discussion to inform subsequent designs.
162
Bridge design challenges similar to the E4C lesson are limiting to student learning because there is only one criteria to achieve—to hold a weight. The designs were not evaluated in any other ways, so a bridge made of 200 popsicle sticks would be considered equal to one that used 20. Providing multiple criteria, including cost, presents a wider range of possibilities for results and for learning through the improvement process. But the goal of engineering is not to build the “perfect” design—it is to construct a solution that works in that context and balances trade-offs in a way that satisfies the client. Not only do projects with multiple criteria provide more realistic representations of professional practice, they also enable every student to learn through improvement because there is no such thing as an objective “success.”
Of the four causes of failure I noted, two are more preventable. Increasing students’ understanding of science and technology concepts and of material properties as they relate to the project are important in avoiding some failures. A common example in the bridge designs was the inability of students to recognize that the decking was not rigid. While that is not the only aspect that can cause failure in bridge designs, it led to many failed designs in my data set.
Students would have benefitted from investigations of how to make their materials more rigid.
To teachers. In addition to curricular design, there are specific things teachers can do to support learning from failure in these engineering projects. A teacher might consider diagnosing the reason(s) for students’ designs failure. Engaging those students in interactive, dialogic conversation (Mortimer & Scott, 2003) will better help them attribute the cause to factors that are able to be improved in their next design. Participating in these projects just because they are
“hands-on” may lead to “activity without learning” (Banilower, et al., 2006). The feedback teachers give is also likely to have a significant impact on the actions of the students, so teachers
163 should consider the role they are playing at the time and how their feedback can support (or constrain) learning and improvement.
The importance of low-stakes testing in improvement cannot be overstated. Teachers might encourage students to frequently test early prototypes to inform their final designs.
However, to avoid an endless cycle of “guess-and-check” methodology, students that engage in thoughtful analyses of the performance during these low-stakes tests will likely learn more than those who do not.
Public tests including the whole class are not efficient in terms of class time. But students will benefit from others’ designs in two ways. First, they will learn from others’ failure, particularly if teachers use that failure as a point of discussion. Second, students will learn from others’ clever design features. Despite our initial inclination that incorporating other people’s ideas into our design is cheating, it actually represents an opportunity for students to innovate on previous designs by innovating existing solutions. These discussions may be facilitated by the teachers in order for these opportunities to be realized, but discussions may be student-driven.
Study Limitations
The study of failure in engineering design presents a number of challenges to be addressed. The small sample size of this investigation and the specific context of the students and teachers engaged in particular classroom activities prevents making claims for all of engineering.
Just as classroom discourse is context-dependent, so are the claims put forth here.
First, I never served as a participant observer (Spradley, 1980) during data collection, a key element of ethnographic methodology, and only used video and student artifact data that
164 were collected for me. Although I could get a sense of the classroom culture through analysis of several hours of classroom video, I was unable to get to know the teachers and students personally. This could have affected the ways in which I interpreted classroom discourse.
Although using only video and student journal data and avoiding interviews prevented adding an additional layer of discourse from the teachers answering questions about failure, this study would have benefitted in some ways to the ability to dig deeper into the thoughts and intentions of the teachers when they taught these engineering units. For example, it would have been helpful to hear the teachers’ perspectives on why they forbid low-stakes testing.
There were also two potential limitations in my sampling. First, the nature of failure in other engineering fields may limit the generalizability of the concepts found here. This threatens the validity of the research because civil engineering may be unique due to the nature of the failures. For the same reasons I chose civil engineering units to study—because their failures are easier to recognize—my constructs may not be generalizable. Second, the teachers in this study were held accountable to teach the units as closely to the curriculum guide as possible, and this was documented by researchers and through the use of an implementation log. Thus, there is a potential for other teachers using EiE that will skip the preliminary activities, opting only for the design challenge. This alteration will likely lead to differences in student designs and lead to results contradicting those found in this study. Ultimately, in this single-researcher study, the interpretation of failure was almost exclusively mine. Although I made every attempt to ground my interpretations within the interactions of students and teachers, what counted as failure was determined by my understanding of it.
165
Aware of these challenges, I made intentional choices to increase validity in my methodology. First, I created and documented an audit trail (Creswell & Miller, 2000) for my work to be transparent for friendly skeptics. Second, I used prolonged engagement (Creswell &
Miller, 2000) to better understand my data. Rather than only analyzing failure events, I mapped phases and sequences during all parts of the selected lessons. In total, I used 139 hours of video data from eight classrooms and analyzed them over 13 months. Third, my data sources are retrievable records that can be accessed by other researchers. The video recording and sampling procedure were documented in the event maps through extensive time-stamping of the phases and sequences of activity indexing the video record. Other researchers can use these data and can serve as checks on my methodology and findings.
Future Directions
Related to the limitations above, I think three lines of future research based on failure and improvement in engineering design will add to the research of this study. The first considers the generalizability of the models for failure and improvement. The second considers expert/novice studies and the epistemic practices engineers use both in similar classroom design projects and in professional work. And the third considers teaching practices that can be applied to many engineering situations.
First, classroom projects in civil engineering create a context for failure that is not true of all fields. While it is easy to determine that a bridge fails because it collapses or a circuit fails because the bulb does not light, some fields have less clear results. For example, the EiE: A Slick
Solution is an environmental engineering unit in which students plan and implement a process for cleaning up an oil spill. The success of the process is judged against a rubric that determines
166 the “environmental impact score” based on the number of oil drops measured in the water and on the “beach.” Presumably, failure or success will be socially constructed and will be of the subjective referent rather than objective because the score has to be interpreted. Considering the ways that students and teachers acknowledge and respond to failure in contexts like these will be an interesting investigation to consider failures in contexts that are much different than the objective failures of civil engineering.
Second, the NGSS (NGSS Lead States, 2013) suggests that students engage in practices of engineers to do classroom projects, and scholars in the field who suggest there are sixteen epistemic practices of engineering, including persisting and learning from failure (Cunningham
& Carlsen, 2014b; Cunningham & Kelly, in review). I would like to investigate the ways that disciplinary experts engage in these classroom projects similar to the study of Crismond (2001) that examined how expert and novices engaged in redesigning simple mechanical devices. One comparison that could be made would be between the practices that civil or structural engineers engage in to design bridges to compare their activities and interactions. Clearly, professional engineers have a wealth of experience and knowledge with which to draw on, but activities like those in EiE units are likely to be engaging enough to get a better understanding of how an engineers approach the problem and design solutions. Then, we can begin to better understand what these expert practices look like in classroom settings, rather than trying to imply them from professional work and impose them on classroom work.
Third, as a teacher educator and a professional development provider, I am keenly aware of the challenges in helping teachers teach complex disciplinary content using the practices of scientists/engineers, particularly for teachers with limited experience with these practices. I am
167 also a proponent of using engineering design contexts to teach both science and engineering content (Johnson, et al., in review). Helping to develop a series of teacher practices related to using engineering design projects in the classroom, particularly related to iteration and improvement from failure interests me. Windschitl, and others (2012) have developed “high- leverage” practices that are meant to be applicable across many science teaching contexts. A similar set of teaching practices in engineering might enable teachers to teach engineering projects effectively without requiring unit-specific training on each module.
The high-leverage practices described in Windschitl, et al. (2012) are not directly applicable to engineering because the goal of the endeavors are different (Cunningham &
Carlsen, 2014b). Rather than the goal of explaining, predicting, or modeling a phenomenon, engineering designers attempt to solve a problem. However, in some ways these practices can be thought of in similar ways. Engineering projects should be based on big ideas, and students and teachers should engage in discourses to make sense of their designs, to use evidence to inform design, and in justifying design decisions. The development of these practices, and “back pocket questions” (Windschitl, et al., 2012) will better enable teachers to support student groups during their design, test, and redesign activities.
Failure in engineering design is an interesting phenomenon to consider by the field of
STEM education. Its characterization as an epistemic practice that is significantly different in engineering leads to several interesting studies comparing scientists and engineers and comparing novices and experts. And a better understanding of failure and ways in which to support students through the process of improvement is timely given the recent adoption of engineering in the NGSS (NGSS Lead States, 2013).
168
Works Cited Abramson, L. Y., Seligman, M. E. P., & Teasdale, J. (1978). Learned helplessness in humans: critique and reformulation. Journal of Abnormal Psychology, 87, 49–74. Allchin, D. (2001). Error types. Perspectives on Science, 9(1), 38-58. Allchin, D. (2003). Scientific myth‐conceptions. Science Education, 87(3), 329-351. Allchin, D. (2012). A Comedy of Scientific Errors. The American Biology Teacher, 74(7), 530-531. Allchin, D. (2012). Teaching the nature of science through scientific errors. Science Education, 96(5), 904-926. Allen, M. S. (2010). On the temporal dynamics of causal attribution in competitive sport. International Review of Sport and Exercise Psychology, 3(1), 3-23. Allen, M. S., Coffee, P., & Greenlees, I. (2012). A theoretical framework and research agenda for studying team attributions in sport. International Review of Sport and Exercise Psychology, 5(2), 121-144. Augustine, N. R. (2005). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Retrieved March, 19, 2008. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191-215. Banilower, E. R., Smith, P. S., Weiss, I. R., & Pasley, J. D. (2006). The Status of K-12 Science Teaching in the United States. In D. W. Sunal & E. L. Wright (Eds.), The impact of state and national standards on K-12 science teaching (p. 440). IAP. Banilower, E. R., Smith, P. S., Weiss, I. R., Malzahn, K. A., Campbell, K. M., & Weis, A. M. (2013). Report of the 2012 national survey of science and mathematics education. Chapel Hill, NC: Horizon Research, Inc. Banilower, E. R., Smith, P. S., Weiss, I. R., & Pasley, J. D. (2006). The status of K-12 science teaching in the United States. In D. W. Sunal & E. L. Wright (Eds.), The impact of state and national standards on K-12 science teaching (pp. 83–122). United States of America: Information Age Publishing, Inc. Barron, B., & Darling-Hammond, L. (2010). Prospects and challenges for inquiry-based approaches to learning. In H. Dumont, D. Istance, & F. Benavides (Eds.), The Nature of Learning: Using Research to Inspire Practice., Educational Research and Innovation (pp. 199–225). Paris: OECD. Retrieved from
http://www.sourceoecd.org/education/97892264086470 Barron, B., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., Bransford, J. D., et al. (1998). Doing with understanding: Lessons from research on problem-and project- based learning. Journal of the Learning Sciences, 271–311. Bažant, Z. P., & Zhou, Y. (2001). Why Did the World Trade Center Collapse?—Simple Analysis. International Journal of Structural Stability and Dynamics, 1(04), 603-615. Bazerman, C. (2006). Analyzing the multidimensionality of texts in education. Handbook of complementary methods in education research, 77-94. Benner, P., (2000). The tradition and skill of interpretive phenomenology in studying health, illness, and caring practices. In: Benner, P. (Ed.), Interpretive Phenomenology. Embodiment, Caring, and Ethics in Health and Illness. Publications, Thousand Oaks, pp. 99–127. Sage
169
Beyer, S. (1990). Gender differences in the accuracy of self-evaluations of performance. Journal of personality and social psychology, 59(5), 960. Blanchard, M. R., Southerland, S. A., Osborn, J. W., Sampson, V. D., Annetta, L. A., & Granger, E. M. (2010). Is inquiry possible in light of accountability? A quantitative comparison of the relative effectiveness of guided inquiry and traditional verification laboratory instruction. Science Education, 94(4), 577–616. Blumenfeld, P., Soloway, E., Marx, R., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning. Educational Psychologist, 26(3), 369–398. Bond, K. A., Biddle, S. J., & Ntoumanis, N. (2001). Self-efficacy and causal attribution in female golfers. International Journal of Sport Psychology, 32(3), 243-256. Brooks, R. B. (1994). Children at risk: fostering resilience and hope. American Journal of Orthopsychiatry, 64(4), 545. Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering education in P‐12 classrooms. Journal of Engineering Education, 97(3), 369-387. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The journal of the learning sciences, 2(2), 141-178. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational researcher, 18(1), 32-42. Burnett, P. C. (2002). Teacher praise and feedback and students' perceptions of the classroom environment. Educational psychology, 22(1), 5-16. Busby, J. S. (2001). Error and distributed cognition in design. Design studies, 22(3), 233-254. Caelli, K., 2000. The changing face of phenomenological research: traditional and American phenomenology in nursing. Qualitative Health Research 10, 366–377. Cajas, F. (2001). The science/technology interaction: Implications for science literacy. Journal of research in science teaching, 38(7), 715-729. Cantrell, P., Pekcan, G., Itani, A., & Velasquez-Bryant, N. (2005, October). Using engineering design curriculum to close science achievement gaps for middle school students. In Frontiers in Education, 2005. FIE'05. Proceedings 35th Annual Conference (pp. S1F- S1F). IEEE. Cantrell, P., Pekcan, G., Itani, A., & Velasquez-Bryant, N. (2005, October). Using engineering design curriculum to close science achievement gaps for middle school students. In Frontiers in Education, 2005. FIE'05. Proceedings 35th Annual Conference (pp. S1F- S1F). IEEE. Carlsen, W. S. (1991). Effects of new biology teachers' subject‐matter knowledge on curricular planning. Science Education, 75(6), 631-647. Carlsen, W.S. & Johnson M.M. (2014) Elementary science learning through engineering design: Effects explored using HLM. A paper presented at the Annual NARST conference. April, 2014. Carr, R. L., & Strobel, J. (2011). Integrating Engineering Design Challenges into Secondary STEM Education. National Center for Engineering and Technology Education. Chan, L. K. (2006). Combined strategy and attributional training for seventh grade average and poor readers. Journal of Research in Reading, 19(2), 111-127.
170
Chandler, J., Fontenot, A. D., & Tate, D. (2011). Problems associated with a lack of cohesive policy in K-12 pre-college engineering. Journal of Pre-College Engineering Education Research (J-PEER), 1(1), 5. Chesler, N. C., Arastoopour, G., D’Angelo, C. M., Bagley, E. A., & Shaffer, D. W. (2013). Design of a Professional Practice Simulator for Educating and Motivating First-Year Engineering Students. Advances in Engineering Education, 3(1), 1-29. Chiu, J. L., & Linn, M. C. (2011). Knowledge integration and WISE engineering. Journal of Pre-College Engineering Education Research (J-PEER), 1(1), 2. Cope, J. (2011). Entrepreneurial learning from failure: An interpretative phenomenological analysis. Journal of Business Venturing, 26(6), 604-623. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five designs. Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into practice, 39(3), 124-130. Crismond, D. (2001). Learning and using science ideas when doing investigate‐and‐redesign tasks: A study of naive, novice, and expert designers doing constrained and scaffolded design work. Journal of Research in Science Teaching, 38(7), 791-820. Crotty, M., 1996. Phenomenology and Nursing Research Churchill Livingstone, Melbourne. Cunningham, C. M. (2009). Engineering is elementary. The Bridge, 30(3), 11-17. Cunningham, C. M., & Carlsen, W. S. (2014a). Teaching Engineering Practices. Journal of Science Teacher Education, 25(2), 197-210. Cunningham, C. M., & Carlsen, W. S. (2014b). Teaching Engineering Practices. Journal of Science Teacher Education, 25(2), 197-210. Cunningham, C.M. and Carlsen W.S. (2014). Precollege engineering education. Handbook of Research on Science Education, Vol. II. Lederman, N.G. and Abell, S.K. (eds). Routledge. Cunningham, C.M. & Kelly, G.J. (2015). Framing engineering practice in elementary school classrooms. Paper presented at the Annual NARST Conference, April 2015. Dieser, R. B., & Ruddell, E. (2002). Effects of attribution retraining during therapeutic recreation on attributions and explanatory styles of adolescents with depression. Therapeutic Recreation Journal, 36(1), 35-47. Dopplet, Y., Mehalik, M., Schunn, C., Silk, E., & Krysinski, D. (2008). Engagement and achievements: A case study of design-based learning in a science context. Journal of Technology Education, 19(2), 21-38. Dowling, M. (2007). From Husserl to van Manen. A review of different phenomenological approaches. International journal of nursing studies, 44(1), 131-142. Driver, R., Squires, A., Rushworth, P.,&Wood-Robinson,V. (1994).Making sense of secondary science: Research into children’s ideas. London: Routledge. Duschl, R. A. (1983). The elementary level science methods course: Breeding ground of an apprehension toward science? A case study. Journal of Research in Science Teaching, 20(8), 745-754.
171
Duschl, R. A. (2008). Science education in three-part harmony: Balancing conceptual, epistemic, and social learning goals. Review Of Research In Education, 32(1), 268– 291. Duschl, R. A., Schweingruber, H. A., & Shouse, A. W. (2007). Taking science to school. Learning and teaching science in grades K-8. Washington, DC: National Academies Press. Dweck, C. S. (1986). Motivational processes affecting learning. American psychologist, 41(10), 1040. Dyson, J., & Coren, G. (2001). Against the odds: an autobiography. London: Texere. Eck, N. (2010). Kinzua Bridge. https://failures.wikispaces.com/Kinzua+Bridge. Retrieved 1/22/16. Elliott, K. (2004). Error as Means to Discovery*. Philosophy of Science, 71(2), 174-197. Erickson, F. (1992). Ethnographic microanalysis of interaction. In M. D. LeCompte, W. L. Milroy, & J. Preissle (Eds.), The handbook of qualitative research in education (pp. 202-224). San Diego: Academic. Festinger, L. (1962). A theory of cognitive dissonance (Vol. 2). Stanford university press. Firestein, S. (2015). Failure: Why Science Is So Successful. Oxford University Press. Fleming, V., Gaidys, U. and Robb, Y. 2003. Hermeneutic research in nursing: developing a Gadamerian-based research method. Nursing Inquiry, 10(2): 113–120 Follette, V. M., & Jacobson, N. S. (1987). Importance of attributions as a predictor of how people cope with failure. Journal of personality and social psychology, 52(6), 1205. Fortune, J., & Peters, G. (1995). Learning from failure-the systems approach. Wiley. Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., & Mamlok‐Naaman, R. (2004). Design‐ based science and student learning. Journal of Research in Science Teaching, 41(10), 1081-1110. Gadamer, G.H., 1989. Truth and Method, second ed. Sheed and Ward, London [translation revised by Weinsheimer J and Marshall DG]. Galli, N., & Vealey, R. S. (2008). Bouncing back” from adversity: Athletes’ experiences of resilience. The Sport Psychologist, 22(3), 316-335. Goodman, I. F., Cunningham, C. M., & Lachapelle, C. (2002). Final report of theWomen's Experiences in College Engineering (WECE) Project: Executive summary Green, J. L., & Stewart, A. (2005). A brief history of linguistic perspectives in qualitative research in education. Handbook of qualitative research in education. Cheltenham: Edward Elgar Ltd. Green, J. L., & Wallat, C. (1981). Mapping instructional conversations: A sociolinguistic ethnography. Ethnography and language in educational settings, 5, 161-195. Gumperz, J. J. (1982). Discourse strategies (Vol. 1). Cambridge University Press. Gumperz, J. J. (2001). Interactional sociolinguistics: A personal perspective. In D. Schiffrin, D. Tannen, & H. E. Hamilton (Eds.), Handbook of discourse analysis . Malden: Blackwell.
172
Gyllenpalm, J, & Wickman, P. (2011). The uses of the term hypothesis and the inquiry emphasis conflation in science teacher education. … Journal of Science Education, 33(14), 1993–2015. Hamza, K. M., & Wickman, P.-O. (2008). Describing and Analyzing Learning in Action : An Empirical Study of the Importance of Misconceptions in Learning Science. Science Education, 92(1), 141 – 164.
Harris, C. J., & Rooks, D. L. (2010). Managing Inquiry-Based Science: Challenges in Enacting Complex Science Instruction in Elementary and Middle School Classrooms. Journal of Science Teacher Education, 21(2), 227–240. Harris, J., & Felix, A. (2010, March). A project-based, STEM-integrated team challenge for elementary and middle school teachers in alternative energy. In Society for Information Technology & Teacher Education International Conference (Vol. 2010, No. 1, pp. 3566-3573). Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112. Hegedus. T. (2015). Cultivating creative practices in elementary engineering classrooms. A paper presented at the Annual NARST conference, April, 2015. Henderson, N., & Milstein, M. M. (2003). Resiliency in schools: Making it happen for students and educators. Corwin Press. Human, T. E. I. (1985). by Henry Petroski. St. Martin's Press. Hynes, M. M. (2009). Teaching middle-school engineering: An investigation of teachers' subject matter and pedagogical content knowledge (Doctoral dissertation, TUFTS UNIVERSITY). Jeffers, A. T., Safferman, A. G., & Safferman, S. I. (2004). Understanding K-12 engineering outreach programs. Journal of professional issues in engineering education and practice, 130(2), 95-108. Johnson, M.M., Croom-Perez, T., Perez, A.A., Tekeley, C., Edelman, R. (in review). From Fish Tank to Fuel Tank: Engineering Photobioreactors in the Classroom. Jussim, L., Coleman, L., & Nassau, S. (1987). The influence of self-esteem on perceptions of performance and feedback. Social Psychology Quarterly, 95-99. Kafle, N. P. (2013). Hermeneutic phenomenological research method simplified. Bodhi: An Interdisciplinary Journal, 5(1), 181-200. Kanter, D. E., & Konstantopoulos, S. (2010). The impact of a project based science curriculum on minority student achievement, attitudes, and careers: The effects of teacher content and pedagogical content knowledge and inquiry based practices. Science Education, 94, 855–887. Kapur, M. (2014). Comparing learning from productive failure and vicarious failure. Journal of the Learning Sciences, 23(4), 651-677. Katehi, L., Pearson, G., & Feder, M. (2009). Engineering in K-12 education. Committee on K- 12 Engineering Education, National Academy of Engineering and National Research Council of the National Academies.
173
Katehi, L., Pearson, G., & Feder, M. (2009). Engineering in K-12 education. Committee on K- 12 Engineering Education, National Academy of Engineering and National Research Council of the National Academies. Kelley, T. R., Brenner, D. C., & Pieper, J. T. (2010). Two approaches to engineering design: Observations in sTEm education. Kelly, G.J. (2004). Discourse and epistemology in science education. Interim report for National Academy of Education Postdoctoral Fellowship 1998-2000. Kelly, G. J. (2008). Inquiry, activity, and epistemic practice. In R. A. Duschl & R. E. Grandy (Eds.), Teaching scientific inquiry: Recommendations for research and implementation (pp. 99–117; 288–291). Rotterdam, the Netherlands: Sense Publishers. Kelly, G. J. (2011). Scientific literacy, discourse, and epistemic practices. In C. Linder, L. Ostman, D. A. Roberts, P.-O. Wickman, G. Erickson, & A. MacKinnon (Eds.), Exploring the landscape of scientific literacy (pp. 61–73). New York, NY: Routledge Kelly, G. J. (2014). Analyzing Classroom Activities: Theoretical and Methodological Considerations. In Topics and Trends in Current Science Education (pp. 353-368). Springer Netherlands. Kelly, G. J., & Brown, C. M. (2003). Communicative demands of learning science through technological design: Third grade students’ construction of solar energy devices. Linguistics & Education, 13(4), 483-532. Kelly, G.J. & Cunningham, C. M. (submitted) Epistemic practices of engineering in education. Kelly, G. J., & Chen, C. (1999). The sound of music: Constructing science as sociocultural practices through oral and written discourse. Journal of Research in Science Teaching, 36, 883-915 Kelly, G. J., Chen, C., & Prothero, W. (2000). The epistemological framing of a discipline: Writing science in university oceanography. Journal of Research in Science Teaching, 37 , 691–718. Kelly, G. J., Crawford, T., & Green, J. (2001). Common tasks and uncommon knowledge: Dissenting voices in the discursive construction of physics across small laboratory groups. Linguistics & Education, 12(2), 135-174. Kind, P. M., Kind, V., Hofstein, A., & Wilson, J. (2011). Peer Argumentation in the School Science Laboratory—Exploring effects of task features. International Journal of Science Education, 33(18), 2527-2558. Kolodner, J. L. (2002). Design Research on Learning Environments. Learning by Design: Iterations of Design Challenges for Better Learning of Science Skills. Cognitive Studies, 9(3), 338-350. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., et al. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting Learning by DesignTM into practice. Journal of the Learning Sciences, 12(4), 495–547. Kolodner, JK (2006). Case-based Reasoning, in Sawyer, R. K. (Ed.). The Cambridge handbook of the learning sciences (Vol. 2, No. 5). New York: Cambridge University Press.
174
Krajcik, J. S., McNeill, K. L., & Reiser, B. J. (2008). Learning-goals-driven design model: Developing curriculum materials that align with national standards and incorporate project-based pedagogy. Science Education, 92(1), 1–32. Kranz, G. (2001). Failure is not an option: Mission control from Mercury to Apollo 13 and beyond. Simon and Schuster. Lachapelle, C. P., & Cunningham, C. M. (2014). Engineering in elementary schools. Engineering in pre-college settings: Synthesizing research, policy, and practices. Lafayette, IN: Purdue Univ. Lachapelle, C. P., Cunningham, C. M., Lee-St. John, T. J., Cannady, M., & Keenan, K. (2010). An investigation of how two Engineering is Elementary curriculum units support student learning. Presented at the P-12 Engineering and Design Education Research Summit, Seaside, OR. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge university press. Laverty, S. M. (2003). Hermeneutic phenomenology and phenomenology: A comparison of historical and methodological considerations. International Journal of Qualitative Methods, 2(3), 21–35. Lawanto, O., & Stewardson, G. (2013). Students’ interest and expectancy for success while engaged in analysis-and creative design activities. International Journal of Technology and Design Education, 23(2), 213-227. Le Foll, D., Rascle, O., & Higgins, N. C. (2008). Attributional feedback-induced changes in functional and dysfunctional attributions, expectations of success, hopefulness, and short-term persistence in a novel sport. Psychology of Sport and Exercise, 9(2), 77-101. Lemke, J. L. (1990). Talking science: Language, learning, and values. Ablex Publishing Corporation, 355 Chestnut Street, Norwood, NJ 07648 (hardback: ISBN-0-89391-565- 3; paperback: ISBN-0-89391-566-1).. Lewis, T. (2006). Design and inquiry: Bases for an accommodation between science and technology education in the curriculum?. Journal of Research in Science Teaching, 43(3), 255-281. Levy, S. T. (2013). Young children’s learning of water physics by constructing working systems. International Journal of Technology and Design Education, 23(3), 537-566. Lottero-Perdue, P. S. (2015, April). The engineering design process as a safe place to try again: Responses to failure by elementary teachers and students. Presented at the NARST, Chicago, IL. Lottero-Perdue, P.S, & Parry, E.A. (2014, June). Perspectives on Failure in the Classroom by Elementary Teachers New to Teaching Engineering. Presented at the 121st ASEE Annual Conference & Exposition, Indianapolis, IN. Lottero-Perdue, P. S. & Parry, E. A. (2015, June). Elementary teachers’ reported responses to student design failures. Presented at the American Society of Engineering Education Annual Conference & Exposition, Seattle, WA. Loucks-Horsley, S., & Olson, S. (Eds.). (2000). Inquiry and the National Science Education Standards: A Guide for Teaching and Learning. National Academies Press. Maddi, S. R., Matthews, M. D., Kelly, D. R., Villarreal, B., & White, M. (2012). The role of hardiness and grit in predicting performance and retention of USMA cadets. Military Psychology, 24(1), 19.
175
Manz, E. (2012). Understanding the codevelopment of modeling practice and ecological knowledge. Science Education, 96(6), 1071-1105. Matson, J. V. (1996). Innovate or die: A personal perspective on the art of innovation. Paradigm Press (Monroe, WI). Mariage, C.S. Englert, M. A. Garmon, T. (2000). The teacher as" more knowledgeable other" in assisting literacy learning with special needs students. Reading & Writing Quarterly, 16(4), 299-336. Martin, A. J., & Marsh, H. W. (2003). Fear of failure: Friend or foe?. Australian Psychologist, 38(1), 31-38. Martin-Krumm, C. P., Sarrazin, P. G., Peterson, C., & Famose, J. P. (2003). Explanatory style and resilience after sports failure. Personality and individual differences, 35(7), 1685- 1695. Maxwell, J. A. (2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational researcher, 33(2), 3-11. Mueller, C. M., & Dweck, C. S. (1998). Praise for intelligence can undermine children's motivation and performance. Journal of personality and social psychology, 75(1), 33.
Muller, C. B., & Collier, J. P. (1995, November). Engineering concepts in the high school classroom: The Dartmouth/Thayer problem-solving methods. In Frontiers in Education Conference, 1995. Proceedings., 1995 (Vol. 2, pp. 4b1-15). IEEE. Mummery, W. K., Schofield, G., & Perry, C. (2004). Bouncing back: the role of coping style, social support and self-concept in resilience of sport performance. Athletic Insight, 6(3), 1-15. National Academy of Engineering (NAE). Committee on Public Understanding of Engineering Messages. (2008). Changing the conversation: Messages for improving public understanding of engineering. National Academies Press. National Research Council (Ed.). (1996). National science education standards. National Academy Press. National Research Council, Schweingruber, H., Keller, T., & Quinn, H. (Eds.). (2012). A Framework for K-12 Science Education:: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press. Nathan, C. Kinzua Viaduct (July, 21, 2003). 2010; Available from: http://failures.wikispaces.com/Kinzua+Bridge. Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, 37(4), 375-401. Nersessian, N. J. (2012). Engineering concepts: The interplay between concept formation and modeling practices in bioengineering sciences. Mind, Culture, and Activity, 19(3), 222- 239. NGSS Lead States. (2013). Next Generation Science Standards: For States, By States. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in higher education, 31(2), 199-218.
176
Ochs, E. (1979). Transcription as theory. In E. Ochs & B.B. Schieffelen (Eds.), Developmental pragmatics ( pp. 43-72). New York: Academic Press. Palmer, P., Gillette, G., Shea, S., Silverman, M., & Spira, G. (Eds.). (2006). The 2006 ESPN Baseball Encyclopedia. Sterling Publishing Company, Inc.. Partnership for 21st Century Skills. (2009). Framework for 21st century learning. Washington, DC: Author. Patton, M. (1990). Qualitative evaluation and research methods. Beverly Hills, CA: Sage. 169- 186. Penner, D. E., Giles, N. D., Lehrer, R., & Schauble, L. (1997). Building functional models: Designing an elbow. Journal of Research in Science Teaching, 34(2), 125–143. Perry, R. P., & Penner, K. S. (1990). Enhancing academic achievement in college students through attributional retraining and instruction. Journal of Educational Psychology, 82(2), 262. Perspectives on Failure in the Classroom by Elementary Teachers New to Teaching Engineering (Research to Practice) Petroski, H. (1985). To engineer is human: The role of failure in successful design. New York: Vintage books. Petroski, H. (1994). Design paradigms: Case histories of error and judgment in engineering. Cambridge University Press. Petroski, H. (2012). To forgive design: Understanding failure. Harvard University Press. Polkinghorne, D. (1983). Methodology for the human sciences: Systems of inquiry. Albany: State University of New York Press. Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science education, 66(2), 211-227. Price, M., Handley, K., Millar, J., & O'Donovan, B. (2010). Feedback: all that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277-289. Purzer, Ş., Moore, T. J., Baker, D., & Berland, L. Supporting the Implementation of NGSS through Research. Retrieved from https://narst.org/ngsspapers/Engineering_June2014.pdf Purzer, Ş., Moore, T. J., Baker, D., & Berland, L. Supporting the Implementation of NGSS through Research.Purzer, Ş., Moore, T. J., Baker, D., & Berland, L. Supporting the Implementation of NGSS through Research. (NRST) Racher, F., Robinson, S., 2003. Are phenomenology and postpostivism strange bedfellows? Western Journal of Nursing Research 25 (5), 464–481. Redmond, A., Thomas, J., High, K., Scott, M., Jordan, P., & Dockers, J. (2011). Enriching science and math through engineering. School Science and Mathematics, 111(8), 399- 408. Richardson, G. E. (2002). The metatheory of resilience and resiliency. Journal of clinical psychology, 58(3), 307-321. Richardson, G. E., Neiger, B. L., Jensen, S., & Kumpfer, K. L. (1990). The resiliency model. Health Education, 21(6), 33-39.
177
Robardey, C. P., Allard, D. W., & Brown, D. M. (1994). An assessment of the effectiveness of full option science system training for third-through sixth-grade teachers. Journal of Elementary Science Education, 6(1), 17-29. Rodgers Commission. (1986). Investigation of the Challenger Accident. Tech. rep. US House Committee on Science and Technology. Roehrig, G. H., Moore, T. J., Wang, H. H., & Park, M. S. (2012). Is Adding the E Enough? Investigating the Impact of K‐12 Engineering Standards on the Implementation of STEM Integration. School Science and Mathematics, 112(1), 31-44. Ross, R. B. (1995). Investigating mechanical failures: the metallurgist's approach. Springer. Roth, W. M. (1996). Learning to talk engineering design: Results from an interpretive study in a grade 4/5 classroom. International Journal of Technology and Design Education, 6(2), 107-135. Roth, W. M. (2001). Learning science through technological design. Journal of Research in Science Teaching, 38(7), 768-790. Sadler, D. R. (2010). Beyond feedback: Developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535-550. Sadler, P. M., Coyle, H. P., & Schwartz, M. (2000). Engineering competitions in the middle school classroom: Key elements in developing effective design challenges. The Journal of the Learning Sciences, 9(3), 299-327. Sarkar, M., & Fletcher, D. (2014). Psychological resilience in sport performers: a review of stressors and protective factors. Journal of sports sciences, (ahead-of-print), 1-16. Schunk, D. H., & Pajares, F. (2002). The development of academic self-efficacy. In A. Wigfield & J. S. Eccles (Eds.), Development of academic motivation (pp. 15–31). San Diego: Academic Press. Seligman, M. E. P. (1975). Helplessness: On depression, development, and death. San Francisco: WH Freemsn. Seligman, M. E., & Fowler, R. D. (2011). Comprehensive Soldier Fitness and the future of psychology. American Psychologist, 66(1), 82. Seligman, M. E., Nolen-Hoeksema, S., Thornton, N., & Thornton, K. M. (1990). Explanatory style as a mechanism of disappointing athletic performance. Psychological Science, 1(2), 143-146. Sezen, A., Tran, M.-D. T., McDonald, S., & Kelly, G. J. (2009). Preservice science teachers’ refl ections upon their micro- teaching experience: An activity theory perspective . Paper presented at the annual meeting of the National Association for Research in Science Teaching, Garden Grove. Silverman, H., 1987. Inscriptions: Between Phenomenology and Structuralism. Routledge Kegan Paul, New York. Spradley, J. P. (1980). Participant observation. Orlando, FL: Harcourt Brace Jovanovich College Publishers. Starks, H., & Trinidad, S. B. (2007). Choose your method: A comparison of phenomenology, discourse analysis, and grounded theory. Qualitative health research, 17(10), 1372- 1380.
178
Stefanich, G. P., & Kelsey, K. W. (1989). Improving science attitudes of preservice elementary teachers. Science Education, 73, 187–194. Stronge, J. H., Ward, T. J., & Grant, L. W. (2011). What makes good teachers good? A cross- case analysis of the connection between teacher effectiveness and student achievement. Journal of Teacher Education, 62(4), 339-355. Suchman, L. (2000). Embodied practices of engineering work. Mind, Culture, and activity, 7(1-2), 4-18. Svarovsky, G. N. (2011). Exploring Complex Engineering Learning Over Time with Epistemic Network Analysis. Journal of Pre-College Engineering Education Research (J-PEER), 1(2), 4. Svarovsky, G. N., & Shaffer, D. W. (2006, October). Design meetings and design notebooks as tools for reflection in the engineering design course. In Frontiers in Education Conference, 36th Annual (pp. 7-12). IEEE. Technotronics Program. (2016). Hands-on Activity: Balsa Towers. Pratt School of Engineering, Duke University. Retrieved from https://www.teachengineering.org/view_activity.php?url=collection/duk_/activities/duk _balsa_tech_act/duk_balsa_tech_act.xml. Thomas, J. (2000). A review of research on project-based learning. Autodesk Foundation, 2000. Retrieved from internal-pdf://Thomas 2000 Project-based learning- 0264231936/Thomas 2000 Project-based learning.pdf van Manen, M. (1997). Researching lived experience: Human science for an action sensitive pedagogy (2nd ed.). Canada: The Althouse Press.
Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: the kappa statistic. Fam Med, 37(5), 360-363. Wang, H. H., Moore, T. J., Roehrig, G. H., & Park, M. S. (2011). STEM integration: Teacher perceptions and practice. Journal of Pre-College Engineering Education Research (J- PEER), 1(2), 2. Washington Department of Transportation (2005). Tacoma Narrows Bridge Machine. http://www.wsdot.wa.gov/TNBhistory/Machine/machine3.htm. Retrieved 1/22/16. Weinberg, R., Butt, J., & Culp, B. (2011). Coaches' views of mental toughness and how it is built. International journal of sport and exercise psychology, 9(2), 156-172. Weiner, B. (1985). Human motivation: Metaphors, theories and research. Newbury Park, CA: Sage. Weiss, I. R. (1978). Report of the 1977 National Survey of Science, Mathematics, and Social Studies Education. Washington, DC: U.S. Government Printing Office. Weiss, I. R., Knapp, M. S., Hollweg, K. S., & Burrill, G. (Eds.). (2001). Investigating the influence of standards: A framework for research in mathematics, science, and technology education. National Academies Press. Werner, E.E. (1982). Vulnerable but invincible:Alongitudinal study of resilient children and youth. New York: McGraw-Hill. Wilson, R. N. (1990). 'Matching error'(spherical aberration) in the Hubble Space Telescope (HST)-Some technical comments. The Messenger, 61, 22-24.
179
Windschitl, M. (2004). Folk theories of “inquiry:” How preservice teachers reproduce the discourse and practices of an atheoretical scientific method. Journal of Research in Science Teaching, 41(5), 481-512. Windschitl, M., Thompson, J., Braaten, M., & Stroupe, D. (2012). Proposing a core set of instructional practices and tools for teachers of science. Science education, 96(5), 878- 903. Wujec, T. (2010, Feb). Tom Wujec: Build a tower, build a team. [Video file]. Retrieved from https://www.ted.com/talks/tom_wujec_build_a_tower?language=en Zubrowski, B. (2002). Integrating Science into Design Technology Projects: Using a Standard Model in the Design Process. Journal of Technology Education, 13(2), 47-65.
180
Appendix A – Excerpt from an Event Map 181
Appendix B – A Comparison of Event Maps and Student Journals
T14142
T1 Student Journals
Rubric Category Score Evidence – from T14142BrL42d1T1 Score Evidence (from #75998 and #75993)
Failure 1 Stability- “car can’t get over, due to a steep 1 Scored a 2 on both strength and stability incline – but didn’t fall off the side” [line (page 33) 85] & Strength - “bridge fails, aide tells them to count them up [line 88,] Attribution 0 No evidence found 0 No evidence found
Aligned Goals 1 “Car test fails again, so boy adds another 1 [we will improve] strength and stability (pg piece to level the deck” [line 118, 34) T141412BrL42D1T1] Aligned Strategy 1 “Lenny still rolling car across. He again 0 We will try to make the bridge lower and tries to use an accordion folded paper to stronger (pg 34) level the top of the deck where it gets stuck. Improvement 0 No evidence – “recording ends during 0 Score decreased from 2 to 1 (pg 46) testing” [line 155, T14142BrL42D1T1] Score 3 2 T2 Student Journals
Rubric Category Score Evidence – from T14142BrL42D1T2 Score Evidence (from #76002 and #75999)
Failure 1 Stability – not a failure – “car makes it 2 1 Initial strength score was 1 (pg 46) times in one direction, twice in the other” [line 64] & Stability – “Leo called it a failure even though it didn’t get under the failure line, but it tilted – it only supported 3” [line 66] Attribution 0 No evidence found 1 We had very little support to hold up the middle (pg33) Aligned Goals 1 “Leo flips the bridge over showing three 1 [we will improve] stability – piers that are (?) randomly placed [line 107] Aligned Strategy 1 Places the third pier and adds tape. Then a 1 We will make a suspension to hold up the fourth. [line 119] bridge while we put weights on the span (pg 34) Improvement 0 No evidence – Recording ends without T2 1 Strength score went from 1 to 2 on page 46 re-testing. I’m not sure why. [line 163]
Score 3 5
182
T14164
T1 Student Journals Note there was never a formal test for strength or stability, as directed in the journals. Rubric Category Score Evidence - from T14164BrL42T1 Score Evidence - from Students 74850, 74846, 74840
Failure 1 “First attempt with car collapses whole No Strength [was the weakest] because we bridge. B goes to get more copy paper.” Score never got to test it [line 77] Attribution 0 No immediate discussion about why it fell. [line 77]
Aligned Goals 0 No evidence found
Aligned Strategy 1 “They discuss adding a layer on top of the existing piers (I think they are trying to No Score – No Failure make it more rigid)” [line 79] Improvement 0 “They invert the bridge and tested the car again. It collapses again” [line 81] Score 2
T2 Note there was never a formal test for strength or stability, as directed in the journals. Rubric Category Score Evidence - from T14164BrL42T2 Score Evidence - from Students 74852, 84842, 74838
Failure 1 “Madden….drops [the car] in the middle 1 Neither [score was higher] because it could and it collapses” [line77] only hold up 1 weight and the car can only roll across it once Attribution 0 Ms James asks Reggie why [it failed] and 0 No evidence Reggie told her they were going to strengthen the deck with craft sticks” [line 80] Aligned Goals 1 Ms James asks Reggie why [it failed] and 0 No evidence Reggie told her they were going to strengthen the deck with craft sticks” [line 80] Aligned Strategy 1 Ms James asks Reggie why [it failed] and 0 No evidence Reggie told her they were going to strengthen the deck with craft sticks” [line 80] Improvement 1 They test the bridge with the car as a weight 0 No evidence and although it sags, it can hold the car [line 90] Score 4 1
183
T14143
T1 Student Journals
Rubric Category Score Evidence – from T14143BrL42d1T1 Evidence – from Students 76571, 76555, 76551, 76557
Failure 1 Weight test. Collapses at 8 [line69] 1 Strength score was only 1 [page 46]
Attribution 0 No evidence found 1 Strength [was the lowest score] because we didn’t have enough support underneath [pg 33] Aligned Goals 1 They talk out loud about why stability was 1 [We will improve] Strength [pg 34] the best score – Farrah verbalizes it was flat [line 82] Aligned Strategy 1 They say they will add railings and more 1 We will add railings and more support support under the arch [line 81] under the arch [pg 34]
Improvement 1 Barge and car tests work. Weight test – fails 1 Strength score increased from 1 to 2 on pg at 9 (compared with 8 last test) [line 112]. 46 of S76571 only.
Score 4 5
Improvement Cycle #2
Failure 1 Barge and car tests work. Weight test - fails 1 Strength score only 2 in S76571 and only 1 at 9 [line 112] in the other three [pg 46] Attribution 0 No evidence found 0 No evidence found
Aligned Goals 1 Although there is no evidence, they earned 1 [We will improve] strength [pg 39] a perfect score on stability, one may reasonably infer they are trying to improve strength – especially when viewed in conjunction with their strategies Aligned Strategy 1 …they are planning to change the material 1 We will put straws on the bottom and of the arch from copy paper to index cards papers on the bottom [pg 39] [line Improvement 1 Bridge breaks at 46 [line 160] 1 Score increased from 1 to 2 [pg 46].
Score 4 4
T2 Student Journals
Rubric Category Score Evidence – from T14143BrL42D1T2 Evidence – Students 76552, 76548, 76560, 76547
Failure 1 Stability- on the fourth trial, the car falls off 1 …when we put two weights in our cup it at the last minute, so they get a ¾. [line 64] fell [pg 33] Strength – Bridge collapses at 2 weights [line 65] Attribution 0 No evidence found 0 No evidence found
Aligned Goals 1 Nelson and Robyn are putting straws end to 1 [We will improve] strength [pg 34] end and overlapping them. I think they will use them for decking reinforcement [line 91] Aligned Strategy 1 They test the strength by putting the barge 1 We will change it by putting straws across and other items to test its strength [line 103] the middle [pg 34]
184
Improvement 1 Car test works [line 132]; They held over 1 I think we improved in all areas [pg 38] 100 [weights], so they quit testing Score 4 4
Improvement Cycle #2
T2 T2
Failure 0 They held over 100, so they quit testing – T They experienced no failure, in stability nor wraps up class in strength. This improvement is unscorable. Attribution
Aligned Goals
No Score – No Failure No Score – No Failure Aligned Strategy
Improvement
Score 0 0
185
T14098
T1 Student Journals
Rubric Category Score Evidence – from T14098BrL42d1T1 Score Evidence – Students 80970, 80967, 80961, 80960
Failure 1 Test car – doesn’t work [line 170] 1 …it held 15 weights [pg 33]
Attribution 1 String gives way and boy identifies it [line 1 I think stability was the lowest because the 170] bridge was a little wabuly (sic) [pg 33]
Aligned Goals 1 Rena reads her answer: strength was best 1 [We will improve] stability [pg34] because it held 27, but the car only went across 3 times [line 185] Aligned Strategy 0 Lacey says she’s on page 34; they discuss 1 We will flatten our paper so the car goes the cost score because they used a lot of over [pg34] tape Improvement 1 Car test – “awesome, yea, it made it” – 1 Stability score improved from 0 to 2 [pg. successful all 4 times [line 297]; Test with 46] weight. Scatter weight all over. Ms. Clay agrees they can place the weights anywhere, strategically” Fails at 20 [line298] Score 4 5
Improvement Cycle #2
Failure 1 Strength - Fails at 20 [line298] 1 Strength score was 1 and stability score was 2 [pg. 46] Attribution 0 Rena brings them two sheets of copy paper 0 No evidence found and they open their journals. There is little discussion. They seem to be drawing a redesign. Aligned Goals 0 They argue about how to use the index card. 1 [We will improve] strength [pg34] I think the idea is to tape the index card to the underside of the deck to make it rigid. They don’t seem to agree [line 353] Aligned Strategy 0 Nealon wants to use straws, but isn’t sure 1 I will improve my bridge by adding more where [line 358] strength and adding a strong pillar [pg 34]
Improvement 0 Fails at 15 because the weights fell off the 0 Strength score remained at 1 side. Rena: “15? That’s worse than the first one!” [line 366] Score 1 3
T2 Student Journals
Rubric Category Score Evidence – from T14098BrL42D1T2 Score Evidence – Students 80964, 80958, 80956
Failure 1 Stability – car test – works first, second, 1 It failed after about 5 weights were added third, fourth times [line 63] [pg. 33] Strength – Ss add weight and count. Ms Clay stops them at 7. “No, it’s already failed” [line 65] Attribution 1 RG announces her plan for improvement. It 0 No evidence found will cost them a point on cost, but will be better. The plan is based on the fact the weights fell off the side, [line 71]
186
Aligned Goals 1 RG announces her plan for improvement. It 1 [We will improve] strength [pg. 34] will cost them a point on cost, but will be better. The plan is based on the fact the weights fell off the side, and they can buy two pieces of paper to add piers, but still allowing space for the barge that will be stronger [line 71]
Aligned Strategy 1 ..they can buy two pieces of paper to add 1 We will improve our bridge by adding more piers, but still allowing space for the barge columns [pg. 34] that will be stronger [line 71]
Improvement 0 Weight test – fails at 14 [line 136] 0 Strength was the lowest because we got a one on it. Score 4 4
Improvement Cycle #2
Failure 1 Weight test – fails at 14 [line 136] 1 Strength [was our lowest score] because we got a 1 on it [pg 38] Attribution 0 No evidence found 0 No evidence
Aligned Goals 1 They start to talk through the questions. 1 [We will improve] strength [pg. 39] Highest score was stability – because the car went across 4 times. Aligned Strategy 1 LG wants index cards instead of paper and 1 We will change the bridge by adding things proposes to cut them for 9 total columns under the bridge [pg. 39]
Improvement 0 The weights are falling off the side. Fails at 0 Strength was our lowest because we only 12. got a 1 on it [pg. 43]
Score 3 3
187
Appendix C – Interaction of Failure Cause, Type, and Teacher Reaction
Knowledge of Knowledge of Poor Craftsmanship Limitation of Science/Technology Materials Materials
Manager Manager Manager Manager 1
Cheerleader Cheerleader Cheerleader Cheerleader Low stakes, intended Strategic Partner Strategic Strategic Strategic 3 Partner Partner Partner
Manager Manager Manager Manager Cheerleader Cheerleader Cheerleader Cheerleader Low stakes, unintended Strategic Partner 1 Strategic 2 Strategic 1 Strategic Partner Partner Partner
Manager 3 Manager 5 Manager Manager
Cheerleader Cheerleader Cheerleader Cheerleader High 1 1 1 Stakes, intended Strategic Partner 1 Strategic 1 Strategic Strategic Partner Partner Partner
Manager Manager Manager Manager
Cheerleader Cheerleader Cheerleader Cheerleader High stakes, unintended Strategic Partner Strategic Strategic Strategic Partner Partner Partner
Manger
Cheerleader 1 Pre- emptive Strategic Partner 8
188
Appendix D - Scoring Engineering Journals for Improvement
Page number Question Example of a “Yes” Example of a “No” 32/33 & 37/38 Did the design fail? “It collapsed when we “We got the highest score you added weight” or can get.” Yes-+1 No-DO NOT Scores of <4 for stability or SCORE <5 for strength 32/33 & 37/38 Was there evidence of an “….because we didn’t have “[it was our lowest score] attribution of failure? enough supports” or because it only got a 2.” “because the deck was too Yes-+1 No-+0 flimsy” “We got a 2 because it collapsed when we added weight”
34 & 39 Was the criterion for The design only earned a The design earned a 1 for improvement (cost, strength, “1” for strength, so they strength but they aimed to or stability) aligned with a aimed to improve strength. improve cost. failure they identified?
Yes-+1 No-+0
34 & 39 Was there evidence of a “We will add more “We will make it stronger” strategy for improvement supports” (if improved aligned with their stated strength is the goal) goals? “We will add another layer to the deck to make it more Yes-+1 No-+0 rigid” (if improved stability is the goal)
37, 42, & 46 Was there evidence that the “We went from holding 2 “It held the same” design improved due to that weights to 24.” “It got worse” strategy? Documented scores from the chart on 46 (as long as improved score was aligned Yes-+1 No-+0 with their goal (tried to improve strength and strength score increased – not necessarily total design score))
VITA Matthew Michael Johnson The Pennsylvania State University 182 Chambers Building University Park, PA 16802 [email protected] EDUCATION The Pennsylvania State University, University Park, PA Ph.D. Curriculum and Instruction, Science Education; January 2012-May 2016 Dissertation: Failure is an option: Responses to failure in elementary engineering design projects The Pennsylvania State University, University Park, PA M.Ed. Curriculum and Instruction, Science Education; January 2010-December 2010 Clarion University of Pennsylvania, Clarion, PA B.S. Education; June 2001-May 2003 Mercyhurst University, Erie, PA B.A Biology; August 1995-May 1999
PUBLICATIONS Johnson, M.M., Croom-Perez, T., Perez, A.A., Tekeley, C., Edelman, R. (in review). From Fish Tank to Fuel Tank: Engineering Photobioreactors in the Classroom. Engstrom, T., Russin, T., Johnson, MM., Ecklund, P. (2015). A computer-controlled classroom model of an atomic force microscope. The Physics Teacher. 56 (9), 536-538. Zhang D, Johnson MM, Miller CP, Pircher T, Geiger JN, Wojchowski DM. (2001). An optimized system for studies of EPO-dependent murine pro-erythroblast development. Experimental Hematology. 29(11), 1278-88. McDonald, C.M., Johnson M, Schunk, D., Kreuter, R., Wigton, B., Chohan, B. and Sykes, D. (2011) A Portable, Low-Cost, LED Fluorimeter for Middle School, High School, and Undergraduate Chemistry Labs. Journal of Chemical Education 88 (8), 1182-1187. Dominguez V, McDonald C, Johnson M, Schunk D, Kreuter R, Wigton, B, Chohan B and Sykes D. (2010). The characterization of a custom-built coulometric Karl-Fischer titrator. Journal of Chemical Education. 87(9), 987-991.
TEACHING EXPERIENCE
Center for Science and the Schools, University Park, PA: May 2011-Present The Pennsylvania State University, University Park, PA: Spring 2012; 1999-2001 West Branch Area Junior/Senior High School, Allport, PA: August 2003-May 2011 Upward Bound Math and Science, University Park, PA: June 2008-August 2010