10

Research Design in Qualitative/Quantitative/ Mixed Methods

Michael R. Harwell University of Minnesota

he editors of this handbook assert that cases, these designs are currently available, and educational inquiry is weakly connected the challenge is for more researchers to employ T to the study of promising ideas. The them (or continue to employ them) in a rigorous implication is that much educational research has manner. In other instances, new designs will need a “nuts and bolts” character that limits creative to be developed. The net effect is that the catalog problem generation or problem solving and of research designs available to educational discourages risk taking. The consequences of this researchers is expanding dramatically. This practice are missed opportunities to advance our change does not threaten or undermine the value understanding of important educational problems of designs that are currently widely used, but and solutions. This is a bold assertion, but it is one rather it increases the pool of research designs that is supported by an examination of published available to support rigorous inquiry. and unpublished educational studies. This chapter explores the role of research A key player in this process is a study’s research design in the study of promising ideas in educa- design. In many educational studies, research tion, assuming two conditions are present. First is design reflects a “cookie cutter” approach in which that the idea being pursued is worth pursuing, the same designs are repetitively and narrowly that is, an idea that could reasonably lead to, and applied in ways that may limit what is studied and support, the formation of important educational how it is studied. Research-based ideas that cannot questions and identification of solutions to be easily mapped onto a small number of designs important educational problems. Second is that acceptable to funders and professional journals every phase of the research process linked to are likely to be abandoned or modified to fit studying a promising idea is rigorous. These con- widely used research designs. ditions simplify an examination of the role of The assertion of this chapter is that the study research design in the study of promising ideas. of promising ideas in education would be well Characteristics of three research methodolo- served by the increased use of research designs gies (qualitative methods, quantitative methods, that draw on multiple traditions of inquiry. In some mixed methods) and their role in studying ideas

147 148 Section III. opportunities and Challenges in Designing and Conducting Inquiry believed to be worth studying are described. I use analyze qualitative data; quantitative studies are, two studies to illustrate strengths in the research among other things, studies that collect and design, as well as opportunities to enhance the analyze quantitative data; and so on. results, and give greater attention to mixed meth- Crotty (1998) described four key features to ods because of their relative newness and poten- consider in research design: the epistemology tial. Three ways that researchers can enhance the that informs the research, the philosophical ability of research designs to better support the stance underlying the methodology in question study of promising ideas in educational studies (e.g., post-positivism, constructivism, pragma- are described. I conclude by arguing that mixed tism, advocacy/participatory; see Morgan, 2007), methods offer an especially promising path toward the methodology itself, and the techniques and using research design in ways that support rigor- procedures used in the research design to collect ous inquiry. data. These features inform the descriptions of research designs below.

Research Design Qualitative Research Methods In educational research, it is usually possible Qualitative research methods focus on discov- (and certainly popular) to characterize a research ering and understanding the experiences, per- study’s methodology as qualitative; as quantita- spectives, and thoughts of participants—that is, tive; or as involving both qualitative and quanti- qualitative research explores meaning, purpose, tative methods, in which case it is typically or reality (Hiatt, 1986). In other words, referred to as mixed methods. The term research design is widely used in education, yet it takes on qualitative research is a situated activity that different meanings in different studies. (The locates the observer in the world. It consists terms research method and research design will be of a set of interpretive, material practices that used inter­changeably in this chapter.) For exam- make the world visible. These practices ple, in one study, research design may reflect the transform the world. They turn the world into a entire research process, from conceptualizing a series of representations, including field notes, problem to the literature review, research ques- interviews, conversations, photographs, tions, methods, and conclusions, whereas in recordings, and memos to the self. At this level, another study, research design refers only to the qualitative research involves an interpretive, method­ology of a study (e.g., data collection and naturalistic approach to the world. This analysis). Perhaps not surprisingly, there is varia- means that qualitative researchers study things tion within and between methodologies in how in their natural settings, attempting to make research design is defined. However, this varia- sense of, or interpret, phenomena in terms of tion does not affect an examination of the role the meanings people bring to them. (Denzin & of research design in promoting rigorous study Lincoln, 2005, p. 3) of promising ideas and, thus, a single definition of research design is not adopted in this chapter. Central to this inquiry is the presence of mul- I assume that research questions are the driving tiple “truths” that are socially constructed (Lincoln force behind the choice of a research design and & Guba, 1985). Qualitative research is usually any changes made to elements of a design as a described as allowing a detailed exploration of a study unfolds. topic of interest in which information is collected Identifying a study’s research design is impor­ by a researcher through case studies, ethnographic tant because it communicates information about work, interviews, and so on. Inherent in this key features of the study, which can differ for approach is the description of the interactions qualitative, quantitative, and mixed methods. among participants and researchers in naturalistic However, one common feature across research settings with few boundaries, resulting in a flexible designs is that at one or more points in the and open research process. These unique interac- research process, data are collected (numbers, tions imply that different results could be obtained words, gestures, etc.), albeit in different ways and from the same participant depending on who for different purposes. Thus, qualitative studies the researcher is, because results are created by a are, among other things, studies that collect and participant and researcher in a given situation Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 149

(pp. 39–40). Thus, replicability and generalizabil- interviews and media collected during the author’s ity are not generally goals of qualitative research. visits to India in ways that took into account his Qualitative research methods are also described identity; Harry, Sturges, and Klingner (2005), as inductive, in the sense that a researcher may who used the methods of grounded theory to construct theories or hypotheses, explanations, develop a theory providing a new perspective on and conceptualizations from details provided by a ethnic representation in special education; Brown participant. Embedded in this approach is the (2009), who studied the perspectives of university perspective that researchers cannot set aside their students identified as learning disabled; and experiences, perceptions, and biases, and thus Chubbuck and Zembylas (2008), who examined cannot pretend to be objective bystanders to the the emotional perspective and teaching practices research. Another important characteristic is that of a White novice teacher at an urban school. the widespread use of qualitative methods in edu- These studies reflect several important fea- cation is relatively new, dating mostly to the 1980s, tures of qualitative research, including a focus with ongoing developments in methodology and on discovering and understanding the experi- reporting guidelines (Denzin, 2006). The relative ences, perspectives, and thoughts of participants newness of this methodology also means that through various strategies of inquiry. The stud- professional norms impacting research, includ- ies were also conducted in naturalistic settings in ing evidence standards, funding issues, and edi- which inquiry was flexible and guided by par- torial practices, are evolving (see, e.g., Cheek, ticipants’ comments, which in some instances 2005; Freeman, deMarrais, Preissle, Roulston, & were used to construct explanations of their St.Pierre, 2007). Good descriptions of qualitative views and perspectives. An important feature of methods appear in Bogdan and Biklen (2003), several of these studies is their use of elements of Creswell (1998), Denzin and Lincoln (2005), different strategies of inquiry. Miles and Huberman (1994), and Patton (2002). There are several categorizations of research Quantitative Research Methods designs in qualitative research, and none is uni- versally agreed upon (see, e.g., Denzin & Lincoln, Quantitative research methods attempt to 2005). Creswell (2003) listed five strategies of maximize objectivity, replicability, and general- inquiry in qualitative research that I treat as syn- izibility of findings, and are typically interested in onymous with research design: narratives, phe- prediction. Integral to this approach is the expec- nomenological studies, grounded theory studies, tation that a researcher will set aside his or her ethnographies, and case studies. Creswell also experiences, perceptions, and biases to ensure described six phases embedded in each research objectivity in the conduct of the study and the design that are more specific than those suggested conclusions that are drawn. Key features of many by Crotty (1998), but still encompass virtually all quantitative studies are the use of instruments aspects of a study: (1) philosophical or theoretical such as tests or surveys to collect data, and reli- perspectives; (2) introduction to a study, which ance on probability theory to test statistical includes the purpose and research questions; hypotheses that correspond to research questions (3) data collection; (4) data analysis; (5) report writ- of interest. Quantitative methods are frequently ing; and (6) standards of quality and verification. described as deductive in nature, in the sense that Journals that publish qualitative methodology inferences from tests of statistical hypotheses lead papers and qualitative research studies in educa- to general inferences about characteristics of a tion include Qualitative Research, Qualitative population. Quantitative methods are also fre- Inquiry, Field Methods, American Educational quently characterized as assuming that there is a Research Journal, Educational Researcher, and the single “truth” that exists, independent of human International Journal of Qualitative Studies in perception (Lincoln & Guba, 1985). Education. Examples of the use of qualitative Trochim and Land (1982) defined quantitative research designs are provided by Stage and Maple research design as the (1996), who used a narrative design to describe the experiences of women who earned a bache- glue that holds the research project together. A lor’s or master’s degree in mathematics and opted design is used to structure the research, to show to earn a doctorate in education; Gaines how all of the major parts of the research (2005), who explored the process of interpreting project—the samples or groups, measures, 150 Section III. opportunities and Challenges in Designing and Conducting Inquiry

treatments or programs, and methods of Quantitative methods have a long history, assignment—work together to try to address the dating to at least the 1930s, that has produced central research questions. (p. 1) strong professional norms that impact research activities, such as the criteria used to make fund- Definitions of quantitative research design are ing decisions (What Works Clearinghouse, complicated by the fact that this term is often 2008) and decisions about the kinds of studies used to identify the experimental design reflect- and results likely to be published (Bozarth & ing the arrangement of independent and depen- Roberts, 1972; Leech, Morgan, Wang, & Gliner, dent variables associated with data collection. 2010; Rosenthal, 1979). Older categorizations of experimental designs Good descriptions of quantitative methods tend to use the language of analysis of variance in appear in Bryman (2004); Kerlinger (1964); describing these layouts—for example, a single- Balnaves and Caputi (2001); Gay, Mills, and factor, completely between-subjects, fixed-effects Airasian (2005); and the Research Methods design or a factorial split-plot design with Knowledge Base (Trochim, 2006). There are also between- and within-subject effects (Kempthorne, several journals that publish quantitative method- 1952; Winer, 1962). These descriptions were typi- ology papers and quantitative research studies in cally linked to an experimental design in which education, including Psychological Methods, subjects were randomly assigned to treatment Journal of Educational and Behavioral Statistics, conditions, although quasi-experimental designs British Journal of Mathematical and Statistical in which intact groups (e.g., African American Psychology, American Educational Research Journal, and White students) are compared, and correla- American Journal of Evaluation, and Educational tional designs in which no definable groups are Evaluation and Policy Analysis. present, also received attention (e.g., Campbell & Examples of quantitative research designs in Stanley, 1963). the educational literature are provided by More recent categorizations of research Howell, Wolf, Campbell, and Peterson (2002), designs largely abandon the analysis-of-variance who used a randomized design to study the frame­work and instead rely on a trichotomy: effectiveness of school vouchers in improving randomized­ designs or, equivalently, randomized achievement; Jacob and Lefgren (2004), who controlled trials in which participants are used a quasi-experimental (regression disconti- assigned at random to treatment conditions, nuity) design to study the relationship between quasi-experimental designs, and correlational receiving a remedial education and achieve- designs (Pedhazur & Schmelkin, 1991; What Works ment; and Garner and Raudenbush (1991), who Clearinghouse, 2008). These newer categorizations used a correlational design to examine the rela- also tend to focus on conditions needed to justify tionship between neighborhood characteristics strong causal inferences (Schneider, Carnoy, and achievement. These studies reflect several Kilpatrick, Schmidt, & Shavelson, 2007; Shadish, important features of quantitative research, Cook, & Campbell, 2002). including an assumption that researchers set A quantitative research design also involves aside their experiences, perceptions, and biases phases that are superficially similar to those in conducting the study and drawing conclu- offered by Crotty (1998) and Creswell (2003) for sions, data collection that did not treat the qualitative research, but that are quite different in researcher as the data collection instrument, and purpose and execution: (1) introduction to a the use of hypothesis testing to make deductive study that includes the purpose and research inferences about characteristics of a population. questions; (2) theoretical perspectives or models; In sum, research design in education has (3) methodology that encompasses sampling and moved from an almost exclusive reliance on quan- an evaluation of external validity, instrumentation titative methods to a more varied approach that that may include an evaluation of construct valid- includes qualitative research methods. As Lincoln ity, experimental design that includes an evalua- and Guba (1985) pointed out, both qualitative tion of internal validity and data collection, and and quantitative research methods emphasize data analysis that includes an evaluation of statis- truth, consistency, applicability, and neutrality tical conclusion validity; (4) reporting the results; while taking different procedural approaches to and (5) conclusions and implications (Pedhazur assure quality. Thus, it might be imagined that & Schmelkin, 1991; Shadish et al., 2002). qualitative and quantitative methods have been Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 151 promoted to educational researchers in the those linked to quantitative methods. According spirit of “Use the methodology or methodolo- to Johnson and Onwuegbuzie (2004), gies that is (are) most appropriate and helpful for your purposes,” but that is not quite the Mixed methods research is formally defined case. Rather, educational research in the last here as the class of research where the researcher two decades has been home to considerable mixes or combines quantitative and qualitative conflict fueled by advocates of each methodol- research techniques, methods, approaches, ogy who have emphasized weaknesses of the concepts or language into a single study. Mixed other in terms of epistemology, importance for methods research also is an attempt to the field, permissible inferences, and so forth legitimate the use of multiple approaches in (see, e.g., Berkenkotter, 1991; Denzin & Lincoln, answering research questions, rather than 2005; Gage, 1989; Howe, 2009; Maxwell, 2004; restricting or constraining researchers’ choices Morse, 2006; Pegues, 2007). Much of the debate (i.e., it rejects dogmatism). It is an expansive has been useful in encouraging researchers to and creative form of research, not a limiting think about features of these methodologies form of research. It is inclusive, pluralistic, and important for their work and for the field (e.g., complementary, and it suggests that researchers Moss et al., 2009), but its prolonged and some- take an eclectic approach to method selection times acrimonious nature has likely prompted and the thinking about and conduct of research. many researchers to simply sit it out. (pp. 17–18)

This definition highlights the potential value of Mixed Methods mixing multiple elements of qualitative and The qualitative versus quantitative debate has quantitative methods, as well as the potential coincided with the rapid development of mixed complexity of doing so. methods, which combine qualitative and quanti- Caracelli and Greene (1997) identified three tative methods in ways that ostensibly bridge their typical uses of a mixed methods study: (1) test- differences in the service of addressing a research ing the agreement of findings obtained from question. The roots of mixed methods are typi- different measuring instruments, (2) clarifying cally traced to the multi-trait, multi-method and building on the results of one method with approach of Campbell and Fiske (1959, cited in another method, and (3) demonstrating how Teddlie & Tashakkori, 2009, p. 31), although it is the results from one method can impact subse- considered a relatively new methodology whose quent methods or inferences drawn from the key philosophical and methodological founda- results. These purposes appear in some form in tions and practice standards have evolved since many mixed methods studies in diverse fields the early 1990s (Tashakkori, 2009). including education (Taylor & Tashakkori, Johnson and Turner (2003) have argued that 1997), psychology (Todd, Nerlich, McKeown, & the fundamental principle of mixed methods Clarke, 2004), criminology (Maruna, 2010), research is that multiple kinds of data should be nursing and health sciences (O’Cathain, 2009), collected with different strategies and methods in family research (Greenstein, 2006), and business ways that reflect complementary strengths and (Bryman & Bell, 2007). In part, recent increases non-overlapping weaknesses, allowing a mixed in the number of mixed methods studies can be methods study to provide insights not possible attributed to increases in funding (Creswell & when only qualitative or quantitative data are col- Plano Clark, 2007; Plano Clark, 2010). Still, lected. Put another way, mixed methods research journal articles and funding programs offer allows for the “opportunity to compensate for abundant evidence that widespread acceptance inherent method weaknesses, capitalize on inher- and funding of mixed methods in educational ent method strengths, and offset inevitable research is a work in progress (e.g., Alise & method biases” (Greene, 2007, p. xiii). Teddlie, 2010; National Research Council, 2002, While mixed methods research combines 2005; O’Cathain, Murphy, & Nicoll, 2007; What qualitative and quantitative methods in ways Works Clearinghouse, 2008). that draw on the strengths of both traditions of Despite their growing popularity, there is not inquiry, it is a clear step away from the boundar- widespread agreement on exactly what consti- ies and practices of those traditions, especially tutes a mixed methods study (Morse, 2010). For 152 Section III. opportunities and Challenges in Designing and Conducting Inquiry example, some authors insist that a mixed meth- However, the pragmatic stance also has its crit- ods study is any study with both qualitative and ics (Mertens, 2003; Sale et al., 2002). Dialectical quantitative data, whereas other authors say a researchers argue that multiple paradigms are mixed methods study must have a mixed meth- compatible and should be used, but their differ- ods question, both qualitative and quantitative ences and the implications for research must be analyses, and integrated inferences (Tashakkori, made clear (Greene & Caracelli, 1997). It is impor- 2009). There is also disagreement regarding tant to emphasize that a researcher who adopts a various aspects of mixed methods, such as when dialectic stance would, other things being equal, mixing should occur (e.g., at the point of design- draw on the same procedures for mixing as one ing a study, during data collection, during data adopting a pragmatic stance. The issue of stances analyses, and/or at the point of interpretation). is certainly not settled, and additional develop- Still other authors have criticized the whole ments on this topic continue to appear in the idea of mixed methods (Denzin, 2006; Sale, mixed methods literature (e.g., design stance of Lohfeld, & Brazil, 2002; Smith & Hodkinson, Creswell & Plano Clark, 2007). 2005), criticism which is sometimes framed in Mixed methods designs seem especially firmly terms of the response of advocates of a particu- rooted in the evaluation literature. An early lar “stance” to arguments for mixing methods: important paper in this area was Greene, Caracelli, the purist stance, the pragmatic stance, and the and Graham (1989), which highlighted five major dialectical stance (Greene & Caracelli, 1997; purposes of (or justifications for) a mixed meth- Johnson & Onwuegbuzie, 2004; Lawrenz & ods evaluation. One is triangulation, which exam- Huffman, 2002). Those adopting a purist stance ines the consistency of findings, such as those argue that mixed methods are inappropriate obtained through different instruments, and because of the incompatibility of the worldview which might include interviews and surveys. or belief system (paradigms) (Tashakkori & According to Green et al., triangulation improves Teddlie, 2003) underlying qualitative and quan- the chances that threats to inferences will be con- titative methods, i.e., qualitative and quantita- trolled. A second purpose is complementarity, tive methods are studying different phenomena which uses qualitative and quantitative data with different methods (Smith & Hodkinson, results to assess overlapping but distinct facets of 2005). Some purists have also raised concerns the phenomenon under study (e.g., in-class that mixed methods designs leave qualitative observations, surveys), and a third is develop- methods in the position of being secondary to ment, in which results from one method influ- quantitative methods (Denzin, 2006; Giddings, ence subsequent methods or steps in the research; 2006; Yin, 2006). for example, interviews with teachers might sug- Researchers who adopt a pragmatic stance gest that an additional end-of-year assessment be argue that paradigm differences are independent added. A fourth purpose of a mixed methods of, and hence can be used in conjunction with, evaluation is initiation, in which results from one one another in the service of addressing a ques- method challenge other results or stimulate new tion (Johnson & Onwuegbuzie, 2004; Morgan, directions for the research; for example, teacher 2007). Wheeldon (2010) summarizes this view: interviews might challenge results provided by administrators in a school district. The fifth and Instead of relying on deductive reasoning and last purpose is expansion, which may clarify general premises to reach specific conclusions, or results or add richness to the findings. inductive approaches that seek general A number of frameworks for mixed meth- conclusions based on specific premises, ods have appeared in this literature, many of pragmatism allows for a more flexible abductive which have built on the work of Greene et al. approach. By focusing on solving practical (1989) (Caracelli & Greene, 1997; Creswell, problems, the debate about the existence of 2003; Johnson & Onwuegbuzie, 2004; Lawrenz objective “truth,” or the value of subjective & Huffman, 2002; Leech & Onwuegbuzie, 2007; perceptions, can be usefully sidestepped. As such, Morse, 2010; Newman, Ridenour, Newman, & pragmatists have no problem with asserting both DeMarco, 2003; Tashakkori & Teddlie, 2003). that there is a single “real world” and that all These frameworks differ in many ways, but individuals have their own unique interpretations they all successfully convey a sense of the large of that world. (p. 88) number of methodological tools available to Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 153 researchers. However, none of these frameworks collection phases (as opposed to only collecting has been widely adopted. qualitative or quantitative data) and the expertise The framework of Creswell (2003) is used here needed to integrate the qualitative and quantita- to categorize research designs in mixed methods, tive findings. Morgan (1998) suggested that the but this choice should not be interpreted to mean sequential explanatory design is the most fre- that it has been widely endorsed by mixed meth- quently used mixed methods approach. ods theoreticians and practitioners. This frame- As an example, consider the quantitative study work is described next, and studies by Howell of Howell et al. (2002). These authors were inter- et al. (2002) and Norman (2008) are used to ested in learning if students who received vouchers illustrate strengths in their research design, as allowing them to enroll in a private school (and well as opportunities to enhance their results by who subsequently enrolled) showed improved employing a mixed methods approach. In doing achievement compared to students who did not so, I assume that different paradigms can be receive a voucher and attended a public school. matched to answer research questions of interest, To explore this question, these authors examined and thus I adopt a pragmatic stance. It is impor- achievement data for more than 4,000 students in tant to emphasize that there are many complexi- three U.S. cities in which vouchers were randomly ties not captured by these examples, including awarded via a lottery. This arrangement produced the nature and timing of the mixing and the use a randomized design in which receiving a voucher of elements from multiple mixed methods (yes, no) was the key independent variable, and approaches in a given research design. student achievement was the key dependent vari- Creswell (2003) described six somewhat over- able. No theoretical model was offered. The lapping mixed methods research designs, referred results showed an average increase in the achieve- to as strategies of inquiry, that guide the con- ment of African American students who received struction of specific features of a mixed methods a voucher and subsequently enrolled in a private study. The designs vary in whether qualitative school compared to African American students and quantitative data are collected sequentially who did not receive a voucher, but not for any or concurrently, the weight given one kind of other groups in the study. From a quantitative data or another, when the mixing is done, and perspective, this study has many strengths, par- the extent to which a theoretical perspective (e.g., ticularly random assignment, that enhance causal post-positivism, constructivism) is present and arguments. guides the research design. Howell et al. (2002) also commented that the The first mixed methods approach described reasons vouchers improved the academic perfor- in Creswell (2003) is the sequential explanatory mance of African American students and not design, in which qualitative data are used to other students were unclear. Is this difference enhance, complement, and in some cases follow related to a student’s view of his or her academic up on unexpected quantitative findings. In this skills, peer related, due to motivation differences approach, the focus is on interpreting and or strong parental support for education, the explaining relationships among variables and student’s interactions with school staff, or other may or may not be guided by a particular theo- factors? The Howell et al. comment suggests that retical perspective. Quantitative data are collected a sequential explanatory research design would and analyzed first, followed by the collection and allow these authors to search for explanations of analysis of qualitative data, meaning that qualita- the finding that vouchers improved the academic tive and quantitative data are not combined performance of African American students but (mixed) in the data analysis; rather, integration not other students. For example, collecting qual- takes place when the findings are interpreted. In itative data at the end of the study through inter- general, results are interpreted in ways that views of a purposively sampled group of parents usually give more weight to the quantitative of students who had and had not received a component. The separate phases of design, data voucher could be helpful in enhancing and collection, and reporting for qualitative and complementing the quantitative findings. quantitative data are considered strengths, A second mixed methods approach, the because this arrangement is relatively easy to sequential exploratory design, is essentially the implement. The weaknesses of this approach are reverse of the sequential explanatory design, the time and resources needed for separate data with quantitative data used to enhance and 154 Section III. opportunities and Challenges in Designing and Conducting Inquiry complement qualitative results. This approach is process of how academic counselors make deci- especially useful when the researcher’s interest is sions, and used this model and a case study in enhancing generalizability, and it may or may approach (Creswell, 1998). Norman purposively not be guided by a theoretical perspective. sampled 6 counselors and 24 students and col- Creswell (2003) pointed out that instrument lected data via in-depth interviews with the construction is an example of this approach, in counselors and students that included their that a draft of an instrument (survey, test) is observations of the advising process, and high piloted with a small number of cases who often school and college records of students’ mathe- provide important qualitative feedback about matics course taking and grades. Norman their experience with the instrument, followed, reported that the counselors generally misinter- after appropriate modifications of the instru- preted the mathematics portions of high school ment, by using the instrument to collect quanti- transcripts, which had important implications tative data. The quantitative results are then for advising a student on which college mathe- used to enhance, complement, and possibly matics course to begin with. For example, a student extend the earlier pilot results. The strengths whose transcript indicated that his or her highest and weaknesses of this approach are similar to completed high school mathematics course was those of the sequential explanatory design. “Integrated Mathematics IV,” a standards-based As an example, consider the mixed methods course, was generally advised to start with a study of Norman (2008). Norman was interested precalculus mathematics course, even though in uncovering the perspectives of college aca- Integrated Mathematics IV is a precalculus demic counselors when advising newly admitted course and the student should have been advised students on the appropriate mathematics course to enroll in Calculus I. Norman also reported with which to begin their college mathematics that counselors who looked at transcripts of career, and integrating this information with students who completed a traditional high quantitative information reflecting the high school mathematics curriculum, in which the school mathematics curriculum students com- highest mathematics course completed was pleted and their scores on a college mathematics listed as precalculus, were more likely to recom- placement test. Norman’s work was motivated by mend that a student enroll in Calculus I. Norman anecdotal evidence that students who completed suggested that counselor views toward standards- a standards-based mathematics curriculum in based curricula may be related to working in a high school were more likely to be advised to mathematics department, because mathematics begin college with a less difficult mathematics departments have generally been quite critical of course than comparable students who completed standards-based curricula (Roitman, 1999; a traditional high school mathematics curricu- Schoenfeld, 2004). lum. Standards-based curricula in high school Norman (2008) also collected and analyzed focus on a variety of mathematics topics every quantitative data for a sample of more than school year in a way that emphasizes problem 1,000 college freshmen that included informa- solving and small-group work, and de-emphasizes tion on the high school mathematics curricu- algorithmic manipulation. Traditional curricula lum they completed; their score on a college typically focus on algebra, algorithms, and repe- mathematics placement exam; and the difficulty tition and depend heavily on the teacher for of their first college mathematics course, which student learning (Schoenfeld, 2004). was captured using a 4-point Likert variable Differential advising of students as a func- (1 = a course that should have been completed tion of the high school mathematics curricu- in high school, which is sometimes referred to as lum they completed implies that some students a developmental course; 4 = a course whose dif- are advised to begin their college mathematics ficulty exceeded that of Calculus I). The results with a course they should not take (e.g., precal- of these analyses suggested that the curriculum culus rather than Calculus I). The implication a student completed was related to his or her of this practice is that students and colleges mathematics placement score and the difficulty may be spending time, money, and effort on level of the student’s first college mathematics unneeded courses. course. In particular, students who completed a Norman (2008) adapted a theoretical decision- standards-based high school mathematics curric- making model (Miller & Miller, 2005) to the ulum were more likely to enroll in a less difficult Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 155 college mathematics course compared to students with it. Here, the study could focus on discover- who completed a traditional curriculum. ing and understanding students’ experiences with Norman’s (2008) study reflects several exem- the advising process and its implications for their plary features of mixed methods research, includ- college experience. A case study approach, using a ing using a case study approach that focused on purposively chosen sample of students who discovering and understanding the experiences, began their college mathematics course taking at perspectives, and thoughts of counselors and stu- different difficulty levels, would be appropriate. dents, and the presence of quantitative data high- Information obtained from interviews and stu- lighting the possible consequences of counselors’ dent academic records could be used to inform misunderstanding the information on a student’s the construction of a survey to be sent to a repre- high school transcript. While the exact timing of sentative sample of students. The survey results the quantitative and qualitative data collection in could be used to improve decision making by Norman’s study was unclear, it appears the data counselors and to enhance generalizability. were collected relatively close in time, suggesting A fourth mixed methods approach is the con- that the results of one data collection did not current triangulation design, which is used when influence the other data collection. However, the the focus is on confirming, cross-validating, or main question focused on counselors, and col- corroborating findings from a single study. lecting and analyzing data for counselors first, Qualitative and quantitative data are collected intentionally using this information to guide the concurrently, such that weaknesses of one kind of collection and analysis of quantitative data for data are ideally offset by strengths of the other students, and integrating the two sets of findings kind. Typically, equal weight is given to the two suggest that a sequential exploratory design kinds of data in mixing the findings, although would have been of value. For example, informa- one kind of data can be weighted more heavily. tion obtained from the qualitative data about The qualitative and quantitative data are analyzed counselors’ misunderstanding of high school separately, and mixing takes place when the find- mathematics course taking could have been used ings are interpreted. Important strengths of this in developing questions for a student survey, such approach are the ability to maximize the infor- as on whether students thought their counselor mation provided by a single study, for example, misunderstood their high school course taking. when interest is in cross-validation, and a shorter A third mixed methods approach is the data collection period compared to the sequential sequential transformative design, in which either data collection approaches. Important weak- qualitative or quantitative data may be collected nesses include the additional complexity associ- first. Here, the theoretical perspective underly- ated with collecting qualitative and quantitative ing the methodology is critical to the conduct of data at the same time and the expertise needed to the study, and the chosen methods should serve usefully apply both methods. Discrepancies the theoretical perspective. Once again, qualita- between the qualitative and quantitative findings tive and quantitative data are analyzed sepa- may also be difficult to reconcile. rately, and the findings are integrated during the In the Howell et al. (2002) study, the primary interpretation phase. This approach is often finding was that the achievement of African used to ensure that the views and perspectives of American students who received a voucher to a diverse range of participants are represented or attend private school was on average higher than when a deeper understanding of a process that is that of African American students who did not changing as a result of being studied is sought. receive a voucher, and that this difference did not Its strengths and weaknesses are similar to those emerge for other student groups. Adopting a con- of the sequential explanatory design. current triangulation design could provide an For example, the decision-making model used explanation for these findings by collecting quali- by Norman (2008) suggested that collecting data tative data in the form of interviews with parents for students who are affected (sometimes adversely) of students who did and did not receive a voucher, by the recommendations provided by counselors and quantitative data in the form of student test is important in evaluating and improving the scores and background information. This would advising process. Norman commented that most offer an opportunity to corroborate findings students follow the recommendation of which from this study with respect to the improved mathematics course to take, even if they disagree achievement of African American students but 156 Section III. opportunities and Challenges in Designing and Conducting Inquiry not other students. For example, a plausible out- A second data-mixing strategy described by come of concurrent data collection is that the Caracelli and Greene (1993) is typology develop- qualitative data suggest that parents of African ment, in which the analysis of one kind of data American students appeared to be more commit- produces a typology or set of categories that is ted to, and enthusiastic about, their students’ used as a framework in analyzing the other kind education in general and the voucher program in of data. In Norman’s (2008) study, analyses of the particular, than parents of other students, and qualitative data could produce themes that allow that this enthusiasm persisted throughout the a variable with nominally scaled categories to be school year (Achievement tests used in this study developed, in which the categories provide an were given at the end of the school year.). In this explanation of why participants became counsel- instance, the qualitative and quantitative infor- ors. This variable could then be used in the quan- mation would provide corroborating evidence titative analysis. that the improved achievement of African A third data-mixing strategy is extreme case American students could be attributed in part to analysis, in which extreme cases identified with receiving a voucher and enrolling in a private one kind of data are examined with the other school, and in part to the support, encourage- kind, with the goal of explaining why these cases ment, and motivation of students’ parents. are extreme. For example, multilevel analyses of The fifth approach is the concurrent nested quantitative data in Norman’s (2008) study may design, in which qualitative and quantitative suggest that some counselors are, with respect to data are collected concurrently and analyzed the sample of counselors, statistical outliers (e.g., together during the analysis phase. Greater students linked to these counselors dispropor- weight is given to one kind of data, in the sense tionately began their college mathematics study that one kind of data is typically embedded in with courses that should have been completed in the other. However, there may or may not be a high school). Qualitative data could be used to try guiding theoretical perspective. A popular appli- to explain why these counselors appeared to dis- cation of this approach is with multilevel struc- proportionately steer students to developmental tures (Tashakkori & Teddlie, 2003), in which college mathematics courses. different levels or units of an organization are The fourth data-mixing strategy described by studied. Strengths of this approach include the Caracelli and Greene (1993) is data consolidation/ shorter data collection period and the multiple merging, in which a careful review of both perspectives embedded in the data, whereas kinds of data leads to the creation of new vari- weaknesses include the level of expertise needed ables or data sets expressed in a qualitative or to execute the study successfully, especially in quantitative metric. The merged data are then mixing the qualitative and quantitative data used in additional analyses. In Norman’s (2008) within the data analysis, and difficulties in rec- study, a review of the qualitative and quantita- onciling conflicting results from the qualitative tive data may suggest new variables. For exam- and quantitative analyses. ple, a review of the counselor and student data In this design, qualitative and quantitative data may suggest constructing a variable capturing are mixed in the analysis phase, a process that can the extent to which students assert themselves in take many different forms (see, e.g., Bazeley, 2009; the advising process. Examples of data mixing Tashakkori & Teddlie, 2003). Caracelli and Greene appear in Caracelli and Greene (1993); Sandelowski, (1993) described four strategies to mix qualitative Voils, and Knafl (2009); and Tashakkori and and quantitative data in the analysis. One is data Teddlie (2003). transformation, in which qualitative data are Norman’s (2008) study seems ready-made for transformed to quantitative data or qualitative a concurrent nested design, given the inclusion of data are transformed into narrative, and the both counselors and students. A key outcome in resulting data are analyzed. In Norman’s (2008) this study was the difficulty level of a student’s first study, this could involve transforming (i.e., rescal­ college mathematics course. Quantitative evi- ing) qualitative data in the form of interviews, dence of a relationship between a student’s high field notes, and so on to a quantitative form that school mathematics curriculum (standards-based, captures key themes in these data. Typically, the traditional) and the difficulty level of his or her transformed qualitative data exhibit a nominal or first college mathematics course could be mixed ordinal scale of measurement. with qualitative data obtained concurrently from Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 157 information provided by counselors. For example, This information could then be used to inform qualitative data could be obtained using a narra- the construction of a survey that would provide tive approach, in which counselors talked about quantitative information for a representative sam- the events in their lives that led them to become ple of newly admitted college students to enhance counselors and to continue to advise students. generalizability. The strengths and weaknesses of Transforming the qualitative counselor data such this approach are similar to those of the other that the transformed variables reflect important concurrent approaches. themes allows this information to be directly Other examples of mixed methods studies included (mixed) with student variables in a include Buck, Cook, Quigley, Eastwood, and quantitative multilevel data analysis, in which Lucas (2009); Day, Sammons, and Gu (2008); students are treated as nested within counselors Onwuegbuzie, Bustamante, and Nelson (2010); (Raudenbush & Bryk, 2002). These analyses could and Onwuegbuzie et al. (2007). Good descrip- explore the impact of nesting and the impact of tions of mixed methods can be found in Bazeley counselor variables on student data and provide a (2009), Creswell (2003), Creswell and Plano powerful explanation of how and potentially why Clark (2007), Greene (2007), Reichardt and students enrolled in a particular college mathe- Rallis (1994), and Tashakkori and Teddlie matics course. (2003). The relative newness of mixed methods The sixth mixed methods approach is the con- means that journals that concentrate on pub- current transformative design. As with the sequen- lishing mixed methods methodology papers tial transformative design, there is a clearly defined and mixed methods studies in education are theoretical perspective that guides the methodol- evolving. They currently include the Journal of ogy. In this approach, qualitative and quantitative Mixed Methods Research, International Journal data are collected concurrently and can be weighted of Multiple Research Approaches, Qualitative equally or unequally during the integration of Research Journal, American Educational Research findings. Qualitative and quantitative data are Journal, Educational Researcher, and Educational typically mixed during the analysis phase. Strengths Evaluation and Policy Analysis. include a shorter data collection period, whereas In sum, the mixed methods approach offers a weaknesses include the need to transform data so collection of flexible research designs that seem that it can be mixed in the analysis phase and dif- well suited to support rigorous examinations of ficulties in reconciling conflicting results using promising ideas. The six designs of Creswell qualitative and quantitative data. (2003) draw on the strengths of qualitative and As an example of applying a concurrent quantitative methods to enhance inquiry in ways transformative design, Norman (2008) reported unlikely to occur with singular applications of evidence that some students were unhappy with these methods. Still, it is important to emphasize the recommendations they received but felt pow- that mixed methods design continues to face a erless to change the process. Suppose that the number of significant challenges (Bryman, 2007; goal of the study was to develop a theory explain- Creswell, 2009; Lawrenz & Huffman, 2002; ing this sense of powerlessness in ways that could Tashakkori, 2009). improve the advising process. Norman’s adapta- One important challenge is resolving out- tion of the Miller and Miller (2005) decision- standing disagreements over appropriate para- making model includes a component that calls digms. The mixed methods literature contains for students who are concerned about their rec- a number of “mission accomplished” statements ommendation to be heard. This could take the implying that important philosophical differ- form of collecting qualitative information from ences have been resolved, primarily by adopting students reflecting their experiences with the a pragmatic stance (e.g., Carey, 1993; Creswell advising process and their view of its impact on & Plano Clark, 2007; Haase & Meyers, 1988; their mathematics course taking, and simultane- Tashakkori & Teddlie, 2003). Yet it is clear that ously collecting qualitative data for a group of important differences remain (Sandelowski, counselors. Developing a theory explaining the 2001; Yin, 2006). A related challenge is achiev- student’s feelings of powerlessness would require ing better agreement on what characterizes a that student and counselor responses be carefully mixed methods study and its components. For examined and categorized to suggest themes to example, what are the components of a mixed guide and inform the construction of the theory. methods study (research questions, design, 158 Section III. opportunities and Challenges in Designing and Conducting Inquiry data collection, data analyses, interpretation), study to compare learning outcomes of a treat- when should the components be mixed, and ment group whose members receive a promising how (and how much) should they be mixed to intervention against those of a control group justify the label mixed methods (see, e.g., using longitudinal data. However, in the service Bazeley, 2009; Bryman, 2007; Johnson & of developing a better understanding of the Onwuegbuzie, 2004; Morse, 2010; Tashakkori intervention, the researcher may decide to add a & Teddle, 2003; Wheeldon, 2010)? Perhaps a preliminary component to the study in which single comprehensive framework outlining the single-subject methods (Kratochwill & Levin, characteristics and components of mixed meth- 1992) will be used to examine the learning tra- ods studies will emerge; on the other hand, it is jectories of a small number of purposively sam- entirely possible that a crazy quilt pattern of pled participants. This would require expertise frameworks will continue to define mixed in single-subject methodology that a researcher methods studies. may or may not possess. Similar examples can be constructed for qualitative and mixed meth- ods studies. Still, the greater challenge for many research- The Study of Promising ers is likely to be modifying personal norms Ideas and Research Design defining scholarship and the adequacy of a contribution to the field. This may require a Earlier descriptions of qualitative, quantitative, researcher to step outside the methodological and mixed methods suggest three related ways boundaries he or she has been trained to honor for researchers to enhance the ability of research and, in some cases, enforce, and to embrace designs to better support the study of promis- research designs the researcher may have been ing ideas. These represent both suggestions taught, and have taught others, are inferior. and challenges. Embracing other methodologies in ways that First, and most important, is for researchers cross disciplinary, funding, and publication to consciously shed narrow definitions of lines, for example, serving as a member of an research design and to embrace more flexible multidisciplinary research team that includes designs that support rather than constrain the (and values) individuals with expertise in quali- study of promising ideas. This may mean using a tative or quantitative methods, will require a familiar research design in an unfamiliar way, certain amount of risk taking but will help to such as substantially modifying elements of a move the field toward more flexible designs. design, for example, the way that data are col- Second, researchers can work (or continue to lected, analyzed, or interpreted in a qualitative work) to modify professional norms in their study, or changing features of the intervention roles as authors, manuscript reviewers, journal being studied or the hypotheses being tested in editors, panel members evaluating grant pro- the middle of a quantitative study because of posals, and so forth, to allow a greater range of preliminary evidence that a different research studies and findings to be supported. This will direction would be more productive. It may require an artful balancing between encourag- mean using a well-known research design to sup- ing risk taking (e.g., studies employing innova- port exploratory rather than confirmatory work, tive interventions or emerging methods of adopting key elements of research designs used qualitative inquiry) and the need to satisfy in other fields, such as dosage-escalation studies standards of research design and reporting that in medicine (Whitehead, Patterson, Webber, help to ensure the integrity of study results. Francis, & Zhou, 2001), or abandoning the sin- Interestingly, there is growing evidence of sup- gular use of qualitative or quantitative methods port for doing just this. Some of this evidence in a study in favor of a mixed methods approach. appears in studies published in journals, such as Part of the challenge of embracing more flex- the American Educational Research Journal, ibility in research designs is technical, to the Educational Evaluation and Policy Analysis, and extent that it requires additional expertise on the the Journal of Mixed Methods Research, that rely part of a researcher or suggests the need to on multidisciplinary research teams. Other evi- develop new research designs. For example, a dence is in the form of new funding programs researcher may plan to conduct a quantitative at the U.S. Department of Education (primarily Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 159 supporting quantitative research), such as the and breadth of a study’s findings. Advances in Race to the Top and Investing in Innovation, data collection, data analysis, report writing, which focus on innovative and ambitious and standards of inquiry and verification have approaches to educational change that draw on been rapid—and in some cases stunning—and multiple disciplines. employing the techniques and procedures that An especially important challenge for modi- reflect these advances should enhance the role fying professional norms lies in the practices of of research design in supporting the study of educational research journals that typically pub- promising ideas. For example, advances in com- lish quantitative studies. The filters enforced by puter hardware and software have significantly the editorial process of these journals often rein- expanded the ways that data can be collected force existing and narrow professional norms. and analyzed, especially non-numerical data Evidence of their success can be found in many such as words and gestures in qualitative research quantitative meta-analyses. In a typical quantita- (Bazeley, 2009; Creswell & Plano Clark, 2007), tive meta-analysis, a sample of published and and the accessibility of techniques to control unpublished studies of a common phenomenon selection bias and take missing data into account in are collected, examined, and combined—for quantitative research (What Works Clearinghouse, example, studies that have introduced interven- 2008). Advances in standards of quality and tions (vouchers, increased teacher professional verification enhance the transparency and development) designed to reduce the achieve- integrity of results (see, e.g., Creswell & Plano ment gap between African American and White Clark, 2007; Shadish et al., 2002). students. Variables capturing key features of each study, such as sample size, percentage of the sample that was African American, and charac- teristics of the interventions are developed, and Conclusion each study is coded using these variables (Cooper, Hedges, & Valentine, 2009). Quantitative analy- The pursuit of promising ideas in educational ses of the resulting data typically provide evi- research sounds noble, and it is. In this spirit, the dence of the impact of editorial practices. task of this chapter was not to reexamine or For example, limiting the analyses to published reignite the conflict between qualitative and studies will typically reveal that all (or virtually all) quantitative methods, nor to assess the peace- reported at least one statistically significant result making capacity of mixed methods, but rather (see, e.g., Borman, Hewes, Overman, & Brown, to examine the role of research design in sup- 2003; Fukkink, & de Glopper, 1998; Purdie, porting the rigorous study of ideas that are Hattie, & Carroll, 2002), a reflection of the well- believed to be worth studying. documented practice of publishing studies with An examination of qualitative and quantita- statistically significant findings (Bozarth & Roberts, tive methods in education suggests that singular 1972; Hewitt, Mitchell, & Torgerson, 2008; Leech applications of these methodologies will con- et al., 2010; Rosenthal, 1979). Other analyses will tinue to play an important role in research study- often reveal that only a handful of research ing new ideas. Still, there is good reason to designs were used in the pool of sampled studies. believe that mixed methods do indeed represent, The absence of published studies that failed as Teddlie and Tashakkori (2003) argued, a ‘‘third to find evidence of a statistically significant methodological movement’’ (p. 5), which is only effect and the prevalence of studies employing now beginning to mature “as a well-established similar research designs send a powerful mes- methodological alternative with agreed-on foun- sage to researchers: Risk taking is risky, and dations, design, and practices” (p. 287). small successes are valued more than large fail- In arguing for the value of mixed methods, ures. This reinforces a “cookie cutter” approach Creswell and Plano Clark (2007) wondered, to research designs and provides further evi- what would happen if dence of the need to strengthen the connection between educational inquiry and the study of quantitative researchers paid more attention to promising ideas. the incredible range of hypotheses that Third, researchers can consciously use tech- qualitative researchers have generated for them? niques and procedures that enhance the depth And what if qualitative researchers spent more 160 Section III. opportunities and Challenges in Designing and Conducting Inquiry

time exploring the range of phenomena that catalog of research designs available to support quantitative researchers have sought to define rigorous inquiry. Still, change that leads to more and test? (p. 59) researchers embracing more flexible research designs more conspicuously depends on modify- These are important and intriguing questions, ing norms in ways that broaden the range of stud- and it seems clear that flexible research designs ies and findings supported in education. That that aggressively support the study of promising change is underway is undeniable; what is less ideas are needed to help answer them. Mixed clear is the scope and pace of the change. methods designs seem especially well suited for In sum, mixed methods research offers an this task. Whether mixed methods will someday especially promising path toward using enjoy an equal partnership with qualitative and research design in ways that support rigorous quantitative research in education or supplant examinations of promising educational ideas. these methodologies is unclear and in many ways The time to fully embrace mixed methods designs unimportant. What is important is expanding the has come.

References Bryman, A., & Bell, E. (2007). Business research methods. New York: Oxford University Press. Alise, M. A., & Teddlie, C. (2010). A continuation of Buck, G., Cook, K., Quigley, C., Eastwood, J., & the paradigm wars? Prevalence rates of Lucas, Y. (2009). Profiles of urban, low SES, methodological approaches across the social/ African American girls’ attitudes toward behavioral sciences. Journal of Mixed Methods science: A sequential explanatory mixed Research, 4, 103–126. methods study. Journal of Mixed Methods Balnaves, M., & Caputi, P. (2001). Introduction to Research, 3, 386–410. quantitative research methods: An investigative Campbell, D. T., & Stanley, J. C. (1963). Experimental approach. Thousand Oaks, CA: Sage. and quasi-experimental designs for research. Bazeley, P. (2009). Mixed methods data analysis. In Chicago: Rand McNally. E. Halcomb & S. Andrew (Eds.), Mixed methods Caracelli, V. J., & Greene, J. C. (1993). Data analysis research for nursing and the health sciences strategies for mixed-method evaluation designs. (pp. 84–118). London: Wiley-Blackwell. Educational Evaluation and Policy Analysis, 15, Berkenkotter, C. (1991). Paradigm debates, turf wars, 195–207. and the conduct of sociocognitive inquiry in Caracelli, V. J., & Greene, J. C. (1997). Crafting mixed- composition. College Composition and method evaluation designs. In J. Greene & Communication, 42, 151–169. V. Caracelli (Eds.), Advances in mixed-method Bogdan, R. C., & Biklen, S. K. (2003). Qualitative evaluation: The challenges and benefits of integrating research for education: An introduction to diverse paradigms. New directions for evaluation theories and methods (4th ed., pp. 7–42). (No. 74, pp. 19–32). San Francisco: Jossey-Bass. Boston: Allyn & Bacon. Carey, J. W. (1993). Linking qualitative and Borman, G. D., Hewes, G. M., Overman. L. T., & quantitative methods: Integrating cultural Brown, S. (2003). Comprehensive school reform factors into public health. Qualitative Health and achievement: A meta-analysis. Review of Research, 3, 298–318. Educational Research, 73, 125–230. Cheek, J. (2005). The practice and politics of funded Bozarth, J. D., & Roberts, R. R. (1972). Signifying qualitative research. In N. K. Denzin & significant significance. American Psychologist, Y. S. Lincoln (Eds.), The SAGE handbook of 27(8), 774–775. qualitative research (3rd ed., pp. 387–409). Brown, S. (2009). Learning to read: Learning disabled Chubbuck, S. M., & Zembylas, M. (2008). The post-secondary students talk back to special emotional ambivalence of socially just teaching: education. International Journal of Qualitative A case study of a novice urban schoolteacher. Studies in Education, 22, 85–98. American Educational Research Journal, 45, Bryman, A. (2004). Social research methods (2nd ed.). 274–318. Oxford, UK: Oxford University Press. Cochran, W. G., & Cox, G. M. (1950). Experimental Bryman, A. (2007). Barriers to integrating designs. New York: Wiley. quantitative and qualitative research. Journal of Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). Mixed Methods Research, 1, 8–22. The handbook of research synthesis and Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 161

meta-analysis (2nd ed.). New York: Russell Sage Greene, J. C. (2007). Mixed methods in social inquiry. Foundation. New York: Wiley. Creswell, J. W. (1998). Qualitative inquiry and Greene, J. C., & Caracelli, V. J. (1997). Advances in research design: Choosing among five traditions. mixed-method evaluation: The challenges and Thousand Oaks, CA: Sage. benefits of integrating diverse paradigms. New Creswell, J. W. (2003). Qualitative, quantitative, and directions for evaluation (No. 74, pp. 19–32). mixed methods approaches (2nd ed.). Thousand San Francisco: Jossey-Bass. Oaks, CA: Sage. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Creswell, J. W. (2009). Mapping the field of mixed Toward a conceptual framework for mixed- methods research. Journal of Mixed Methods method evaluation design. Educational Research, 3, 95–108. Evaluation and Policy Analysis, 11, 255–274. Creswell, J. W., & Plano Clark, V. L. (2007). Designing Greenstein, T. N. (2006). Methods of family research and conducting mixed methods research. (2nd ed.). Thousand Oaks, CA: Sage. Thousand Oaks, CA: Sage. Haase, J. E., & Myers, S. T. (1988). Reconciling Crotty, M. (1998). The foundations of social research: paradigm assumptions of qualitative and Meaning and perspective in the research process. quantitative research. Western Journal of Nursing London: Sage. Research, 10, 128–137. Day, C., Sammons, P., & Gu, Q. (2008). Combining Harry, B., Sturges, K. M., & Klingner, J. K. (2005). qualitative and quantitative methodologies in Mapping the process: An exemplar of process research on teachers’ lives, work, and and challenge in grounded theory analysis. effectiveness: From integration to synergy. Educational Researcher, 34, 3–13. Educational Researcher, 37, 330–342. Hewitt, C., Mitchell, N., & Torgerson, D. (2008). Denzin, N. K. (2006). The elephant in the living room: Listen to the data when results are not Or extending the conversation about the politics significant. British Medical Journal, 336, of evidence. Qualitative Research, 9, 139–160. 23–25. Denzin, N. K., & Lincoln, Y. S. (2005). Introduction. Hiatt, J. F. (1986). Spirituality, medicine, and healing. In N. K. Denzin & Y. S. Lincoln (Eds.), The Southern Medical Journal, 79, 736–743. SAGE handbook of qualitative research (3rd ed., Howe, K. R. (2009). Positivist dogmas, rhetoric, and pp. 1–29). Thousand Oaks, CA: Sage. the education science question. Educational Freeman, M., de Marrais, K., Preissle, J., Roulston, K., Researcher, 38, 428–440. & St.Pierre, E. A. (2007). Standards of evidence Howell, W. G., Wolf, P. J., Campbell, D. E., & in qualitative research: An incitement to Peterson, P. E. (2002). School vouchers and discourse. Educational Researcher, 36, 25–32. academic performance: Results from three Fukkink, R. G., & de Glopper, K. (1998). Effects of randomized field trials. Journal of Policy instruction in deriving word meaning from Analysis and Management, 21, 191–217. context: A meta-analysis. Review of Educational Jacob, B. A., & Lefgren, L. (2004). Remedial Research, 68, 450–469. education and student achievement: A Gage, N. L. (1989). The paradigm wars and their regression discontinuity analysis. Review of aftermath: A “historical” sketch of research on Economics and Statistics, 86, 226–244. teaching since 1989. Educational Researcher, 18, Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed 4–10. methods research: A research paradigm whose Gaines, E. (2005). Interpreting India, identity, and time has come. Educational Researcher, 33, media from the field: Exploring the 14–26. communicative nature of the exotic other. Johnson, R. B., & Turner, L. A. (2003). Data Qualitative Inquiry, 11, 518–534. collection strategies in mixed methods research. Garner, C., & Raudenbush, S. J. (1991). Neighborhood In A. Tashakkori & C. Teddlie (Eds.), Handbook effects on educational attainment: A multilevel of mixed methods in social and behavioral analysis of the influence of pupil ability, family, research (pp. 297–319). Thousand Oaks, CA: school, and neighborhood. Sociology of Sage. Education, 64, 251–262. Kempthorne, O. (1952). Design and analysis of Gay, L. R., Mills, G., & Airasian, P. W. (2005). experiments. New York: Wiley. Educational research: Competencies for analysis Kerlinger, F. (1964). Foundations of behavioral and applications (8th ed.). Upper Saddle River, research. New York: Holt. NJ: Merrill. Kratochwill, T. R., & Levin, J. R. (1992). Single-case Giddings, L. S. (2006). Mixed-methods research: research design and analysis: New directions for Positivism dressed in drag? Journal of Research psychology and education. Hillsdale, NJ: in Nursing, 11(3), 195–203. Lawrence Erlbaum. 162 Section III. opportunities and Challenges in Designing and Conducting Inquiry

Lawrenz, F., & Huffman, D. (2002). The archipelago Education Research. Washington, DC: National approach to mixed method evaluation. Academies Press. American Journal of Evaluation, 23, 331–338. National Research Council. (2005). Advancing Leech, N. L., Morgan, G. A., Wang, J., & Gliner, J. A. scientific research in education (L. Towne, (2010). Statistical significance and evidence- L. L. Wise, & T. M. Winters, Eds.). Committee based approach: Implications of non-significant on Research in Education. Washington, DC: results. Journal of Non-significant Results in National Academies Press. Education, 1, 1–12. Newman, I., Ridenour, C. S., Newman, C., & Leech, N. L., & Onwuegbuzie, A. J. (2007). A DeMarco, G. M. P. (2003). A typology of typology of mixed methods research designs. research purposes and its relationship to mixed Quality & Quantity: International Journal of methods. In A. Tashakkori & C. Teddlie (Eds.), Methodology, 43, 265–275. Handbook of mixed methods in social and Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic behavioral research (pp. 167–188). Thousand inquiry. Newbury Park, CA: Sage. Oaks, CA: Sage. Maruna, S. (2010). Mixed method research in Norman, K. (2008). High school mathematics criminology: Why not go both ways? In curriculum and the process and accuracy of A. R. Piquero & D. Weisburd (Eds.), Handbook initial mathematics placement for students of quantitative criminology (pp. 123–140). who are admitted into one of the STEM New York: Springer. programs at a research institution. Maxwell, J. A. (2004). Causal explanation, qualitative Unpublished dissertation, University of research, and scientific inquiry in education. Minnesota, Twin Cities. Educational Researcher, 33, 3–11. O’Cathain, A. (2009). Mixed methods research in the Mertens, D. M. (2003). Mixed methods and the health sciences: A quiet revolution. Journal of politics of human research: The transformative- Mixed Methods Research, 3, 3–6. emancipatory perspective. In A. Tashakkori & O’Cathain, A., Murphy, E., & Nicoll, J. (2007). Why, C. Teddlie (Eds.), Handbook of mixed methods and how, mixed methods research is undertaken in social and behavioral research (pp. 135–164). in health services research in England: A mixed Thousand Oaks, CA: Sage. methods study. BMC Health Services Research, 7. Miles, M. B., & Huberman, M. A. (1994). Available at http://www.biomedcentral Qualitative data analysis (2nd ed.). Thousand .com/1472-6963/7/85 Oaks, CA: Sage. Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A. Miller, M., & Miller, T. (2005). Theoretical (2010). Mixed research as a tool for developing application of Holland’s theory to individual quantitative instruments. Journal of Mixed decision-making styles: Implications for career Methods Research, 4, 56–78. counselors. Journal of Employment Counseling, Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M. T., 41, 20–28. Filer, J. D., Wiedmaier, C. D., & Moore, C. W. Morgan, D. L. (1998). Practical strategies for (2007). Students’ perceptions of characteristics combining qualitative and quantitative of effective college teachers: A validity study of methods: Applications to health research. a teaching evaluation form using a mixed- Qualitative Health Research, 3, 362–376. methods analysis. American Educational Morgan, D. L. (2007). Paradigms lost and Research Journal, 44, 113–160. pragmatism regained: Methodological Patton, M. Q. (2002). Qualitative research and implications of combining qualitative and evaluation methods (3rd ed.). Thousand Oaks, quantitative methods. Journal of Mixed Methods CA: Sage. Research, 1, 48–76. Pedhazur, E. J., & Schmelkin, L. P. (1991). Morse, J. M. (2006). The politics of evidence. Measurement, design, and analysis: An Qualitative Health Research, 16, 395–404. integrated approach. Hillsdale, NJ: Lawrence Morse, J. M. (2010). Simultaneous and sequential Erlbaum. qualitative mixed method designs. Qualitative Pegues, H. (2007). Of paradigm wars: Inquiry, 16, 483–491. Constructivism, objectivism, and postmodern Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, stratagem. Educational Forum, 71, 316–330. R. E., Lather, P. A., & Schneider, B. L. (2009). Plano Clark, V. L. (2010). The adoption and practice Learning from our differences: A dialogue of mixed methods: U.S. trends in federally across perspectives on quality in education funded health-related research. Qualitative research. Educational Researcher, 38, 501–517. Inquiry, 16, 428–440. National Research Council. (2002). Scientific research Purdie, N., Hattie, J., & Carroll, A. (2002). A review in education (R. J. Shavelson & L. Towne, Eds.). of the research on interventions for attention Committee on Scientific Principles for deficit hyperactivity disorder: What works Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 163

best? Review of Educational Research, 72, mathematics pipeline. American Educational 61–99. Research Journal, 33, 23–51. Raudenbush, S. W., & Bryk, A. S. (2002). Tashakkori, A. (2009). Are we there yet? The state of Hierarchical linear models: Applications and the mixed methods community. Journal of data analysis methods (2nd ed). Newbury Park, Mixed Methods Research, 3, 287–291. CA: Sage. Tashakkori, A., & Teddlie, C. (2003). Mixed Reichardt, C. S., & Rallis, S. F. (Eds.). (1994). The methodology: Combining qualitative and qualitative–quantitative debate: New quantitative approaches. Thousand Oaks, CA: perspectives. New Directions for Program Sage. Evaluation (no. 61). San Francisco: Jossey-Bass. Taylor, S. J., & Tashakkori, A. (1997). Toward an Roitman, J. (1999). Beyond the math wars. understanding of teachers’ desire for Contemporary Issues in Mathematics Education, participation in decision making. Journal of 36, 123–134. School Leadership, 7, 1–20. Rosenthal, R. (1979). The “file drawer problem” and Teddlie, C., & Tashakkori, A. (2009). Foundations of tolerance for null results. Psychological Bulletin, mixed methods research: Integrating 86, 638–641. quantitative and qualitative approaches in the Sale, J. E. M., Lohfeld, L. H., & Brazil, K. (2002). social and behavioral sciences. Thousand Oaks, Revisiting the quantitative–qualitative debate: CA: Sage. Implications for mixed methods research. Todd, Z., Nerlich, B., McKeown, S., & Clarke, D. D. Quality & Quantity, 36, 43–53. (2004). Mixing methods in psychology: The Sandelowski, M. (2001). Real qualitative researchers integration of qualitative and quantitative do not count: The use of numbers in methods in theory and practice. East Sussex, UK: qualitative research. Research in Nursing & Psychology Press. Health, 24, 230–240. Trochim, W. M. K. (2006). Research methods Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On knowledge base (2nd ed.). Available at http:// quantitizing. Journal of Mixed Methods www.socialresearchmethods.net/kb/ Research, 3, 208–222. Trochim, W. M. K., & Land, D. A. (1982). Designing Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., designs for research. The Researcher, 1, 1–6. & Shavelson, R. J. (2007). Estimating causal What Works Clearinghouse. (2008). Procedures and effects using experimental and observational standards handbook (version 2.0). Washington, designs. Washington, DC: American Educational DC: U.S. Department of Education. Available at Research Association. http://ies.ed.gov/ncee/wwc/help/idocviewer/ Schoenfeld, A. H. (2004). The math wars. doc.aspx?docid=19&tocid=1 Educational Policy, 18, 253–286. Wheeldon, J. (2010). Mapping mixed methods Shadish, W. R., Cook, T. D., & Campbell, D. T. research: Methods, measures, and meaning. (2002). Experimental and quasi-experimental Journal of Mixed Methods Research, 4, 87–102. designs for generalized causal inference (2nd ed). Whitehead, J., Patterson, S., Webber, D., Francis, F., & Boston: Houghton Mifflin. Zhou, Y. (2001). Easy-to-implement Bayesian Smith, J. K., & Hodkinson, P. (2005). Relativism, methods for dose-escalation studies in healthy criteria, and politics. In N. K. Denzin & volunteers. Biostatistics, 2, 47–61. Y. S. Lincoln (Eds.), The SAGE handbook of Winer, B. J. (1962). Statistical principles in qualitative research (3rd ed., pp. 915–932). experimental design. New York: McGraw-Hill. Thousand Oaks, CA: Sage. Yin, R. K. (2006). Mixed methods research: Are the Stage, F. K., & Maple, S. A. (1996). Incompatible methods genuinely integrated or merely goals: Narratives of graduate women in the parallel? Research in the Schools, 13, 41–47.

11

Intellect, Light, and Shadow in Research Design

John P. Bean Indiana University

s people are known by the company research contain legitimate professional infor­ they keep, scholars should be known mation that is then passed on to the next gen­ A by the ideas they keep. A poultice eration of scholars in a field. The promise of against stale ideas resides in the generation of research is that it will give us, as a society, what new ones. Creating valuable new ideas, ideas we need to know to improve our lives. that transform and energize a discipline, is the An axiological question arises for those plan­ province of researchers. Ideas, which are directly ning to do research: Is research of value because related to research topics, are also directly con­ it is true or because it is useful? Truth, the mean­ strained or expanded by research design. Big ing of which is contested by philosophers, the ideas change the way a generation of researchers existence of which is contested by postmodern­ thinks and works, and these ideas transform ists, and the use of which is contested by critical practice. Little ideas are refinements of big theorists, might be unattainable. I use the term ideas, if they are of any use at all. The tension truth as shorthand to mean that which is consis­ between studying big ideas that do not fit neatly tent with observation—if observations can be into existing methods and studying safe and made—and is identified through procedures small ideas that produce predictable but trivial accepted in the discipline. results creates a backdrop against which schol­ arship evolves. Research, as a form of scholarship and cre­ The Bright Promise ative work, is at the core of the academic enter­ of Knowledge prise. It is through research that universities contribute knowledge to society. Research Basic research emphasizes the search for disci­ provides the basis for what is taught in the dis­ plinary truth, whereas applied research empha­ ciplines and how members of a discipline under­ sizes finding out something useful. Both goals stand their professional work. The design of are attractive, and one does not preclude the research has a direct effect on what is discovered, other. Conjoined with the axiological question is the ideas that are created, and what forms of a metaphysical one: Is there an objective reality

165 166 Section III. opportunities and Challenges in Designing and Conducting Inquiry out there that we can discover, or is the world a extension, the purpose of research is to contrib­ product of the imagination, constructed in the ute theories or refinements of existing theories minds of individuals and groups? Researchers to science. Theory is a kind of abstraction, a design studies based on what they believe knowl­ simplification of reality that applies in similar edge to be. The search for an objective truth circumstances and not just to the specific case at involves a different path from the search for hand. For researchers, theories focus attention, individual meaning or a consensus about inter­ limit choices, and provide explanations— subjective meaning. characteristics that give good theories a central In the professions, as opposed to the basic role in research design. For actors in the educa­ disciplines, utility is necessary. Professionals tional environment, they are practical for the provide service to a client based on superior same reasons. knowledge developed from a long study of the Theories about social behavior have inherent disciplinary research. According to Shils (1984), limits. These are identified by Thorngate (1976) what separates academic knowledge from com­ and elaborated on by Weick (1979). Thorngate mon knowledge is that academic knowledge is developed a postulate of commensurate com­ developed by a rigorous methodology. Research­ plexity in which there are trade-offs among a ers in a pure discipline (Biglan, 1973) attempt to theory being general, a theory being accurate, establish truth in that discipline. Researchers in and a theory being simple. A theory cannot be all a profession have as their purpose not to attain three simultaneously; general accurate theories pure knowledge but rather praxis, that is, to are not simple, accurate simple theories are not attain the best knowledge that can be applied in general, and simple general theories are not service of their clients’ needs. To offer education accurate. Weick provides examples of each. In as a societal good, money is spent, programs developing a research design, the theory used, or are funded, and teachers are trained and the one the researcher is trying to develop, has hired. If research is to inform these processes, more important effects on research design than then it is difficult to escape from pragmatism does anything except the topic chosen for the and positivism. research. Theory drives hypotheses. The choice Research that advances methodology is of of a theory to use or develop reflects the value to a field by developing better researchers, researcher’s interest in being general, simple, or but such research is not always of direct use to a accurate and shapes the study accordingly. From researcher’s clientele. A field that emphasizes my observations, I would suggest that educa­ internal debates about philosophy, methodol­ tional research errs on the side of being simple ogy, and definitions can be very lively but is in and, with luck, accurate. danger of becoming irrelevant. Well-designed Theory is emphasized in research because it research should deliver new understandings and provides explanation. Without meaningful new theories. The ultimate test of its value to the descriptions of the situation—that is, without public will not rest with internal elaboration or identifying new ideas to be understood and with faculty members charming other faculty related to each other by theories—research members; rather, it will be seen with improving would not move forward. Designing research understanding, teaching, learning, and organiz­ that identifies the ways in which people in a ing in a heterogeneous society. In what follows, I given situation view their worlds is a sensible discuss some of the primary considerations that starting place for meaningful research. Without should inform research designs. important things to be studied, no theories would need to be developed, and a rigorous methodology to estimate relationships based on Theories theory would not be necessary. Educational researchers are interested in finding out how one thing is related to another; Topics and Ideas describing a set of phenomena; and establishing a basis on which to make claims, predictions, How does one go about selecting substantive and explanations. Braithwaite (1955) writes that issues connected by a theory? Texts covering the purpose of science is theory and that, by research in education or the behavioral sciences Chapter 11. Intellect, Light, and Shadow in Research Design 167 have tended to either be mute on the question spiritual. Of these, verbal/linguistic and logical/ or provide only minimal advice (Gall, Borg, & mathematical topics have dominated educational Gall, 2002; Kerlinger, 1973; Krathwohl, 1988), research, while the others have received little with some improvement lately (Creswell, 2007). study or have been the province of specialized The selection of topics for study is neither inno­ fields (e.g., music). cent nor rational. It is not innocent, because The topics we choose reflect a worldview of being selected gives a topic legitimacy, creates educational research: study the specific rather the power of knowledge for those affected by than the general, emphasize topics related to the topic, and creates invisibility for those linguistic and logical cognitive development, excluded. It is not rational, because researchers and use methods that favor such studies. We choose topics to study based on professional also favor topics that reflect social justice, cog­ interests, not professional mandates, and on nitive development that leads students into self-interest based on what researchers value, economically productive lives, and those topics are curious about, or perceive they will profit that fit within predetermined disciplinary spe­ from. What is studied in a discipline becomes cialties. If education deals with the whole stu­ what is taught in the discipline. Just like what is dent, as Bowen (1977) suggests, it would included in the curriculum is a political deci­ behoove researchers to look for topics that sion as well as an educational decision, what is consider more kinds of intelligence than those studied is not neutral; it implies that what is associated with linguistic and mathematical studied is valuable. reasoning. Having become obsessed with cer­ It is axiomatic that the most important part tainty in our research, we have eschewed broad of any study is the choice of a topic; that is, research topics that require integrating various research findings depend on what is being components of human intelligence in favor of researched. A good topic, when well studied, being able to say with great certainty some­ improves descriptions in a field, better explains thing of diminutive importance. Highly spe­ how theories operate in the discipline, and cialized methodologies require highly refined shows how this knowledge can be applied to topics, which provide little opportunity for benefit clients and society as a whole. The transformational outcomes and the develop­ research problem to be addressed and the state­ ment of big ideas. ment of the purpose of the study focus research activities and limit the scope of the study. If the Choosing Topics purpose is too broad, then the research cannot be accomplished with reasonable effort. If the Identifying a research problem is the starting purpose is too narrow, then the study is trivial. place for research. Research problems involve The statement of purpose is the most important unresolved real-world conditions or situations, sentence in a research proposal. Researchers and it is the research problem that is nested need to avoid making Type III errors—asking between the topic and the purpose. Theoretical the wrong question or not asking the right research problems deal with “we don’t know question—or what I consider to be Type IV why,” descriptive research problems deal with errors, that is, studying the wrong thing. “we don’t know what,” and practical research Because we are well trained, we academics can problems deal with “we don’t know how.” The justify studying practically anything. Politicians best researchers are attracted to uncertainty, can mock this attribute of the academy, in the paradoxes, anomalies, contradictions, and ambi­ past handing out “Golden Fleece” awards for guities in the field. The significance of a prob­ research that seemed to be fleecing the taxpayer. lem is often based on the way in which it inter­ Rather than focusing exclusively on what we do sects with theoretical uncertainty and practical study, it is important to recognize what we choose importance. not to study. Nearly 30 years ago, Gardner (1983) Probably the best way of finding a problem to identified eight areas of intelligence: verbal/ study is to do extensive reading in an important linguistic, logical/mathematical, visual/special, topical area and find out what is poorly under­ physical/kinesthetic/intrapersonal/social, musi­ stood. Many articles contain sections that give cal, intrapersonal/emotional, naturalistic, and suggestions for future research, and many have 168 Section III. opportunities and Challenges in Designing and Conducting Inquiry glaring shortcomings that suggest a problem out the research? Only the third question is should be reexamined from a different perspec­ related to methodology. tive. Some researchers choose a methodology and then try to find a problem to match it. This Different Approaches to Research approach creates an unnecessary constraint on what is to be studied, and the foolishness of this There are many ways in which to study the sequence cannot be emphasized enough. The same topic, and these produce different results. cart does not pull the horse. Mitroff and Kilmann (1978) describe four Other researchers are told what to study by a approaches to research based on a Jungian superior, such as an advisor, a provider of analysis of our predispositions to approach deci­ resources, an admired scholar in the field, or a sion making and obtain information, similar to coinvestigator. So long as the relationship is not the Myers-Briggs Type Indicator tests (Myers- exploitative, joining an ongoing research agenda Briggs, 1962). The four approaches include the has the clear advantage of the person being clear following: the scientist, the conceptual theorist, about what he or she will study. It has the disad­ the conceptual humanist, and the individual vantage of not learning how to define one’s own humanist. Given the topic college student reten- research problem. Typically, a study involves the tion, consider the way in which each methodol­ following iterative process: Approach a topic of ogy is chosen based on the researcher’s interests, interest, read in the area, write a brief statement is different from the other methods, and pro­ of purpose, discuss this statement with others, duces different results. Descriptive phrases below reflect, read, write, talk, and so on, until a com­ are taken from tables in Mitroff and Kilmann on pelling problem with a realistic scope has been the pages indicated. identified and clearly expressed. For the scientist, the approach should be A hot topic is one where there is a lot of objective, causal, cumulative, and progressive, interest and a great likelihood of getting a emphasizing reliability and external validity and project funded, finding others to participate in separating the scientist from the observed. It the study, and publishing the results. Good aims at precise, unambiguous empirical knowl­ topics allow young scholars to demonstrate edge using strict logic (Mitroff & Kilmann, 1978, their research skills and increase the likeli­ p. 34). The norms of this approach are known as hood of getting further support for their the CUDOS: research and publications in their fields. For better or worse, it usually means becoming Communism, indicating that scientific more specialized. With luck, a good topic is knowledge is common property; one the researcher loves. Here, love is a pas­ Universalism, indicating that scientific knowledge sionate attachment to the research and an should be independent of the personality of the enjoyment of the research process. This attach­ individual scientist; ment should not be confused with a lack of Disinterestedness, such that the scientist should objectivity, but it involves caring about an observe what happens and not advocate a theory increased understanding of a topic and a will­ or experimental outcome; and ingness to put forward the effort that results in influential research. Organized After deciding what to study and why such a Skepticism, where scientists should be critical of study is worthwhile, the final part of designing their own and others’ ideas. (Merton, 1942/1973, research is to decide on the processes by which p. 269) the research is to be accomplished. Many people, in thinking about research design, think that An example of the scientific study of reten­ they need only be concerned with research tion would be an organizational experiment methodology (Kerlinger, 1973, cited in Daniel, based on the hypothesis that higher-achieving 1996). The following three questions always students are more likely to remain enrolled in affect research design: (1) What will the college. Students would be randomly assigned to researcher study? (2) Why is the research a treatment group or a control group. In the important? (3) How will the researcher carry treatment group, students would participate in a Chapter 11. Intellect, Light, and Shadow in Research Design 169 retention program, such as developing study better understand how the issues facing students skills, but otherwise would have experiences no contribute to their leaving, and tries to alleviate different from those of the control group. After those conditions. The purpose is to increase the a given time period, the researcher would find overall retention rate with the belief that stu­ out whether the retention rate for students who dents who complete college lead richer lives. participated in the retention program was sig­ An individual humanist addresses inquiry as nificantly different from that for students who a personal, value-constituted, interested, and did not. This information would be used to sup­ partisan activity, engaging in poetic, political, port or negate the hypothesis. acausal, and nonrational discourse in pursuit The conceptual theorist is involved with of knowledge. Intense personal knowledge research that is impersonal, value-free, disinter­ and experience are highly valued, aiming to help ested, imaginative, and problematic, involving this person to know himself or herself uniquely multiple causation, purposeful ambiguity, and and to achieve her or his own self-determina­ uncertainty. The theorist is interested in the tion. The logic of the unique and singular has conflict between antithetical imaginative theo­ mythical, mystical, and transcendental over­ ries, comprehensive holistic theories, and ever tones that operate as counternorms to the expanding research programs to produce con­ CUDOS (Mitroff & Kilmann, 1978, p. 95). A flicting schemas using dialectical and indetermi­ retention study from this perspective would try nate logics (Mitroff & Kilmann, 1978, p. 56). A to develop a detailed understanding of a single theorist conducting retention research would student in the full context of his or her life. It provide at least two theoretical explanations of could take the form of an “N of 1” case study: a retention behavior, use survey methods to gather phenomenological inquiry into who the student information, analyze the data using statistics, is, what the student finds at school, and how find out whether the data supported one theory staying or leaving school would be better for this more than the other, and use that information to particular individual. make more theories. Much of the empirical The purpose of presenting these four research reported in educational journals is a perspectives—and many more can be imagined— combination of the scientist and theorist— is to illustrate that there is no best way in which theory guiding social science inquiry into to study a topic. Different kinds of studies make educational structures and processes. different kinds of assumptions about what is The conceptual humanist (although I find important to know, serve different needs for social humanist to be a more accurate descrip­ different people involved in the studies, and tion) approaches research as a value-constituted, produce different kinds of outcomes. The four interested activity that is holistic, political, and perspectives were presented in what was once imaginative; where multiple causations are pres­ considered the normative order of acceptability: ent in an uncertain and problematic social envi­ science-based research, theory development, ronment; and with a deep concern for humanity. action research, and phenomenology. One may This approach recognizes the importance of be no more correct than the others. Some are the relationship between the inquirer and the more acceptable to certain audiences than to subject and has the aim of promoting human others, and each produces a particular out­ development on the widest possible scale. The come that favors some stakeholders more than it normative outcomes of such research would be does others. economic plenty, aesthetic beauty, and human welfare. Similar to an action researcher, the Methodology and the Scientific Approach social humanist prefers small-group dynamics where both the inquirer and the participants Methodology is often considered the core of learn to know themselves better and work research design. Kerlinger (1973, cited in Daniel, together to improve the situation (Mitroff & 1996) described as one of the research myths the Kilmann, 1978, p. 76). A retention researcher idea that research design and research methods using this approach could develop an ongoing were synonymous, even though many research­ program of action-oriented ethnographic ers held this view. Methodology is the tool used research studies, where the researcher comes to to accomplish part of the study, specifically, how 170 Section III. opportunities and Challenges in Designing and Conducting Inquiry to obtain and analyze data. It is subservient to In the same way that social science research choosing an important topic to study, matching has imitated the natural sciences, educational the research problem and the methodology, and research has imitated social science research. knowing what the results mean and how they Research in these areas may be separated more can be applied. To do good research, the meth­ by the topic studied than by the rigor of the odology used should be appropriate for the methodology. As with social science research problem addressed. This is a necessary condi­ in general, educational research might pre­ tion but not a sufficient one. An elegantly ana­ tend a level of control so as to carry out an lyzed data set that was composed of ambiguously experiment. Researchers give themselves sol­ measured data that addressed a question of triv­ ace that “other things are equal,” or “other ial importance is not likely to enter the annals of effects are random,” or “spuriousness is not a great research. problem,” and proceed as if the social world Educational research is part of the social sci­ were simple and understandable in the same ence research tradition, a tradition that was way that the traditional world of the pure sci­ influenced by research in the natural sciences. ences can be. The natural sciences use the scientific method to solve research problems or support a perspec­ A Traditional Approach to Designing tive. The method contains a series of sequential Educational Research steps similar to the following: Identify a prob­ lem, gather information from the literature Graduate programs in education typically are about this question, develop a hypothesis in the arranged in departments that reflect academic context of a theory, collect data related to the subspecialties such as history of education, soci­ hypothesis, analyze the data, and draw a conclu­ ology of education, anthropology of education, sion related to the truthfulness of the hypothesis counseling psychology, experimental psychol­ and correctness of the theory. ogy, higher education, school administration, Scientists, as logical purists, build argu­ and curriculum and instruction. Most of these ments on falsifiability and the law of the programs require a course in research design excluded middle. This law states that A and appropriate for their field. My discussion not-A cannot exist simultaneously. But if A revolves around quantitative and qualitative stands for “this program helps students to approaches to research, terms that reify catego­ learn” and not-A stands for “this program does ries that are themselves overlapping and arbi­ not help students to learn,” then both can be trary, but are commonly used to describe true, as in the case of aptitude–treatment inter­ research courses. actions; that is, a treatment could be effective A simple way of looking at a proposal is to see for a high-aptitude student but not effective for how it answers the three questions posed earlier: a low-aptitude student. If both are true, then What is this study about? Why is the study the law of the excluded middle is violated and important? How will the researcher conduct the falsifiability cannot be demonstrated. This sit­ study? The study itself must cover these three uation is problematic for scientific research questions and answer two additional questions: in education. What did the researcher find? What do the find­ Education is not a scientifically based process, ings mean? Research designs revolve around a partly because the term education is ideological limited number of elements. Their exact use and and idiosyncratic, much different from the term exposition vary depending on the particular temperature. At best, scientific research can shed approach taken. The purpose and promise of light on narrowly defined educational behaviors, these elements have been identified and dis­ and researchers can hope for—but cannot cussed by a number of textbooks such as those guarantee—a cumulative effect. When a govern­ of Gall and colleagues (2002) and Creswell ment policy assumes that education is equivalent (2007). These texts identify many of the issues to improving the score on a test, the society will facing researchers, especially those who are new not have a moral compass and will not be edu­ to the process. cated. Feyerabend (1993) holds the view that if Although not suitable for all studies, well- we do not separate scientific research and the designed quantitative research usually ad- state, as we have separated the church and the dresses the areas presented in the following state, irreparable harm will be done. outline: Chapter 11. Intellect, Light, and Shadow in Research Design 171

(1998)—biography, ethnography, grounded the­ I. Introduction to the topic of the research ory, case study, and phenomenology—which can a. Background and context in which that be used singly or in combination. General head­ topic has been studied ings appropriate for qualitative studies include the b. Importance of studying the topic, topic, focus and purpose, significance, related liter­ including ature, methodology, presentation of the data, inter­ c. Practical value of the study pretation of the data, and conclusions. Detailed d. Research problem to be addressed headings would be similar to the following: e. Purpose of the study f. Objectives or questions to be addressed g. Definitions and related constructs I. Topic to be studied h. Assumptions used in the study a. The overall interest focusing on what i. Limitations of the study will be explained or described j. Scope of the study b. Organizing metaphor (like grounded II. Using the literature to build conceptual theory) arguments c. The mystery and the detective a. Relevant theories to guide the study d. Hermeneutic elements b. The findings of other researchers that e. The significance of the study identify important variables related to f. Why the reader should be interested. the purpose II. Getting information away from the source c. Identification of the dependent a. Relevant literature variable, independent variables, and b. Theories theories linking these variables c. Findings for content area, themes, foci, III. Methodology to be used in the study or analogous situations a. Site, sample, or selection procedure for III. The selected qualitative method respondents, including a. How information is obtained b. How the data will be gathered b. How sense is made from it c. How the data will be measured c. Site selection d. How the data will be analyzed d. Informant selection e. Why these methods are appropriate. e. Data collection [This information usually completes f. Data analysis the design of a research proposal and g. Data evaluation appears as the first part of a finished [The methodology can be in a separate study. Finished studies also include the chapter, part of the first chapter, in an following.] appendix, or woven into the chapters IV. Findings that present respondent information. a. A description of the sample actually This section is followed by one or more analyzed in the study chapters which does the following.] b. Description of the data IV. Present the text as natural answers to c. Treatment of missing cases natural questions. d. Possible or known biases e. Description of how the researcher met a. Presentation formats include stories, the assumptions required to use the tables, interviews, documents, chosen statistics narratives, photographs, videotapes, f. Presentation of the data vignettes, texts of various kinds, g. Support or lack of support for the descriptions, and routines (see hypotheses or theories used Schwartzman [1993], from which h. Discussion of the findings some of these headings were taken). i. Conclusions b. Raw data can be presented without comment, presented and interpreted V. Summary of the study simultaneously, or presented and a. Practical implications of the study interpreted in a later section. b. Areas for future research V. Findings a. Conclusions Qualitative research can involve using the b. Recommendations for others five research traditions identified by Creswell 172 Section III. opportunities and Challenges in Designing and Conducting Inquiry

The structure of qualitative studies is more be analogous to the study of turnover in work idiosyncratic than the more formal structure of organizations, and the literature in one area can traditional quantitative studies. In qualitative be used to reinforce the literature in the other research, the sample is the study, and the reasons area (Bean, 1980). The use of literature in quan­ for selecting the sample need to be emphasized. titative studies, however, can differ substantially Most researchers, when approaching a topic from the use of literature in qualitative studies. they care about, have tentative hypotheses about In a quantitative study, the literature is used what causes what or predispositions to thinking to identify the importance of the dependent that the world operates according to certain variable, relevant independent variables, and principles that also apply in this area. People theories that bind these factors together, to jus­ bias their observations based on their experi­ tify the use of statistical procedures and to pro­ ence. All of us know that we have certain biases, vide a context for the discussion. In qualitative and we can try to counter those biases in our studies, a premium is placed on the ability to see research by looking for evidence that directly what is before the researcher. Our ability to contradicts what we expect. In the unconscious, observe is both heightened and diminished by there is a second set of biases of which, by defini­ our prior knowledge and expectations (Bean, tion, we are not aware. Sometimes peer readers 1997). It is heightened by making ourselves can help the researcher to discover what is miss­ aware of important details to observe, and it is ing or what is inappropriately under- or over­ diminished because we focus only on those emphasized in the study. details. Due to the preconceived notions of the After analyzing the data in a quantitative researcher, those factors actually influencing the study, the researcher presents the findings. respondents’ world might not be identified. Typically, it is rather straightforward, because When the literature shapes the way in which we the data to be gathered and the analyses pro­ view the world, what is actually before us is posed for the data were specified in the proposal replaced by what we expect to see. for the study. For qualitative researchers, the A review of the literature, as a stand-alone data, the findings, and the method might not be section summarizing research in the topical distinct. The narrative that presents selected area, makes little sense. The literature, as a com­ questions and answers can represent findings pendium of related information, should be used based on data that came from the method by to advance arguments related to the importance which questions were developed. As the previ­ of the subject. It should identify topical areas ous sentence suggests, it is a convoluted process. that are either well or poorly understood, iden­ The presentation might revolve around respon­ tify and describe relevant theories, identify and dents’ experiences and understandings, a chro­ describe appropriate methodologies to study the nology of events, or themes supported by topic, describe dependent and independent respondents’ statements. In ethnographic stud­ variables if relevant, provide definitions, and ies, the descriptions of lives in context can provide a context to discuss the findings from stand on their own (Lawrence-Lightfoot, 1995). the study. Thick descriptions (Geertz, 1973) might pro­ Google Scholar, Dissertation Abstracts vide greater insight into the education of a stu­ International, ERIC Documents, the ISI Web of dent in a school than would an analysis of Knowledge, and the proceedings of relevant variables. In most studies, some analysis of the professional organizations all can be used to descriptions is expected. This predisposition is access current research. A condemning retort is part of the legacy of pragmatism; researchers that the literature in a study is dated. This phrase in education are expected to identify how has some interesting subtexts. The first assump­ knowledge gained from the study can improve tion is that the most recent research is the best professional practice. research and that previous research is irrelevant. In designing a study, the background, con­ A second assumption is that all research is of text, importance of the topic, and presumed limited generalizability over time so that if it is practical value of the study come from the lit­ older than, say, 5 years, it is irrelevant. In either erature written about the topic or analogous case, dated research is of marginal value. By literatures in similar fields. For example, the study extension, the research that a person is currently of college student retention can be considered to conducting is also of marginal value because it Chapter 11. Intellect, Light, and Shadow in Research Design 173 will be useful for only 5 years. This planned linear modeling (GLIM) and hierarchical linear obsolescence of research becomes a justification modeling (HLM). for a frenzied increase in the rate of publication The increase in complexity is associated with and is counterproductive in terms of identifying an increase in agitated exchanges between statis­ important and durable ideas in the field. ticians about whose method is correct. An Literature should not be weighed nor consid­ improved methodology has not been matched ered dated after 5 years. by these studies becoming more influential in In traditional quantitative research, the topic policy making or practice (Kezar, 2000). The contains the dependent variable, and the factors debate is sometimes invisible to the public, tak­ associated with it identify the independent vari­ ing place between the author of a piece of ables that have been found to have important research and the consulting editors who review effects on the dependent variable. In studies that the research, and sometimes it appears in jour­ are not codifications—that is, not extensive nals such as the Educational Researcher. reviews of the literature for the heuristic pur­ pose of organizing what is known about a Data topic—citing the literature should be done for the purpose of building an argument, not sim­ Quantitative studies require data that can be ply to show familiarity with the canon. used in statistical analyses. The sources of data Since the 1960s, the number of statistical can vary widely—historical documents, govern­ analyses available for researchers to include in mental records, organizational records, inter­ their designs has increased dramatically. Five views, standardized surveys, questionnaires commercial statistical packages bear the initials developed as part of the research protocol for a SAS, SPSS, BMDP, GLIM, and HLM. The devel­ particular study, unobtrusive measures, obser­ opment of these statistical packages has allowed vations, participant observation, and so on. The ever more complex analyses to be performed. quality of the research depends on the quality of National data sets from the National Center for the data analyzed; data analysis has only a sec­ Educational Statistics (NCES) and other sources ondary influence. have provided the opportunity to bring order to The quality of the data varies greatly. Good vast amounts of data. research design requires that the researcher For the description of large-scale phenom­ understand the strengths and weaknesses of the ena, these data sets can be very valuable. For data. Historical data can reflect the biases and analyzing the causes of behavior, however, the ideological preferences of those who recorded it. attempt to gain a broad vision masks individual People who provide data can intentionally dis­ or small-group differences. Longitudinal studies tort it to put themselves in a better light, for almost always suffer from decay; that is, mea­ example, reporting that they had higher grades sures may differ from year to year and respon­ than they actually did. Survey data might come dents drop out of the study. So the comparisons from a biased sample reflecting only the experi­ from year to year might not be the result of what ences of high–socioeconomic status respondents. people report; rather, they might be the result of Questions in a survey might be ambiguously changes in who is doing the reporting. written, or a single item might contain two The availability of data and the means to questions with different answers, for example, analyze them raised the level of expectation “How satisfied are you with your salary and in some journals that such analyses should fringe benefits?” Survey data that require a be the norm. What is certain is that during forced-choice response might not represent the the past 50 years, the sophistication of analyses real interests of the respondent. A respondent has increased. The literature shifted from might have no opinion on most of the questions normed surveys that reported frequencies, to and refuse to answer them. Other respondents chi squares, to analyses of variance (ANOVAs) might not want to reveal personal information and simple correlations, to factor analysis and and so might misrepresent their actual incomes, multiple regression, to causal modeling with whether they have ever plagiarized, or how ordinary least squares path analysis, to maxi­ much they use drugs or alcohol. Although the mum likelihood used in linear structural rela­ questionnaire is not missing any data, the data tions (LISREL) modeling, and to generalized provided might be intentionally inaccurate. 174 Section III. opportunities and Challenges in Designing and Conducting Inquiry

In other cases, respondents might not under­ Milgram, 1974) and medical research where stand the questions, might not care about the participants were given diseases or intentionally answers given, or might become fatigued while denied treatment (Jones, 1993). filling out the questionnaire, so that the accuracy The bureaucratic response to these ethical of the responses is different for the beginning violations was to create rules that would include and the end of the questionnaire. A well-written everybody doing any kind of research that question should reflect one bit of information involved contact with living people. Bureaucratic about the respondent unambiguously and reli­ actors, evaluating research they are not conduct­ ably, and the answer to the question should ing themselves, become the gatekeepers of ethi­ match observable facts. cal behavior. This responsibility is misplaced; It is acceptable to use problematic data if the researchers themselves should be responsible for analyst understands and acknowledges the prob­ protecting the interests of participants in their lems that exist in the data. For example, a data set studies. I am not naïve enough to think that all might not be random but might be completely researchers are ethical or that institutional representative of one subgroup in the population review boards (IRBs) or protection of human studied. The bias in this sample can make con­ subjects committees will go away. The problem clusions drawn about the well-represented group is that ethical judgments about research have accurate, but the conclusions would not apply to been made extrinsic to the research process. the whole population. Although not representa­ Researchers need to design research that does tive, the data might be useful to see whether a just what the committees want—to protect the hypothesized relationship exists at all, that is, as participants of a study from harm. If researchers a test of theory. are not socialized to provide these protections, Data gathered from face-to-face interviews IRBs might not help. The enforcement system for qualitative research have the potential to used, which involves taking federal support away yield a gold mine of insights into the people’s from ethical researchers because they happen to lives and situations. There is no substitute for be at an institution where one person did not prolonged and focused conversations between comply with the guidelines, is a collective pun­ trusted parties to discover what is important to ishment which is itself unethical. IRBs have the the interviewees and how respondents under­ enormous power of being able to block research, stand key elements in their own lives. When and the potential for abusing power must be mishandled, interview data can reflect what the kept under constant scrutiny. interviewees think the interviewers want to hear, For qualitative researchers especially, com­ normatively appropriate responses, the fears and plying with a written informed consent form biases of the interviewees, and the fears and can damage the trust required to conduct a biases of the interviewers. Data flaws become study. The study of any group that dislikes limitations of the study for which the only authority is made impossible, or at least less reli­ response is to caution the reader that the results able, by asking participants at the outset to sign are far from certain. a form that says, “You should know that this researcher does not intend to hurt you.” A jour­ nalist and an ethnographer can conduct and Ethics and Institutional Review Boards publish identical studies. However, the journal­ Before proceeding with an examination of ist needs no informed consent from those who research methods, there are some ethical and are interviewed for the story, whereas the eth­ legal considerations that have obtrusively en-­ nographer at a research institute needs IRB tered the development of a research protocol. In permission to ask the same questions. The jour­ line with designing research to be useful, it nalist is protected by freedom of speech, whereas should also be designed to be ethical. The most academic freedom, according to IRB rules, pro­ obvious ethical problems arise when a research vides no such protection for the researcher. procedure causes harm to those who are asked While much of the antagonism between or forced to participate in the process. There are IRBs and the faculty involve what are seen as several well-known cases of abuse, including nuisance parameters, as hurdles to be jumped, psychological studies where participants were put IRB guidelines constitute a direct attack on in unusually stressful situations (Baumrind, 1964; academic freedom. When a faculty member has Chapter 11. Intellect, Light, and Shadow in Research Design 175 to get permission to “do research as he or she at that point that their probabilistic, relative sees fit,” there is no academic freedom. There nature comes into sharpest focus. (p. 116) are three traditional threats to academic free­ dom: economics, religion, and politics. One Does what works 90% of the time for the could argue that IRBs are simply a new form of participants in a study work for one particular the economic threat, being that failure to fol­ teacher in one particular class dealing with one low IRB guidelines will result in the loss of particular subject? Tutoring generally helps stu­ substantial federal funding for research at the dents to learn how to read, but for a student who institution. Too often, I fear, these offices dis­ is acting out against authority, and who views place their goals of protecting human subjects the tutor as an authority figure, tutoring might and replace them with the goal of making prevent the student from learning to read. researchers follow their rules. The direct and As a “reductionistic fallacy” (Lincoln & opportunity costs to institutions in wasting Guba, 1985, p. 117), generalization simplifies faculty time is real, but the greater problem is decision making and simultaneously reduces researcher self-censorship—not doing research the understanding of the particular. Teachers because of the fear that it will not be approved operate in particular environments, and the by the IRB. Academic freedom is the most cen­ findings from a “scientific” study with a high tral value in American higher education, and degree of generalizability do not ensure a pro­ anything that puts it at risk needs greater justi­ gram’s utility in a given classroom. The purpose fication than IRBs have provided for their of scientific research is to eliminate uncertainty behavior. Regardless of what happens to IRBs, so that the operator can predict and control the research should be designed to protect every­ future. Applied to education, this goal is not a one, to benefit the participants in the research, research norm or a teaching norm but rather a and to protect society from ignorance. political norm.

Generalizability The Shadow of Research Design Generalizability is the central bulwark of the scientific research in education approach. In a Research is seductive because it promises to give 2002 National Research Council report, the edi­ the participants, as producers or consumers, tors observed, “Regularity in the patterns across those things that they imagine they want. We are groups and across time—rather than replication seduced by research, viewing it as a beatific pro­ per se—is a source of generalization. The goal of cess by which we can glimpse the bright light of such scientific methods, of course, remains the pure knowledge. Scholars would have no agenda same: to identify generalized patterns” (p. 82). other than furthering knowledge, a value shared Generalizability is a powerful statistical tool by those who fund research, publish it, and base that allows researchers to make predictions policy on it. It would be a collegial environment about patterns of behavior in a population, such where differences exist only about approach, all as the percentage of people who will vote as participants share the ultimate goals of research, independents, based on a measure of that behav­ and no ethical problems exist. These utopian ior taken from a sample of the population. It is goals include a greater understanding of indi­ attractive to policy makers because it suggests vidual and group processes in a given discipline, the extent to which a particular solution will with the potential to apply these findings to work everywhere in the population. As the improve both individual lives and society col­ behavior in question gets more complicated, lectively. Researchers would design their studies such as how students learn ethical behavior, for the sole purpose of sharing information to generalization is of more limited value. Lincoln better understand the issues at hand and distrib­ and Guba (1985) describe this limitation well: ute the knowledge widely so that it can improve practice. Generalizations are nomothetic in nature, that is, The shadow of research design comes as a lawlike, but in order to use them—for purposes series of dispositions and paradoxes when the of prediction or control, say—the generalizations person designing research must make decisions must be applied to particulars. And it is precisely for which the search for disciplinary truth 176 Section III. opportunities and Challenges in Designing and Conducting Inquiry provides no direction. A researcher has some (Parnes, 1992). However, the content of the steps control, but not complete control, over deciding is not well understood. If it were, then everyone what research to conduct. A researcher has lim­ would be creative and have plenty of excellent ited control, or no control, over how research is problems around which to design research. funded, how it is evaluated, and how it is used. To be creative, as opposed to simply novel, a The shadow of research appears when one con­ researcher should be well-versed in substantive fronts the lack of creativity in research and psy­ knowledge of the topic and the limitations of chological barriers to the free flow of ideas. It the chosen methodology. Creativity has at its occurs when a researcher encounters difficulties foundation a sense of play—of suspending nor­ related to the disciplinary research environment mal constraints so as to see new patterns, possi­ and the primary and secondary social environ­ bilities, or connections. Play is usually an “idea ments associated with the research. non grata” in a workaholic environment, although the hermeneutical philosopher Godamer con­ sidered play to be an expression of great serious­ The Loss of Creativity ness (Neill & Ridley, 1995). Play is characterized If the world were static, then creativity would by just those things that are likely to lead a not be necessary; what worked in the past would researcher into creative work, including taking continue to work in the future. In a dynamic risks, testing new ideas in safety, avoiding rigidity, social world existing in a turbulent ecology, the and suspending judgment (Schwartzman, 1978). generation of new ideas is necessary for survival. A risk-averse, judgmental, assessment-oriented In the natural world, mutation is a random pro­ environment focused on short-term gains will cess, and selection occurs where the fit to the have a negative effect on creativity. If proposals natural environment of the new form has advan­ are assessed by published criteria, then how can tages over existing forms. In the social world, new projects that do not fit established criteria creativity is the source of variation and must be funded? We live in a judgment-rich environ­ be present before selection can take place. ment, where we have been socialized for years Without creativity in identifying problems to be into viewing work as something that will be addressed or methods to be used, a field of study graded. Peer reviews, editorial reviews, adminis­ would atrophy. trative reviews, and granting agency reviews If research has a core more important than occur regularly. Faculty work can be assessed on anything else, it is creativity. Without creativity, an annual basis, with the expectation of prod­ researchers would only repeat themselves. ucts in hand making the time frame for com­ Without creativity, the questions we pose, the pleting work within a year or less. In graduate starting place for research design, would be end­ schools, students are steered out of creative lessly repetitive. Creativity allows the clientele of projects because such projects are too risky. It is researchers—be they the public, practitioners, unfortunate when research is designed not out or other researchers—to bring new ideas into of the possibility of success but rather out of the their intellectual or practical lives. They can fear of failure. In the process, creativity—researchers’ agree or disagree with each other’s findings. best friend and asset—is shunted to the rear. They can find fault in their methodologies. But Academic reproduction (Bourdieu, 1984/1988; the new ideas remain as work to be examined, Bourdieu & Passeron, 1990) ensures repro­ understood, enacted, selected, and retained for duction, not evolution. Creativity suffers in the use (Weick, 1979). current context of conducting research, and Getzels and Csikszentmihalyi (1976) describe producing valuable new understandings is ever problem finding as being at the heart of the more difficult. creative process. Educational researchers who invent the best problems have the greatest Fear and the Researcher’s Ego chance of contributing to their fields. A good problem implies the way in which it should be There are a number of personal factors that studied. A superb methodology will not make affect research design. Morgan (1997) describes up for a poor research problem. Structured pro­ “psychic prisons” as a metaphor for the ways cesses for becoming more creative that empha­ in which our imaginations become trapped. size steps to be followed have been identified Whatever neuroses we have can emerge in the Chapter 11. Intellect, Light, and Shadow in Research Design 177 task of developing research. Researchers can fix­ There is a competitive aspect to designing ate on certain ideas, repress others, idealize research. Instead of a “best knowledge for the states, or project their own views on the data. A discipline” model, it involves “I got there first,” frequently occurring form of projection occurs “I’m right and you’re wrong,” “I win the argu­ when the conclusions of a research study are not ment,” “My theory is right,” “I got the grant and connected to the data. Researchers project their you didn’t,” “My university is ranked higher than beliefs onto the data, concluding what they your university,” and the like. These are the con­ wanted to conclude before they began conduct­ cerns of the ego, extrinsic to the creation of ing the study. knowledge, casting a shadow on research. From Fear has an unacknowledged influence on the point of view of designing research to dis­ research design that manifests itself in a variety cover knowledge, it is bizarre that information is of ways. The first is internal censorship. Certain not shared. From the point of view that research topics and methods are never given serious con­ is not about improving knowledge but rather is sideration because to do so would be to invite about supporting the ego, making a name for trouble, at least in the minds of the researchers. oneself, and providing research overhead to For example, during the 1970s, many people one’s institution, it makes perfect sense. The did not consider qualitative research to be an impulse is to design research in order to win appropriate form of educational research. Fear­ some imaginary (or real) competition, not ing rejection by colleagues, granting agencies, because it is vital to the field. advisers, or editors, researchers steered them­ selves away from the use of qualitative research. Disciplinary Norms and Groupthink It was not surprising that much of the emphasis of Lincoln and Guba’s (1985) Naturalistic The kind of study a researcher can conduct Inquiry is a justification for, and not an explana­ depends on the development of the field. Mature tion of, this kind of study. Researchers engaged fields, such as the arts and sciences, medicine, in self-censorship by avoiding Black studies, and engineering (Parsons & Platt, 1973), have a women’s studies, GLBT (gay, lesbian, bisexual, long tradition of theories and methods that are transgendered) studies, and the study of emo­ thought to be appropriate to use when conduct­ tional aspects of organizational behavior ing research. Cultural studies, critical theory, (Fineman, 2000). and other postmodern approaches have pre­ Fear also underlies what has been called the ferred methods that keep other disciplines vital “imposter syndrome” (Harvey & Katz, 1985), by challenging their traditions. Research norms where researchers might fear that they are fakes. become institutionalized through accepting This problem can show up in an obsessive need papers for professional meetings and publica­ to review the literature because a researcher tion. A disciplinary language develops, and a “doesn’t know enough yet.” A researcher might kind of parochialism develops in citation pat­ fear not being professional enough or not being terns: Cite from journals in the field only. thorough enough and might analyze data in an Disciplines within education and in the profes­ endless pattern of trivial changes. This fear is a sions have become ever more specialized. pathology, not a motivator. Research followed suit and led the way to disci­ Research can also be conducted in service to plinary specialization. the ego, not the discipline, where the researcher Research design reflects this specialization in is driven by the extrinsic value of research. This topic and method. Specialization can have the drive results in designing research for maximum advantage of accuracy and the disadvantage of visibility, regardless of substance. Finding the triviality. Researchers who venture outside the smallest publishable unit in a data set inflates norms can be transformational if they are lucky one’s résumé but clutters journals. The ego or can be ignored or ridiculed if they are not. thrives on high levels of productivity. The disci­ New ideas are sometimes blocked by the disci­ pline thrives on high levels of quality. The cur­ plinary equivalent of groupthink. Groupthink, rent research market defines what is acceptable, first described by Janis (1972), includes many and clever marketing may be more important to factors that limit creativity and risk taking, one’s ego than a quiet but long-term contribution including sharing stereotypes that guide the to the field. decision, exerting direct pressure on others, 178 Section III. opportunities and Challenges in Designing and Conducting Inquiry maintaining the illusion of unanimity and Simple solutions are often wrong. Random­ invulnerability, and using mind guards to pro­ ization means that individual student differ­ tect the group from negative information. ences will not be a factor in the research and that Groupthink is more than a norm; it is an all kinds of students can expect to benefit exclusionary process designed to protect the equally from the program. The results are group from outside influence. Groupthink in designed to mask individual differences to see research can limit the topics studied and the whether the program worked for the majority. It methodology used. The long period during works if the mean of the criterion variable for which editors silenced the voices of women; the treatment group is significantly higher than African Americans; and the gay, lesbian, bisex­ the mean of the control group. Like randomiza­ ual, and transgendered community in education tion, means are designed to mask individual is one example. Another currently exists among differences. Berliner (2002) makes the point that those in education who support only “scientific there is a “ubiquity of interactions” and that a research” (National Research Council, 2002). program could have remarkable positive effects Berliner (2002) suggests that the problem is not on a small segment of the treated population, one of science but rather one of politics and none of which would be discovered by this money. Those who label good research in educa­ research design. A program could benefit gifted tion as “scientific” are stuck in groupthink, as are students, African American students, girls, ath­ those who consider the research method as letes, or special needs students in a manner essential and all else as trivial. When methodol­ invisible to scientific methods. ogy precedes identifying the problem to be stud­ Much of the contentiousness about educa­ ied, groupthink wins and research suffers. tional research design centers on whether the research is scientific, a desiderata identified by the National Research Council’s (NRC) publica­ Methodology and Methodological tion of Scientific Research in Education (2002). Correctness The debate revolves around using scientific At the extreme, the result is “methodological methodologies to examine educational issues. correctness,” a term I coin as a play on “political The NRC’s position is generally supported by correctness.” It is associated with taking oneself some (Feuer, Towne, & Shavelson, 2002; Slavin, very seriously and is related to academic funda­ 2002) and cautioned against or rejected by oth­ mentalism, where skepticism is replaced by ers (Berliner, 2002; Erickson & Gutierrez, 2002; dogma. Methodological correctness means that Olson, 2004; St.Pierre, 2002). Research design, the purpose of research is to optimize method­ research funding, and politics are intercon­ ology. It is an example of goal displacement, nected (Burkhardt & Schoenfeld, 2003). The where the purpose of research is no longer to Obama administration has done much to restore find out something important but rather to use the importance of scientific knowledge in policy method flawlessly. The hegemony of methodol­ making, but one can never assume that such a ogists in determining the value of research has a change is permanent. chilling effect on exploring new approaches to A research article, like the tip of an iceberg, research, on studying topics not studied previ­ contains only a small percentage of the informa­ ously, and on studying topics that do not lend tion that the author encountered in the study. themselves to study using preferred methods. Given this situation, research becomes an enact­ Institutionalized methodological correctness ment of the map–territory relationship, that is, takes the form of guidelines, where if the guide­ the relationship between the object studied and lines are not followed, the result is funding not the symbol for that object—the research report being given or results not being taken seriously. (Bateson, 2000). How complete does the symbol The U.S. Department of Education has provided need to be to represent some objective reality? A User-Friendly Guide, one that is not “friendly” Borges (1998), in his story “On Exactitude in at all, that can be summarized as follows: The Science,” provides a fictional example of an only rigorous evidence that can be used to eval­ empire that was so enamored of mapmaking uate an educational intervention comes from that the cartographers were encouraged to make research using randomized controlled trials maps that were larger and more accurate. In the (Institute of Education Sciences, 2003). end, they made a map that was so detailed, it Chapter 11. Intellect, Light, and Shadow in Research Design 179 needed to be exactly the same size as the land it published. The imposter syndrome is ever-present: described. As a map, it was perfectly useless. “How do I, as an editor, make decisions that will In this case, greater accuracy and greater make me look like I know what I’m doing?” The methodological correctness diminished utility. ordinary response is risk aversion: “If I don’t take Bateson (2000) argues that maps are useful not chances, I’m least likely to look like an imposter.” because they are literal representations but Editors are likely to reject methodologically rather because they are in some way analogous flawed research in favor of methodologically to reality. Research provides a map, an analog of correct research. Imaginatively flawed research, reality. If Bateson is right, then it might be more research whose consequences are trivial for the appropriate to design and evaluate research not discipline or practitioners, can be published if the on the basis of how correct the methodology is methods are correct but with predictable disdain or how literally it represents reality, but rather from the public (Kezar, 2000). I have heard of no on how useful it is for understanding and acting cases where an editor has written an author say­ in our environments. ing, “The ideas in this article are so compelling that I’m going to publish it even though it con­ tains obvious methodological flaws.” Editorial The Primary Research Audience referees work at the pleasure of the editor, and if Research design is affected by the primary they are to be retained, they work in line with the research audience for the study. For doctoral editorial vision. Reviewers are often shown the students, the primary audience is their advisors comments of other referees so that they can and other members of their research committees. compare their responses. Feedback provides an For faculty, the primary audience is journal and implicit pressure to conform. publishing house editors and grantors. Refereed The upward drift in methodology can be con­ journal editors are the gatekeepers of much of sidered paradoxical. To get published, authors the research that is published, which in turn use sophisticated methodologies. The newer and influences what is taught and who is tenured and more complex the method is, the fewer the peo­ promoted at schools that value research. ple who will be able to evaluate the article, and Recognizing this power, a researcher responds the fewer the practitioners who will be able to to the real or imagined preferences for topic or understand the research and judge whether using method of this audience. The obvious way of the results would be beneficial. In attempting to finding editorial preferences is to read the jour­ gain the approval of other researchers, a researcher nal, see what is being published, and use a similar might not care whether an article advances prac­ approach to one’s own study. Doctoral students tice in the field. Good research can do both; some would be prudent to read dissertations directed publications do neither. by a prospective dissertation advisor to see what these preferences actually are. This situation The Secondary Research Audience begs the question, should these gatekeepers set the research agenda? Editors of research journals It is a desirable state when secondary research usually have been successful researchers in their audiences—other researchers, practitioners, and fields and have published widely. The advisory the public—are more important than the pri­ board that hires an editor increases a journal’s mary ones. From an altruistic perspective, it is prestige by hiring the most prestigious editor it for these audiences that the research is con­ can find. The editor then seeks out other suc­ ducted. Research should be designed to be useful cessful researchers in the field and brings them to the discipline and to advance theoretical or on board. This selection procedure produces a empirical understanding of what is happening in conservative bias: It rewards what has worked some area of education. Does this research pro­ in the past. vide new ideas, new understandings, and new One model for the editorial process is that practices that advance the ways in which profes­ reviewers have had long experience in the field sionals and practitioners in the field can serve the and make prudent judgments about what stud­ public good? An affirmative answer would justify ies will advance educational practice or knowl­ the use of public and philanthropic resources in edge. Another model views editorial decisions as pursuit of educational knowledge. Good research being on show because what editors approve is should benefit everybody. 180 Section III. opportunities and Challenges in Designing and Conducting Inquiry

A measure of the value of research is not just To design research is to make a map, an anal­ acceptability to the editorial process, that is, the ogy of what happens in the world. Research merit indicated by its publication; rather, the design depends on what is being studied and impact of a piece of research on theory or prac­ what the researcher wants to find out. The dou­ tice becomes the ultimate measure of its value ble entendre of “wants to find out” is intentional. or worth. Research that does not meet at least The researcher wants to find out something minimal methodological acceptability is not about, say, how to improve literacy rates in rural published and does not become available to its areas. The researcher also wants to find out that potential audience. Assuming that it does reach a his or her hypothesis is true, for example, that larger audience, does it affect future research in tutoring improves literacy. the field? The choice of the topic focuses the endeavor. A well-designed study should include, at the The choice of method limits what can be discov­ end of the article, recommendations for future ered, emphasizing some possibilities and elimi­ research, but in practice, these recommenda­ nating others. Each choice involves trade-offs. tions tend to focus on narrow methodological Each methodology chosen should, if done well, concerns, such as improving a questionnaire supply some beneficial information. There is and using a different sample. The implicit rec­ one best way in which to find out something ommendation for future researchers is that they extremely simple, such as the mean length of continue to advance the theoretical orientation time it takes students to memorize a list of spell­ of the line of research. A second concluding sec­ ing words. As the question addressed becomes tion should deal with the practical applications broader and more complex, it can be studied of the study. The form of the application is along using a variety of designs. There is no best way these lines: “If your educational world is similar in which to study education; each approach to the one in which this study was conducted, emphasizes some things and is silent on others. here are the things you should do, based on my Political and research ideologies can drive findings, that would improve educational prac­ research or be ignored. tice and understanding in your world.” Research could be designed purely on the Educational research can be influential not basis of curiosity if the researcher wants to know because of its quality but rather because the find­ something. The methodology is likely to be ings confirm what policy makers already believe. emergent as the researcher plays with the topic, This situation is distressing because it means that thinking of it without preconception; delighting an excellent study will affect policy only if policy in possibility; and creating an ongoing dialogue makers want it to affect policy. When two studies with the topic, methods, other researchers in the are excellent but lead to opposite conclusions, field, the persons being studied, and so on. policy makers lose confidence in the research and Research can also be designed around extrin­ return to intuition to set policy. The politics of sic reasons: “How can I make myself famous, educational research seems to be one of its promoted, tenured, or rich on the basis of my salient features (Cooper & Randall, 1999). research?” For that research, the researcher should let money and disciplinary popularity lead the endeavor. For research to affect policy, Conclusion one should follow the money out of governmen­ tal or other granting agencies and heed their The reporting of research can be viewed as sto­ guidelines for topics and methods. Research rytelling, as part of a mythic process of identify­ should be designed to meet their expectations ing who we are. In storytelling, we seek to using methods they prefer. An effective presen­ remember the past, invent the present, and envi­ tation of the results might demand that they be sion the future (Keen & Valley-Fox, 1989). presented in the most simple or most mystifying Research can be viewed as a similar process in forms. remembering the past by examining the litera­ Designing research for altruistic purposes, ture; inventing the present by conducting the to benefit humanity, is more complicated, study and describing the findings; and envision­ because what benefits one group might not ben­ ing the future where this research influences efit another. Any discovery can have wonderful thought, policy, and practice. unanticipated consequences. Basic research has Chapter 11. Intellect, Light, and Shadow in Research Design 181 grand possibilities, but the environment must In the near future, methodological correctness thrive on patience and failure—on trying many will likely maintain its salience. I would expect new things that do not work to find the few that humanistic and aesthetic values will be that do. Such environments are rare. Research neglected in research in the face of issues of designed to solve well-defined problems—applied social justice and pragmatism. Capitalistic ele­ research—can also benefit humanity. Other ments related to the costs of education and the applied research is intended to profit the patent ways in which the education system provides a holder. Research designed to provide an educa­ suitable labor force for the nation’s economy tional environment that will save humanity will likely be emphasized. Whatever work we should get top billing, but who could agree on do or we neglect, our research must refresh what that research would be? our intellect.

References Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Bateson, G. (2000). Steps to an ecology of mind: Thousand Oaks, CA: Sage. Collected essays in anthropology, psychiatry, Creswell, J. W. (2007). Educational research (3rd ed.). evolution, and epistemology. Chicago: University Thousand Oaks, CA: Sage. of Chicago Press. Daniel, L. G. (1996). Kerlinger’s research myths. Baumrind, D. (1964). Some thoughts on ethics of Practical Assessment, Research, & Evaluation, research: After reading Milgram’s behavioral 5(4). Available at http://pareonline.net/getvn study of obedience. American Psychologist, 19, .asp?v=5&n=4 421–423. Erickson, F., & Gutierrez, K. (2002). Culture, rigor, Bean, J. P. (1980). Dropouts and turnover: The and science in educational research. Educational synthesis and test of a causal model of student Researcher, 31(8), 21–24. attrition. Research in Higher Education, 12, Feuer, M. J., Towne, L., & Shavelson, R. J. (2002). 155–187. Scientific culture and educational research. Bean, J. P. (1997, March). How painting can inform Educational Researcher, 31(8), 4–14. qualitative inquiry. Paper presented at the Feyerabend, P. (1993). Against method (3rd ed.). meeting of the American Educational Research New York: Verso. Association, Chicago. Fineman, S. (2000). Emotions in organizations. Berliner, D. (2002). Educational research: The Thousand Oaks, CA: Sage. hardest science of all. Educational Researcher, Gall, M. D., Borg, W., & Gall, J. P. (2002). Educational 31(8), 18–20. research: An introduction (7th ed.). Boston: Biglan, A. (1973). The characteristics of subject Allyn & Bacon. matter in different academic areas. Journal of Gardner, H. (1983). Frames of mind: The theory of Applied Psychology, 57, 195–203. multiple intelligences. New York: Basic Books. Borges, J. L. (1998). Collected fictions (A. Hurley, Geertz, C. (1973). Thick description: Toward an Trans.). New York: Penguin Books. interpretive theory of culture. In C. Geertz, The Bourdieu, P. (1988). Homo academicus (P. Collier, interpretation of cultures (pp. 3–32). New York: Trans.). Stanford, CA: Stanford University Press. Basic Books. (Original work published 1984) Getzels, J. W., & Csikszentmihalyi, M. (1976). The Bourdieu, P., & Passeron, J. C. (1990). Reproduction creative vision: A longitudinal study of problem in education, society, and culture (R. Nice, finding in art. New York: Wiley. Trans.). London: Sage. Harvey, J., & Katz, C. (1985). If I’m so successful, why Bowen, H. R. (1977). Investment in learning. San do I feel like a fake? The imposter phenomenon. Francisco: Jossey-Bass. New York: St. Martin’s. Braithwaite, R. (1955). Scientific explanation. Institute of Education Sciences. (2003). Identifying Cambridge, UK: Cambridge University Press. and implementing educational practices Burkhardt, H., & Schoenfeld, A. H. (2003). supported by rigorous evidence. Washington, DC: Improving educational research: Toward a more U.S. Department of Education. useful, more influential, and better-funded Janis, I. (1972). Victims of groupthink: A psychological enterprise. Educational Researcher, 32(9), 3–14. study of foreign-policy decisions and fiascoes. Cooper, B., & Randall, E. (1999). Accuracy or Boston: Houghton Mifflin. advocacy: The politics of research in education. Jones, J. (1993). Bad blood: The Tuskegee syphilis Thousand Oaks, CA: Corwin. experiment (Rev. ed.). New York: Free Press. 182 Section III. opportunities and Challenges in Designing and Conducting Inquiry

Keen, S., & Valley-Fox, A. (1989). Your mythic Education Research. Washington, DC: National journey. New York: Putnam. Academies Press. Kerlinger, F. (1973). Foundations of behavioral Neill, A., & Ridley, A. (Eds.). (1995). The philosophy research (2nd ed.). New York: Holt, Rinehart & of art. New York: McGraw-Hill. Winston. Olson, D. R. (2004). The triumph of hope over Kezar, A. (2000). Higher education research at the experience in the search for “what works”: A millennium: Still trees without fruit? Review of response to Slavin. Educational Researcher, Higher Education, 23, 443–468. 33(1), 24–26. Krathwohl, D. R. (1988). How to prepare a research Parnes, S. J. (1992). Source book for creative problem proposal: Guidelines for funding and dissertations solving. Buffalo, NY: Creative Foundation Press. in the social and behavioral sciences (3rd ed.). Parsons, T., & Platt, G. M. (1973). The American Syracuse, NY: Syracuse University Press. university. Cambridge, MA: Harvard University Lawrence-Lightfoot, S. (1995). I’ve known rivers: Press. Lives of loss and liberation. New York: Penguin Schwartzman, H. (1978). Transformations: The Books. anthropology of children’s play. New York: Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Plenum. inquiry. Beverly Hills, CA: Sage. Schwartzman, H. (1993). Ethnography in Merton, R. K. (1973). The normative structure of organizations (Qualitative Research Methods science. In N. W. Storer (Ed.), The sociology of Series, No. 27). Newbury Park, CA: Sage. science (pp. 267–278). Chicago: University of Shils, E. (1984). The academic ethic. Chicago: Chicago Press. (Original work published 1942) University of Chicago Press. Milgram, S. (1974). Obedience to authority. Slavin, R. E. (2002). Evidence-based education New York: Harper & Row. policies: Transforming educational practice Mitroff, I. I., & Kilmann, R. H. (1978). and research. Educational Researcher, 31(7), Methodological approaches to social science. San 15–21. Francisco: Jossey-Bass. St.Pierre, E. A. (2002). “Science” rejects Morgan, B. (1997). Images of organization (2nd ed.). postmodernism. Educational Researcher, 31(8), Thousand Oaks, CA: Sage. 25–27. Myers-Briggs, I. (1962). Manual for the Myers-Briggs Thorngate, W. (1976). In general vs. it all depends: Type Indicator. Princeton, NJ: Educational Some comments on the Gergen–Schlenker Testing Service. debate. Academy of Management Journal, 25, National Research Council. (2002). Scientific research 185–192. in education (R. J. Shavelson & L. Towne, Eds.). Weick, K. (1979). The social psychology of organizing Committee on Scientific Principles for (2nd ed.). Reading, MA: Addison-Wesley.