Determining Validity in Qualitative Inquiry

THEORY INTO PRACTICE / Summer 2000 Getting Good Qualitative Data

John W. Creswell Dana L. Miller Determining Validity in Qualitative Inquiry

RITING ABOUT VALIDITY IN QUALITATIVE in- qualitative projects (e.g., Lincoln & Guba, 1985; W quiry is challenging on many levels. Mul- Maxwell, 1996; Merriam, 1998). Qualitative research- tiple perspectives about it flood the pages of books ers routinely employ member checking, triangulation, (e.g., Lincoln & Guba, 1985; Maxwell, 1996; Mer- thick description, peer reviews, and external audits. riam, 1998; Schwandt, 1997) and articles and chap- Researchers engage in one or more of these proce- ters (e.g., Altheide & Johnson, 1994; Lather, 1993; dures and report results in their investigations. Maxwell, 1992). In these texts, readers are treated to As helpful as they are, these discussions about a confusing array of terms for validity, including au- validity procedures provide little guidance as to thenticity, goodness, verisimilitude, adequacy, trust- why one procedure might be selected for use by worthiness, plausibility, validity, validation, and researchers over other procedures. In this article, credibility. Various authors have constructed diverse we suggest that the choice of validity procedures typologies of validity (e.g., Maxwell’s five types, is governed by two perspectives: the lens research- 1992; Lather’s four frames, 1993; and Schwandt’s ers choose to validate their studies and researchers’ four positions, 1997). It is little wonder that Don- paradigm assumptions. We advance a two-dimen- moyer (1996), who wrote an editorial on validity sional framework that can help researchers identi- in the Educational Researcher, commented on the fy appropriate validity procedures for their studies. diverse perspectives of validity by contrasting Miles The use of this framework can provide a ra-

Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 and Huberman’s (1994) “traditional conception of tionale for choice of a procedure beyond what the validity” with Lather’s (1993) “ironic validity” (p. setting and participants will bear and what col- 21). Novice researchers, in particular, can become leagues and faculty advisers recommend. The increasingly perplexed in attempting to understand framework helps researchers select procedures the notion of validity in qualitative inquiry. based on who assesses the credibility of a study and There is a general consensus, however, that their own philosophical positions toward qualitative qualitative inquirers need to demonstrate that their inquiry. We begin by discussing the two perspectives studies are credible. To this end, several authors iden- of the framework and then identify nine validity pro- tify common procedures for establishing validity in cedures that fit the framework. We end by describing how the lens and paradigm assumptions help guide John W. Creswell is professor of educational psychology our choice of validity procedures. at the University of Nebraska-Lincoln; Dana L. Miller In this discussion we define validity as how is assistant professor of research methods at Doane accurately the account represents participants’ re- College, Lincoln, Nebraska. alities of the social phenomena and is credible to

them (Schwandt, 1997). Procedures for validity the interpretations accurately represent them. A include those strategies used by researchers to es- third lens may be the credibility of an account by tablish the credibility of their study. Throughout individuals external to the study. Reviewers not this discussion, we make the assumption that va- affiliated with the project may help establish validity refers not to the data but to the inferences lidity as well as various readers for whom the ac- drawn from them (Hammersley & Atkinson, 1983). count is written.

The Lens Used by the Researcher Paradigm Assumptions When we refer to the lens, we mean that the The lens researchers use—their own, study inquirer uses a viewpoint for establishing validity participants, or individuals external to the project— in a study. Qualitative inquirers bring to their stud- is not the only perspective that governs the choice ies a different lens toward validity than that brought of validity procedures. Researchers’ paradigm as- to traditional, quantitative studies. sumptions or worldviews (Guba & Lincoln, 1994) In quantitative research, investigators are also shape their selection of procedures. As sug- most concerned about the specific inferences made gested by Ratcliffe (1983), from test scores on psychometric instruments (i.e., Quite different notions of what constitutes validity the construct, criterion, and content validity of inter- have enjoyed the status of dominant paradigm at dif- pretations of scores) (AERA, APA, & NCME, 1982) ferent times, in different historical contexts, and un- and the internal and external validity of experimental der different prevailing modes of thought and epistemology. (p. 158) and quasi-experimental designs (Campbell & Stan- ley, 1966). In contrast, qualitative researchers use a Three paradigm assumptions, labeled by Guba lens not based on scores, instruments, or research and Lincoln (1994) as postpostivist, constructivist, designs but a lens established using the views of peo- and critical influence researchers’ choice of valid- ple who conduct, participate in, or read and review a ity procedures. These assumptions have been asso- study. ciated with different historical moments in the For example, one lens to determine the cred- evolution of qualitative inquiry (Denzin & Lincoln, ibility of a study is the particular lens of the re- 1994). A brief overview of these paradigm assump- searcher. Researchers determine how long to remain tions is advanced here. in the field, whether the data are saturated to es- The postpostivist researcher assumes that qual- tablish good themes or categories, and how the itative research consists of rigorous methods and sys- analysis of the data evolves into a persuasive nar- tematic forms of inquiry. Identified by Denzin and rative. Patton (1980) describes this process as one Lincoln as the “modernist” phase of qualitative in- where qualitative analysts return to their data “over quiry (1994, p. 8), this philosophical perspective and over again to see if the constructs, categories, emerged in social science research during the 1970s explanations, and interpretations make sense” (p. and continues today. Individuals embracing the

Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 339). Altheide and Johnson (1994) refer to it as postpostivist position both recognize and support “validity-as-reflexive-accounting” (p. 489) where re- validity, look for quantitative equivalence of it, searchers, the topic, and the sense-making process and actively employ procedures for establishing interact. validity using specific protocols. Maxwell (1996), Qualitative inquirers may use a second lens in Qualitative Research Design: An Interactive to establish the validity of their account: the par- Approach, for example, exemplifies postpostivist ticipants in the study. The qualitative paradigm assumptions toward qualitative validity. assumes that reality is socially constructed and it The constructivist or interpretive position is what participants perceive it to be. This lens emerged during the period of 1970 to 1987 (Denzin suggests the importance of checking how accurately & Lincoln, 1994), and it is reflected in stances to- participants’ realities have been represented in the ward validity today. Constructivists believe in plu- final account. Those who employ this lens seek to ralistic, interpretive, open-ended, and contextualized actively involve participants in assessing whether (e.g., sensitive to place and situation) perspectives

125 THEORY INTO PRACTICE / Summer 2000 Getting Good Qualitative Data

toward reality. The validity procedures reflected 522). To this end, researchers engage in validity pro- in this thinking present criteria with labels distinct cedures of self-disclosure and collaboration with from quantitative approaches, such as trustworthi- participants in a study. These procedures help to ness (i.e., credibility, transferability, dependability, minimize further the inequality that participants and confirmability), and authenticity (i.e., fairness, often feel. For example, Carspecken’s Critical Eth- enlarges personal constructions, leads to improved nography in Educational Research (1996) reports understanding of constructions of others, stimulates validity procedures for tracking bias and interviews action, and empowers action). The classical work with oneself as ways for researchers to be situated by Lincoln and Guba, Naturalistic Inquiry (1985), in a study. provides extensive discussions about these forms of trustworthiness and authenticity. Validity Within Lens and Paradigms A third paradigm assumption is the critical As shown in Table 1, we use the lens and perspective. This perspective emerged during the paradigm assumptions to create a two-dimensional 1980s as the “crisis in representation” (Denzin & framework for locating nine different types of va- Lincoln, 1994, p. 9). As a challenge and critique lidity procedures. The discussion now turns to these of the modern state, the critical perspective holds nine procedures with a brief definition of each, that researchers should uncover the hidden assump- their location within a lens and paradigm perspec- tions about how narrative accounts are constructed, tive, and approaches for implementing each proce- read, and interpreted. What governs our perspec- dure. This list is not exhaustive but includes those tive about narratives is our historical situatedness procedures commonly used and cited in qualitative of inquiry, a situatedness based on social, politi- literature. cal, cultural, economic, ethnic, and gender ante- cedents of the studied situations. The implication Triangulation for validity of this perspective is that validity is Triangulation is a validity procedure where called into question, its assumptions interrogated researchers search for convergence among multi- and challenged, and the researchers need to be re- ple and different sources of information to form flexive and disclose what they bring to a narrative. themes or categories in a study. The term comes Richardson (1994) uses the metaphor of a from military navigation at sea where sailors trian- crystal as an image for validity: “Crystals are prisms gulated among different distant points to determine that reflect externalities and refract within themselves. their ship’s bearing (Jick, 1979). Denzin (1978) . . . What we see depends on our angle of repose” (p. identified four types of triangulation: across data

Table 1 Validity Procedures Within Qualitative Lens and Paradigm Assumptions

Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 Paradigm assump- Postpositivist or Constructivist tion/Lens Systematic Paradigm Paradigm Critical Paradigm

Lens of the Triangulation Disconfirming Researcher Researcher evidence reflexivity

Lens of Study Member checking Prolonged engage- Collaboration Participants ment in the field

Lens of People Ex- The audit trail Thick, rich Peer debriefing ternal to the Study description (Reviewers, Readers)

126 Creswell and Miller Determining Validity

sources (i.e., participants), theories, methods (i.e., ing beliefs and biases early in the research process interview, observations, documents), and among to allow readers to understand their positions, and different investigators. then to bracket or suspend those researcher biases As a validity procedure, triangulation is a step as the study proceeds. This validity procedure uses taken by researchers employing only the research- the lens of the researcher but is clearly positioned er’s lens, and it is a systematic process of sorting within the critical paradigm where individuals re- through the data to find common themes or cate- flect on the social, cultural, and historical forces gories by eliminating overlapping areas. A popular that shape their interpretation. practice is for qualitative inquirers to provide cor- Researchers might use several options for in- roborating evidence collected through multiple corporating this reflexivity into a narrative account. methods, such as observations, interviews, and doc- They may create a separate section on the “role of uments to locate major and minor themes. The nar- the researcher,” provide an epilogue, use interpre- rative account is valid because researchers go tive commentary throughout the discussion of the through this process and rely on multiple forms of findings, or bracket themselves out by describing evidence rather than a single incident or data point personal experiences as used in phenomenological in the study. methods (Moustakas, 1994).

Disconfirming evidence Member checking A procedure closely related to triangulation With member checking, the validity proce- is the search by researchers for disconfirming or dure shifts from the researchers to participants in negative evidence (Miles & Huberman, 1994). It the study. Lincoln and Guba (1985) describe mem- is the process where investigators first establish ber checks as “the most crucial technique for es- the preliminary themes or categories in a study tablishing credibility” (p. 314) in a study. It consists and then search through the data for evidence that of taking data and interpretations back to the par- is consistent with or disconfirms these themes. In ticipants in the study so that they can confirm the this process, researchers rely on their own lens, credibility of the information and narrative account. and this represents a constructivist approach in that With the lens focused on participants, the research- it is less systematic than other procedures and re- ers systematically check the data and the narrative lies on examining all of the multiple perspectives account. on a theme or category. Several procedures facilitate this process. A In practice, the search for disconfirming evi- popular strategy is to convene a focus group of dence is a difficult process because researchers have participants to review the findings. Alternatively, the proclivity to find confirming rather than dis- researchers may have participants view the raw data confirming evidence. Further, the disconfirming (e.g., transcriptions or observational field notes) evidence should not outweigh the confirming evi- and comment on their accuracy. Throughout this process, the researchers ask participants if the Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 dence. As evidence for the validity of a narrative account, however, this search for disconfirming themes or categories make sense, whether they are evidence provides further support of the account’s developed with sufficient evidence, and whether credibility because reality, according to construc- the overall account is realistic and accurate. In turn, tivists, is multiple and complex. researchers incorporate participants’ comments into the final narrative. In this way, the participants Researcher reflexivity add credibility to the qualitative study by having a A third validity procedure is for researchers to chance to react to both the data and the final nar- self-disclose their assumptions, beliefs, and biases. rative. This is the process whereby researchers report on personal beliefs, values, and biases that may shape Prolonged engagement in the field their inquiry. It is particularly important for re- Another validity procedure is for researchers searchers to acknowledge and describe their enter- to stay at the research site for a prolonged period

127 THEORY INTO PRACTICE / Summer 2000 Getting Good Qualitative Data

of time. Fetterman (1989) contends that “working turning to individuals external to the project, such with people day in and day out for long periods of as auditors—formally brought into the study—or time is what gives ethnographic research its valid- readers who examine the narrative account and at- ity and vitality” (p. 46). During repeated observa- test to its credibility. In establishing an audit trail, tion, the researchers build trust with participants, researchers provide clear documentation of all re- find gatekeepers to allow access to people and sites, search decisions and activities. They may provide establish rapport so that participants are comfort- evidence of the audit trail throughout the account or able disclosing information, and reciprocate by giv- in the appendices. Researchers may also use an ex- ing back to people being studied. This lens is ternal auditor to review their study. The goal of a focused on gaining a credible account by building formal audit is to examine both the process and prod- a tight and holistic case. uct of the inquiry, and determine the trustworthi- Being in the field over time solidifies evi- ness of the findings. dence because researchers can check out the data Lincoln and Guba (1985) use the analogy of and their hunches and compare interview data with a fiscal audit to describe this process. The audit is observational data. It is not a process that is sys- often used in formal studies, such as in disserta- tematically established, but constructivists recog- tions, particularly when committee members are nize that the longer they stay in the field, the more trained quantitatively and may be skeptical about the pluralistic perspectives will be heard from par- qualitative studies. Certain audiences appreciate the ticipants and the better the understanding of the rigor of the audit process, and the lens for estab- context of participant views. In practice, prolonged lishing credibility becomes someone external to the engagement in the field has no set duration, but project. It is a systematic procedure in that the ethnographers, for example, spend from 4 months reviewer writes an analysis after carefully study- to a year at a site. ing the documentation provided by the researcher. An audit trail is established by researchers Collaboration documenting the inquiry process through journal- Credible data also come from close collabo- ing and memoing, keeping a research log of all ration with participants throughout the process of activities, developing a data collection chronolo- research. Collaboration means that the participants gy, and recording data analysis procedures clearly. are involved in the study as co-researchers or in The external auditor examines this documentation less formal arrangements. This validity lens is one with the following questions in mind: Are the find- of building the participant’s view into the study. It ings grounded in the data? Are inferences logical? belongs to a critical paradigm perspective because Is the category structure appropriate? Can inquiry the intent of the process is to respect and support decisions and methodological shifts be justified? participants in a study, not further marginalize What is the degree of researcher bias? What strate- them. gies were used for increasing credibility? (Schwandt In practice, collaboration may assume multi- Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 & Halpern, 1988). Through this process of docu- ple forms. For example, participants may help form menting a study and a review of the documenta- the research questions, assist with data collection tion by an external auditor, the narrative account and analysis, and be involved in writing the narra- becomes credible. tive account. Some qualitative researchers may share the profits, such as book royalties or co-au- Thick, rich description thorship publication rights. By actively involving Another procedure for establishing credibility participants in their studies, qualitative inquirers in a study is to describe the setting, the participants, add further credibility to their narrative accounts. and the themes of a qualitative study in rich detail. According to Denzin (1989), “thick descriptions are The audit trail deep, dense, detailed accounts. . . . Thin descriptions, Now the lens for establishing validity shifts by contrast, lack detail, and simply report facts” (p. again. The credibility of a study is established by 83). The purpose of a thick description is that it

128 Creswell and Miller Determining Validity

creates verisimilitude, statements that produce for herent in the study design, such as triangulation of the readers the feeling that they have experienced, methods, prolonged observations in the field, and or could experience, the events being described in the use of thick, rich descriptions. In deciding to a study. Thus, credibility is established through use a formal audit or peer debriefer, researchers the lens of readers who read a narrative account should consider their audiences, the availability of and are transported into a setting or situation. such individuals, and the expense of using them. To use this procedure for establishing credi- Member checking is always important as well as bility, researchers employ a constructivist perspec- keeping research logs to document the rigor of our tive to contextualize the people or sites studied. research processes. When faced with students or The process of writing using thick description is faculty committees that seek rigor and a systemat- to provide as much detail as possible. It may in- ic review of procedures, the process of establish- volve describing a small slice of interaction, expe- ing a clear audit trail is most important. rience, or action; locating individuals in specific As we review the nine validity procedures, situations; bringing a relationship or an interaction we acknowledge the importance of all three lenses alive between two or more persons; or providing a and that their emphasis in a study will vary de- detailed rendering of how people feel (Denzin, pending on the project, the audience for whom we 1989). are writing, and the people available to provide an With this vivid detail, the researchers help assessment of our project. Our primary lens, how- readers understand that the account is credible. Rich ever, is always that of the participants in a study, description also enables readers to make decisions and we have become more reflexive in our studies, about the applicability of the findings to other set- acknowledging the inseparableness of the research- tings or similar contexts. er and the process of inquiry. As for our paradigm stances, we most close- Peer debriefing ly align ourselves with the use of systematic pro- A peer review or debriefing is the review of cedures, employing rigorous standards and clearly the data and research process by someone who is identified procedures (e.g., Creswell, 1998). How- familiar with the research or the phenomenon be- ever, we also resonate with the critical perspective ing explored. A peer reviewer provides support, and engage in collaborative research practices that plays devil’s advocate, challenges the researchers’ are respectful of the individuals we study (e.g., assumptions, pushes the researchers to the next step Miller, Creswell, & Olander, 1998). What is most methodologically, and asks hard questions about important is that the credibility of the account be methods and interpretations (Lincoln & Guba, conveyed in a qualitative study. We suggest that 1985). the use of validity procedures requires thinking The lens for establishing credibility is some- beyond specific procedures—to acknowledge the one external to the study, and a critical paradigm lens being employed in a study and the paradigm is operating because of the close collaboration be- assumptions of the researchers. Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009 tween the external reviewer and the qualitative researcher. This procedure is best used over time References during the process of an entire study. Peer debrief- Altheide, D.L., & Johnson, J.M. (1994). Criteria for ers can provide written feedback to researchers or assessing interpretive validity in qualitative re- simply serve as a sounding board for ideas. By search. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 485-499). seeking the assistance of peer debriefers, research- Thousand Oaks, CA: Sage. ers add credibility to a study. American Educational Research Association, American Psychological Association, & National Council on Positioning Ourselves Measurement in Education. (1982). Standards for educational and psychological testing. Washington, Our approach is to use several validity pro- DC: American Educational Research Association. cedures in our studies. Certainly some strategies Campbell, D.T., & Stanley, J.C. (1966). Experimental are easier to use than others, particularly those in- and quasi-experimental designs for research. In

129 THEORY INTO PRACTICE / Summer 2000 Getting Good Qualitative Data

N.L. Gage, Handbook of research on teaching (pp. Lather, P. (1993). Fertile obsession: Validity after poststruc- 1-76). Chicago, IL: Rand-McNally. turalism. The Sociological Quarterly, 34, 673-693. Carspecken, P.F. (1996). Critical ethnography in edu- Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. cational research: A theoretical and practical Newbury Park, CA: Sage. guide. New York: Routledge. Maxwell, J.A. (1992). Understanding and validity in qual- Creswell, J.W. (1998). Qualitative inquiry and research itative research. Harvard Educational Review, 62, design: Choosing among five traditions. Thousand 279-300. Oaks, CA: Sage. Maxwell, J.A. (1996). Qualitative research design: An Denzin, N.K. (1978). The research act: A theoretical interactive approach. Thousand Oaks, CA: Sage. orientation to sociological methods (2nd ed.). New Merriam, S.B. (1998). Qualitative research and case study York: McGraw-Hill. applications in education. San Francisco: Jossey-Bass. Miles, M.B., & Huberman, A.M. (1994). Qualitative data Denzin, N.K. (1989). Interpretive interactionism. New- nd bury Park, CA: Sage. analysis: An expanded sourcebook (2 ed.). New- Denzin, N.K., & Lincoln, Y.S. (1994). Introduction: En- bury Park, CA: Sage. tering the field of qualitative research. In N.K. Den- Miller, D.L., Creswell, J. W., & Olander, L.S. (1998). Writing and retelling multiple ethnographic tales of zin & Y.S. Lincoln (Eds.), Handbook of qualitative a soup kitchen for the homeless. Qualitative Inquiry, research (pp. 1-17). Thousand Oaks, CA: Sage. 4, 469-491. Donmoyer, R. (1996). Educational research in an era Moustakas, C. (1994). Phenomenological research meth- of paradigm proliferation: What’s a journal editor ods. Thousand Oaks, CA: Sage. to do? Educational Researcher, 25(2), 19-25. Patton, M.Q. (1980). Qualitative evaluation methods. Fetterman, D.M. (1989). Ethnography: Step by step Newbury Park, CA: Sage. (Applied Social Research Methods Series, No. 17). Ratcliffe, J.W. (1983). Notions of validity in qualitative Newbury Park, CA: Sage. research methodology. Knowledge: Creation, Diffu- Guba, E.G., & Lincoln, Y.S. (1994). Competing para- sion, Utilization, 5(2), 147-167. digms in qualitative research. In N.K. Denzin & Richardson, L. (1994). Writing: A method of inquiry. In Y.S. Lincoln (Eds.), Handbook of qualitative re- N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qual- search (pp. 105-117). Thousand Oaks, CA: Sage. itative research (pp. 516-529). Thousand Oaks, CA: Hammersley, M., & Atkinson, P. (1983). Ethnogra- Sage. phy: Principles in practice (2nd ed.). New York: Schwandt, T.A. (1997). Qualitative inquiry: A dictionary Routledge. of terms. Thousand Oaks, CA: Sage. Jick, T.D. (1979). Mixing qualitative and quantitative Schwandt, T.A., & Halpern, E.S. (1988). Linking audit- methods: Triangulation in action. Administrative Sci- ing and metaevaluation: Enhancing quality in ap- ence Quarterly, 24, 602-611. plied research. Newbury Park, CA: Sage. Downloaded By: [Canadian Research Knowledge Network] At: 11:25 15 February 2009

130