QUALITATIVE METHODS, TRANSPARENCY, AND QUALITATIVE DATA ANALYSIS SOFTWARE: TOWARD AN UNDERSTANDING OF TRANSPARENCY IN MOTION by KRISTI JACKSON B.A., Kenyon College, 1987 M.Ed., University of Vermont, 1990 A dissertation submitted to the School of Education, University of Colorado Dissertation committee Margaret Eisenhart, Chair Ken Howe Susan Jurow Ben Kirshner Stefanie Mollborn (Sociology), outside member Dissertation defense March 3, 2014, 1:30 P.M. University of Colorado, Boulder Education room 142 (Dean’s Conference Room) This dissertation entitled: Qualitative Methods, Transparency, and Qualitative Data Analysis Software: Toward an Understanding of Transparency in Motion written by Kristi Jackson has been approved for the School of Education _____________________________________________ Margaret Eisenhart _____________________________________________ Ken Howe _____________________________________________ Susan Jurow _____________________________________________ Ben Kirshner _____________________________________________ Stefanie Mollborn Date: March 3, 2014 The final copy of this thesis has been examined by the signatories, and we Find that both the content and the form meet acceptable presentation standards Of scholarly work in the above mentioned discipline. IRB protocol # 13-0326 iii ABSTRACT Jackson, Kristi (Ph.D., Education) Qualitative Methods, Transparency, and Qualitative Data Analysis Software: Toward an Understanding of Transparency in Motion Dissertation directed by Professor Margaret Eisenhart This study used in-depth, individual interviews to engage seven doctoral students and a paired member of their dissertation committee in discussions about qualitative research transparency and the use of NVivo, a Qualitative Data Analysis Software (QDAS), in pursuing it. The study also used artifacts (an exemplary qualitative research article of the participant’s choice and the student’s written dissertation) to examine specific researcher practices within particular contexts. The design and analysis were based on weak social constructionist (Schwandt, 2007), boundary object (Star, 1989; Star & Griesemer, 1989) and boundary-work (Gieryn, 1983, 1999) perspectives to facilitate a focus on: 1) The way transparency was used to coordinate activity in the absence of consensus. 2) The discursive strategies participants employed to describe various camps (e.g., qualitative and quantitative researchers) and to simultaneously stake claims to their understanding of transparency. The analysis produced four key findings. First, the personal experiences of handling their qualitative data during analysis influenced the students’ pursuit of transparency, long before any consideration of being transparent in the presentation of findings. Next, the students faced unpredictable issues when pursuing transparency, to which they responded in situ , considering a wide range of contextual factors. This was true even when informed by ideal types (Star & iv Griesemer, 1989) such as the American Educational Research Association (2006) guidelines that provided a framework for pursuing the principle of transparency. Thirdly, the QDAS-enabled visualizations students used while working with NVivo to interpret the data were described as a helpful (and sometimes indispensable) aspect of pursuing transparency. Finally, this situational use of visualizations to pursue transparency was positioned to re-examine, verify, and sometimes challenge their interpretations of their data over time as a form of self- interrogation, with less emphasis on showing their results to an audience. Together, these findings lead to a new conceptualization of transparency in motion , a process of tacking back and forth between situated practice of transparency and transparency as an ideal type. The findings also conclude with several proposals for advancing a transparency pedagogy . These proposals are provided to help qualitative researchers move beyond the often implicit, static, and post-hoc invocations of transparency in their work. DEDICATION In loving memory of my dear friend, Kenneth Lee Goodwin (August 17, 1948 – August 17, 2013). My life and work are better because of his curiosity and support. vi ACKNOWLEDGMENTS This research benefitted tremendously from the sound guidance and sharp intellect of Margaret Eisenhart, my advisor and chair. I thank her and the rest of my committee members – Kenneth Howe, Susan Jurow, Ben Kirshner and Stefanie Mollborn – for bolstering the research design and engaging in an interesting discussion of the merits and limitations of the findings. During my pursuit of this doctorate, friends and family tolerated lengthy gaps in communication and a general lack of attention. Despite this, they consistently and lovingly supported me; I am aware of my good fortune in this regard. In particular, I am thankful for the consistently well- timed encouragement from Leslye Steptoe. Finally, I thank the many QDAS pioneers, critics, and proponents for more than twenty-five years of lively experimentation with – and discussion about – the role of software in qualitative analysis. In particular, I owe a great deal to Lyn Richards, Tom Richards, and Pat Bazeley for their long-lasting contributions to my work with and understanding of QDAS. vii TABLE OF CONTENTS TABLE OF CONTENTS vii LIST OF TABLES ix LIST OF FIGURES x CHAPTER ONE: PURPOSE 1 Push to Pursue Transparency 1 A Working Understanding of Transparency 3 The History of QDAS 4 Common Enterprises Among QDAS and Methods Experts 9 Separate Sub-Communities 10 QDAS Iterativity and Flexibility 12 The Problem of Separate Communities 20 CHAPTER TWO: LITERATURE REVIEW 25 Searching for and Categorizing Materials 25 Using NVivo to Examine the Literature 26 Literature Review Findings 29 Research Questions 53 CHAPTER THREE: METHODS 55 Positions 55 Theories 58 An Additional Research Question 63 Data Collection 63 Analysis 65 Challenges 68 CHAPTER FOUR: FINDINGS AND CONCLUSIONS 72 Overview 72 The Ethics of Transparency 73 Exemplary Qualitative Research 79 Transparency and NVivo 83 Interview Discussions about NVivo 83 Transparency as an Ideal Type 98 Transparency as a Boundary Object 99 Transparency and Boundary-work 104 Research Answers 108 CHAPTER FIVE: IMPLICATIONS 113 Parameters 113 Key Findings 116 Relevance 131 Future Research 139 REFERENCES 141 APPENDIX A: LITERATURE SEARCH LOG 152 APPENDIX B: ANNOTATED TRANSPARENCY BIBLIOGRAPHY 158 viii APPENDIX C: MEYRICK’S LITERATURE AND EXPERT PANEL 175 APPENDIX D: RECRUITMENT AND SCREENING 177 APPENDIX E: CONSENT 179 APPENDIX F: INTERVIEW QUESTIONS 187 APPENDIX G: DEBRIEF QUESTION 194 APPENDIX H: TRANSCRIPTION DECISIONS 197 APPENDIX I: DESCRIPTION OF ARTICLES 201 APPENDIX J: DESCRIPTION OF STUDENTS 203 APPENDIX K: STOP WORDS 205 ix LIST OF TABLES Table 2.1 37 Unlabeled table from Wenger Table 4.1 87 Source and reference counts for the Logical Connections node Table A.1 153-154 Search parameters in literature databases for items related to qualitative research transparency x LIST OF FIGURES Figure 1.1 14 Construction of a relationship between two items in NVivo Figure 1.2 14 Visualization of a relationship in the modeler within NVivo Figure 1.3 15 Visualization of multiple relationships in the modeler within NVivo Figure 1.4 16 Coding a specific passage into a researcher-identified relationship in NVivo Figure 1.5 17 Opening an item in the NVivo model Figure 1.6 17 Examining the content in a model item in NVivo Figure 1.7 18 Jumping from a decontextualized quote to the source in NVivo Figure 2.1 45 Meyrick’s quality framework for qualitative research Figure 4.1 88 Word cloud of the eight most frequently used words from Transparency node Figure 4.2 89 Data types collected from each pair Figure 4.3 90 Coding stripes and highlighting in the Transparency node Figure 4.4 91 A memo in the database with a See Also Link Figure 4.5 92 A section of the transcript connected to the memo in Figure 4.4 Figure 5.1 119 Transparency in motion Figure 5.2 137 A memo in the database with a See Also Link Figure 5.3 138 An MSWord document exported from NVivo 1 CHAPTER ONE: PURPOSE Push to Pursue Transparency Many scholars invoke the principle of transparency in discussions of education research standards. According to this principle, aspects of the data and research process must be made visible to allow scholars to assess the merits of education research findings and recommendations. In Scientific Research in Education (Shavelson & Towne, 2002), expert panels are given a directive to “adhere to scientific principles of transparency of method, assessing uncertainty, and subjecting findings to the skeptical eye of the broader scientific community” (p. 155). Similarly, in Standards for Reporting on Empirical Social Science Research in AERA Publications, the preamble states that the standards are “intended to promote empirical research reporting that is warranted and transparent” (American Educational Research Association, 2006, p. 33). However, when scholars mention “transparency,” they rarely define the term. This is also true when narrowing the focus to discussions about transparency in qualitative research methods texts. In the section on “transparency” in the Sage Encyclopedia of Qualitative Research Methods (2008), Hiles states that the “need for transparency has become most urgent” even though it is “often simply taken for granted” (p. 891). According to Hiles, researcher reflexivity and the dissemination of results to a target audience are the two focal points for transparency. Researcher