Pragmatic and Dialectic Mixed Method Approaches: An Empirical Comparison A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY Anne E. Betzner IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Frances Lawrenz, Adviser December, 2008 © Anne E. Betzner 2008 i ACKNOWLEDGEMENTS My graduate experience has been immeasurably enriched by the support and encouragement of my adviser, Dr. Frances Lawrenz. Her critical review of my work was an invaluable learning tool that I will carry forward in my career. Thank you for your insight and responsiveness. I would also like to thank committee members Dr. Michael Harwell, Dr. Robert delMas, and Dr. Debra Ingram for their valuable insights and the dedication of their time. A special thank you to Dr. Stuart Yeh for encouraging me to pursue an evaluation of smoke-free ordinances. This study embodies the love and support that I have received to conduct and complete it. First, credit is due to my family, especially my mother, Carole Betzner, whose unwavering value for education ignited in me the desire and courage necessary to pursue a doctoral degree. The completion of this study is her accomplishment as well as mine. The love of my husband can also be found on every page in this volume. He will admit that his romantic attention distracted me from my work, what with all the wooing and the marriage. However, no person has been a greater advocate for me to complete my degree. His respect and selfless concern for my work has been an enduring example of how to be the best spouse ever. This dissertation would not have been possible without the funding of ClearWay Minnesota SM , and the braintrust of the ClearWay team, especially Jessie Saul, Ph.D., Barbara Schillo, Ph.D., Ann Wendling, M.D., Marietta Dreher, and Chris Thokles. Many thanks to ClearWay Minnesota. ii No funding amount in the world could replace the support that Professional Data Analysts, Inc. provided to me in the completion of this degree, especially from Dr. Michael Luxenberg. He opened the door to the opportunity of conducting this study and provided me with the support and latitude to complete it. Special thanks to Julie Rainey for her assistance in thinking through many issues regarding the telephone survey and focus groups. She also deserves credit for hearing more about this study than she ever wanted. I’d also like to especially thank all the staff at Professional Data Analysts, Inc. who assisted with the evaluation for ClearWay Minnesota, including Traci Capesius, Matt Christenson, Marcy Huggins, Annette Kavanaugh, and Laura Stopyro. I could not have completed my degree without the support and friendship of Drs. Jeffrey Harring and Ann Ooms. During our time together as students, they were all you could ever wish for as friends. The example of their scholarship and completed degrees was an inspiration to me long after our paths diverged. An extra special thanks to Dr. Andrew Zieffler who went above and beyond the call of duty as a fellow student and friend to see me through to the completion of my degree. It is with sadness that I remember my father, David Betzner, who completely believed me when I blithely stated that anyone could complete a Ph.D. if they worked hard enough. I have since found the extent to which that statement is both true and false. But the part of me that inherited my dad’s obsessive nature surely helped in the process. I’d also like to thank my grandmother, Mary Prokay, R.N., for always talking to me like an adult. I know she would be proud of my accomplishment. iii ABSTRACT Mixed methods are increasingly used in the fields of evaluation, health sciences, and education in order to meet the diverse information needs of funders and stakeholders. However, a consensus has yet to develop on the theoretical underpinnings of the methodology. A side-by-side assessment of two competing theoretical approaches to mixed methods, the dialectic and pragmatic, can assist researchers to optimize their use of mixed methods methodology and contribute to the growth of mixed methods theory. This study empirically compares the dialectic and pragmatic approaches to mixed methods and probes key issues underlying the methodology, including unique yield from mixed method studies, the importance of paradigmatic divergence between methods, and the financial demands of mixed method studies. A secondary analysis of a real-world evaluation, this study explores five research questions regarding the convergence, divergence and uniqueness of single method findings; the extent to which mixed methods produce unique findings over and above single methods presented side- by-side; the extent to which studies meet key criteria for validity; stakeholders’ perceptions of the utility and credibility of the studies; and the cost of single methods. The pragmatic mixed method study was developed by integrating a post- positivistic telephone survey with weakly interpretive focus groups at the point of interpretation using pragmatic criteria. The dialectic study mixed the same post- positivistic telephone survey with strongly interpretive phenomenological interviews iv using a Hegelian-inspired dialectic format. All three single methods were examined by a method expert in the field who affirmed the methodologies used. Findings suggest that both mixed method approaches produced unique conclusions that would not have been available by presenting single methods side-by- side. However, the dialectic method produced more complex convergence and more divergence, leading it to be more generative than the pragmatic method. The use of stronger as compared to weaker interpretive methods contributed to the generative quality of the dialectic approach. Overall, the dialectic method appears more suitable to exploring more complex phenomenon as compared to the pragmatic approach. However, these conclusions are drawn from one study of one real-world evaluation. Much more scholarship is needed to explore the issues raised here. v TABLE OF CONTENTS CHAPTER 1 INTRODUCTION ............................................................................ 1 1.1. Statement of the Problem .................................................................... 1 1.1.1. The Need for Research on Mixed Methods ........................ 1 1.1.2. Two Competing Theories of Mixed Methods..................... 4 1.1.3. Conclusions ......................................................................... 6 1.2. Purpose of the Study ........................................................................... 7 1.3. Conceptual Framework ....................................................................... 8 1.4. Research Questions ............................................................................. 8 1.5. Research Design................................................................................ 12 1.6. Overview of the Dissertation............................................................. 12 CHAPTER 2 LITERATURE REVIEW ............................................................... 14 2.1. Paradigms.......................................................................................... 14 2.1.1. Defining Three Key Paradigms along a Continuum......................................................................... 15 2.1.2. Defining Terms ................................................................. 18 2.1.3. The Relationship between Methods and Paradigms .......................................................................... 21 2.2. An Historical Review of Mixed Methods ......................................... 27 2.2.1. Mixed Methods from a Post-Positivist Perspective ........................................................................ 28 2.2.2. The Growth of Qualitative Methods and the Emergence of Mixed Methods .......................................... 29 2.2.3. Triangulation and Role of Qualitative Methods in Mixed Methods ................................................................. 30 2.2.4. The Development of the Dialectic Approach to Mixed Methods ................................................................. 33 2.2.5. An Emerging Pragmatic Approach to Mixed Method Studies.................................................................. 41 vi 2.2.6. Dialectic and Pragmatic Approaches as Generative Endeavors ....................................................... 42 2.3. Applying Pragmatism to Mixed Methods ......................................... 43 2.4. Applying Dialectics to Mixed Methods ............................................ 48 2.5 Validity............................................................................................... 56 2.5.1. Trustworthiness: A Unified Framework ........................... 57 2.5.2. The Standards for Program Evaluation ............................. 71 CHAPTER THREE RESEARCH METHODS..................................................... 74 3.1 Overview of Methodology ................................................................. 74 3.2 Timeline for Secondary Data ............................................................. 76 3.3 Research Approach and Rationale ..................................................... 78 3.4. Research Questions ........................................................................... 79 3.4.1. Research Question 1: Comparison of Single Method Results to One Another........................................ 80 3.4.2. Research Question 2: Comparison of Mixed Method Results to Each Other and Single Methods............................................................................