Reporting Credibility in Educational Evaluation Studies That Use Qualitative Methods: A
Total Page:16
File Type:pdf, Size:1020Kb
Reporting Credibility in Educational Evaluation Studies that Use Qualitative Methods: A Mixed Methods Research Synthesis A dissertation presented to the faculty of The Patton College of Education of Ohio University In partial fulfillment of the requirements for the degree Doctor of Philosophy Hongjing Liao May 2015 ©2015 Hongjing Liao. All Rights Reserved. 2 This dissertation titled Reporting Credibility in Educational Evaluation Studies that Use Qualitative Methods: A Mixed Methods Research Synthesis by HONGJING LIAO has been approved for the Department of Educational Studies and The Patton College of Education by Krisanna Machtmes Associate Professor of Educational Studies Renée A. Middleton Dean, The Patton College of Education 3 Abstract LIAO, HONGJING, Ph.D., May 2015, Educational Research and Evaluation, Social Studies Education Reporting Credibility in Educational Evaluation Studies That Use Qualitative Methods: A Mixed Methods Research Synthesis Director of Dissertation: Krisanna Machtmes Qualitative methods have increasingly been applied in program evaluations of policies and interventions (Patton, 2003; Sandelowski & Barroso, 2007), and establishing the quality of evaluation findings is of essential importance. Yet little is known about how qualitative quality criteria of evaluation reports have been applied in practice (Brandon & Singh, 2009; Gephart, 2004). One of the ways to assess such practice can be to examine the reporting of credibility techniques in published evaluation reports that use qualitative methods. This study examined the practice of establishing qualitative credibility in the context of program evaluation by carrying out a research synthesis of the related studies published in six leading evaluation journals (from 2003 to 2012). Mixed methods are used to identify key credibility techniques, document the frequency of the techniques, and to describe their use and properties. There were 118 articles found eligible for analysis. The finding suggests that the reporting of credibility techniques has been relatively steady over the past decade. The majority of articles addressed basic methodological quality of the study, but most authors are not sensitive in addressing credibility of their qualitative findings with credibility techniques. 4 Dedication To my loving parents, Wei, and Spring: thank you for being there for me. To my five-years-ago self: good choice, and you made it. 5 Acknowledgments I would like to thank my dissertation committee members who were more than generous with their expertise and precious time. Special thanks to Dr. John Hitchcock, for being such a great advisor, mentor, and dear friend. I would like to thank him for his countless hours of reading, editing, and discussing with me in person and on-line throughout the entire process. His guidance and encouragement have made this a pleasant and rewarding journey. I would like to extend my sincere appreciation to Dr. Krisanna Machtmes, my Chair, for taking me on and providing all support for me to move forward; to Dr. Gordon Brooks for his detailed feedback, kind advice, and patient explanations on statistics and everything else; to Dr. Adah Ward Randolph for listening, caring, and sharing, and setting a great example of a strong mind; to Dr. Valerie Conley for broadening my horizons even further and offering me inspiring opportunities. I would like to thank my fellow doctoral students and friends at OU for their support, feedback, and friendship. To Leslie and Moira for spending time helping me with my codebook and providing feedback to my ideas and drafts. To Chris for his super efficient help with journals and articles. To Gab for proofreading several of my drafts. To Dianne, Weiyu, and Yanju for valuable suggestions and boosting my confidence when thing were difficult. Last, but certainly not least, thanks to my family, for the love and support that can only come from a great mother and father; for the personal support and tolerance from my husband, and for the unconditional love from my daughter, who constantly reminds me to strive to become the best I can be. 6 Table of Contents Page Abstract ............................................................................................................................... 3 Dedication ........................................................................................................................... 4 Acknowledgments ............................................................................................................... 5 Table of Contents ................................................................................................................ 6 List of Tables ...................................................................................................................... 9 List of Figures ................................................................................................................... 10 Chapter 1: Introduction ..................................................................................................... 11 Purpose of the Study ..................................................................................................... 16 Research Questions ....................................................................................................... 17 Significance .................................................................................................................. 18 Basic Assumptions ........................................................................................................ 24 Definition of Terms ...................................................................................................... 27 Limitations .................................................................................................................... 30 Delimitations ................................................................................................................. 32 Chapter 2: Review of Literature ....................................................................................... 35 Program Evaluation Standards ...................................................................................... 35 The Joint Committee Evaluation Standards .................................................................. 37 Qualitative Criteria: Concepts, Terms and Connotations ............................................. 39 Terms. ....................................................................................................................... 41 What qualitative criteria are not. ............................................................................... 43 Criteria becoming specific and generic. .................................................................... 44 Credibility: Definition and Approaches ........................................................................ 47 Definition. ................................................................................................................. 47 Approaches to promote credibility. ........................................................................... 51 Credibility Techniques .................................................................................................. 53 Triangulation. ............................................................................................................ 53 Peer debriefing. ......................................................................................................... 57 Audit trails. ................................................................................................................ 57 7 Negative case analysis. ............................................................................................. 58 Thick descriptions. .................................................................................................... 59 Prolonged engagement and persistent observation. .................................................. 59 Reflexivity. ................................................................................................................ 60 Combined Use of Credibility Techniques .................................................................... 61 Credibility Techniques and Evaluation Standards: Connections and Examples .......... 63 Mixed Methods Synthesis ............................................................................................. 68 Synthesis features. ..................................................................................................... 71 The mixed method research features. ....................................................................... 74 The mixed method research quality criteria. ............................................................. 75 Summary ....................................................................................................................... 80 Chapter 3: Research Design and Methodology ................................................................ 82 Introduction ................................................................................................................... 82 Data Collection ............................................................................................................. 85 Journal selection. ....................................................................................................... 85 Article selection. ....................................................................................................... 87 Searching procedures and