AN ABSTRACT OF THE DISSERTATION OF Gideon R. Litherland for the degree of Doctor of Philosophy in Counseling presented on July 29, 2020. Title: Cross-Validation of Two Supervision Instruments with Counseling Trainees from CACREP-accredited Programs Abstract approved: ______________________________________________________ Kok-Mun Ng Supervision is considered a pivotal professional intervention for counselors-in-training as they develop during graduate education and beyond. But, research on clinical supervision suffers from a lack of common instruments that can be utilized across disciplines, including counseling, and national boundaries. While a complex phenomenon to empirically address through research, supervision scholars have called for greater scrutiny in the types of instruments, measurement models, and research designs employed so that the field of supervision may be advanced (Goodyear et al., 2016). Additionally, how supervision-related phenomena are conceptualized, like supervision effectiveness and supervisor competence, within the research is foundational to the development of a robust research ecosystem. In order to contribute to the need for psychometrically robust supervision instruments that are grounded in robust theory, this dissertation project included two supervision measurement cross-validation studies on counselors-in-training in the United States. In these studies, I examined the psychometric properties of two different supervision instruments that share a similar theoretical conceptualization of supervision: (a) effective supervision (Study 1) and (b) supervisor competence (Study 2). These two measures were developed in non-U.S. clinical settings but whose psychometric properties and utility have yet to be verified for U.S.-based counseling practitioners. The overarching research question that both studies sought to address was: “Do existing supervision evaluation instruments maintain rigorous psychometric evidence for a sample of CITs from CACREP-accredited programs?” Conceptually, supervision effectiveness and supervisor competence were defined terms of the Proctor Model of Supervision. Each study drew from one sampling of 86 participants who were master’s-level counselors-in-training at CACREP-accredited programs from every region in the United States. The first study considered the psychometric properties of an instrument, the Manchester Clinical Supervision Scale (MCSS-26), that has been long-utilized to measure clinical supervision effectiveness outside of the U.S. In addition to the overarching research question identified above, Study 1 evaluated item-level performance and instrument-level internal consistency, concurrent validity, and social desirability threats to validity. The MCSS-26 was subjected to item-level analysis using a Generalized Partial Credit Model (GPCM) to explore the item difficulty, discrimination, and satisfaction to theoretical assumptions. Results of the study indicate acceptable instrument-level validity and reliability but poor item-level fitness for multiple items from the original 26-item instrument. Based on sample data, results suggest the revision of the MCSS-26 to a 9-item instrument that more appropriately fits within the item- response theoretical model of analysis for the study sample. Fitness indices for the revised scale suggest a better model fit compared to the fitness indices of the original instrument. Further revision, through continued research, is necessary in order to critically revise the MCSS-26 for use with a US-based counselor-in-training population. The second study examined an instrument that assesses supervisor competence, the Supervision Evaluation and Supervisor Competence (SE-SC) scale, from the supervisee’s perspective. Study 2, similar to Study 1, evaluated the item-level and instrument-level psychometrics of the subscales of the SE-SC. Item performance, internal consistency, concurrent validity, and social desirability threats to validity were all considered. The SE-SC was subjected to item-level analysis based on a GPCM that resulted in difficulty and discrimination parameters while also considering key theoretical assumptions of the model. Data from the current sample indicate acceptable instrument validity and reliability; however, item-level fitness to the model was poor, or “misfitting,” for a number of items. Results of Study 2 indicate the need for ongoing refinement of the SE-SC before use with a U.S.-based CIT population. Results further indicate that a 15-item revised SE-SC could be further developed with scrutiny. The revised scale possessed improved fitness indices compared to the original instrument, indicating a better fit to the GPCM. Supervision instruments that are relevant for U.S.-based CIT are sorely needed and considered critical to the development of the supervision scholarship in the years to come. As two supervision instruments that have been used to assess effectiveness and supervisor competence, the findings from both studies cast doubt on their utility for the population of U.S.- based CIT. Implications for Study 1 and Study 2 are presented with respect to instrument revision/development, counselor education and training, and the common measurement approach in supervision research. Additionally, findings from both studies suggest the urgency of constructing, refining, and developing psychometrically robust supervision instruments that can precisely assess supervision effectiveness and supervisor competence in future research. Overall, each study contributes to the supervision scholarship by casting doubt on two extant supervision instruments for use with a U.S.-based CIT population. ©Copyright by Gideon R. Litherland July 29, 2020 All Rights Reserved Cross-Validation of Two Supervision Instruments with Counseling Trainees from CACREP-accredited Programs by Gideon R. Litherland A DISSERTATION submitted to Oregon State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy Presented July 29, 2020 Commencement June 2021 Doctor of Philosophy dissertation of Gideon R. Litherland presented on July 29, 2020 APPROVED: Major Professor, representing Counseling Dean of the College of Education Dean of the Graduate School I understand that my dissertation will become part of the permanent collection of Oregon State University libraries. My signature below authorizes release of my dissertation to any reader upon request. Gideon R. Litherland, Author ACKNOWLEDGEMENTS I am ever grateful to the many people who have helped, challenged, and supported me along my professional path. This doctoral degree would not have been possible without their energy and investment in my success and growth as a person and professional. These brief acknowledgements do not do the gratitude I have for them justice! I would like to thank my committee members, Dr. Arien Muzacz, Dr. Abraham Cazares- Cervantes, and Dr. Thomas Miller. I appreciate your constant support, guidance, and time. Each of you have contributed to the completion of my doctoral studies and have supported me to get to this point. To my mentors and friends, thank you. The grace, compassion, and guidance you have provided me over the years has shaped me in ways words cannot express. I am a better human for knowing each of you and I fully intend to pay it forward. In gratitude Jane Bello-Brunson, Darlene Grega, Dr. Fran Giordano, and Carolyn Schneider. To my co-chair and mentor, Dr. Thom Field, you jumped into this dissertation project with a clear intention to be facilitative from the get go. I am grateful to you for sharing your expertise and wisdom with such direct clarity and, at times, helping me see another perspective. Thank you for your service. To my advisor, chair, and mentor, Dr. Kok-Mun Ng, you have taught me more than just how to be a professional. Your humor, generosity of self, and scholarly rigor are a rare combination that I strive to reflect in my ongoing personal development. Thank you for your constant support, feedback, and connection. To my fellow doctoral candidates and co-conspirators in fostering excellence in counseling, Gretchen Schulthes, Christy Cosper, Rachel Ware Zooi, Nineka Dyson, and Roberta Miranda, I am indebted to each of you for pushing me to be a more compassionate and critical thinking. You are my village and I do believe it took our paths to cross for me to arrive at this point. Love to you all. To the Litherlands, Downings, Neimans, and Roberts, thank you for all of the love, patience, and understanding you have sent my way over the years. I would not have been able to complete this degree without you all. You keep me humble and grounded. To Spencer Neiman, my fiancé and husband-to-be, this accomplishment is as much yours as it is mine. From the beginning, you have encouraged me to imagine what could be instead of just seeing how things are. I would not be where I am today without you in my life. Thank you, love you. TABLE OF CONTENTS Page Chapter 1: Thematic Introduction ................................................................................................... 1 Instrument Development Issues ....................................................................................................... 1 Building an Empirical Body Through Cross-Validation .................................................................... 3 Studying Effectiveness and Competence in Supervision .................................................................... 4 The Proctor Model ........................................................................................................................