Development and Validation of the Statistics Assessment of Graduate Students
Total Page:16
File Type:pdf, Size:1020Kb
University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange Doctoral Dissertations Graduate School 12-2016 Development and Validation of the Statistics Assessment of Graduate Students Dammika Lakmal Walpitage University of Tennessee, Knoxville, [email protected] Follow this and additional works at: https://trace.tennessee.edu/utk_graddiss Part of the Applied Statistics Commons, Educational Assessment, Evaluation, and Research Commons, Science and Mathematics Education Commons, and the Social Statistics Commons Recommended Citation Walpitage, Dammika Lakmal, "Development and Validation of the Statistics Assessment of Graduate Students. " PhD diss., University of Tennessee, 2016. https://trace.tennessee.edu/utk_graddiss/4113 This Dissertation is brought to you for free and open access by the Graduate School at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Doctoral Dissertations by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a dissertation written by Dammika Lakmal Walpitage entitled "Development and Validation of the Statistics Assessment of Graduate Students." I have examined the final electronic copy of this dissertation for form and content and recommend that it be accepted in partial fulfillment of the equirr ements for the degree of Doctor of Philosophy, with a major in Educational Psychology and Research. Gary J. Skolits, Major Professor We have read this dissertation and recommend its acceptance: Jennifer A. Morrow, Ralph S. McCallum, Hamparsum Bozdogan Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official studentecor r ds.) Development and Validation of the Statistics Assessment of Graduate Students A Dissertation Presented for the Doctor of Philosophy Degree The University of Tennessee, Knoxville Dammika Lakmal Walpitage December 2016 Copyright © 2016 by Dammika Lakmal Walpitage All rights reserved. ii DEDICATION To my mother, father, and grandparents, who introduced me to the joy of learning and prepared me with unconditional love to chase my dreams. To my wife and two daughters, whose endless affection and encouragement made me able to achieve such success and honor. iii ACKNOWLEDGEMENTS I wish to thank my committee members who were more than generous with their expertise and precious time. A special thanks to Dr. Gary Skolits, my advisor and committee chairperson, for his countless hours of advising, constructive comments, encouragement, and patience throughout my entire journey of doctoral studies. Thank you Dr. Jennifer Morrow, Dr. Steve McCallum, and Dr. Hamparsum Bozdogan for providing me with continued support and encouragement. I also must thank Dr. Melisa Martin, Kelly Smith and Maya Mingo, and members of Psycho-educational studies research group. I specifically thank Dr. Kent Wagoner for his invaluable assistance with item development for this project. I express my appreciation to all my friends for helping and inspiring me to achieve more. I also extend my heartfelt thanks to my family. Thank you so much for always being there for me. iv ABSTRACT This study developed the Statistics Assessment of Graduate Students (SAGS) instrument, and established its preliminary item characteristics, reliability, and validity evidence. Even though there are limited number of assessments available for measuring different aspects of statistical cognition, these previously available assessments have numerous limitations. The SAGS instrument was developed using Rasch modeling approach to create a new measure of statistical research methodology knowledge of graduate students in education and other behavioral and social sciences. Thirty-five multiple-choice questions were written with stems representing applied research situations and response options distinguishing between appropriate use of various statistical tests or procedures. A focus group meeting with upper level graduate students was held in order to revise the initial instrument. Then, a six-person expert panel reviewed the revised items for content validity and to improve the quality of the instrument. The finalized SAGS instrument with 25 cognitive questions and demographic questionnaire was administered online, and 132 participants fully completed the instrument. Results showed that, one SAGS item was not consistent with the Rasch model. This item and distractors of two other items were flagged to be modified during future administrations. Reliability indices, separation indices, constructs maps, and known group comparisons provided the supportive evidence for reliability and validity. Preliminary simulation study conducted with higher order IRT models rejected three parameter logistic (3 PL) model and indicated no impact of guessing parameter when describing the observed data. A simulation study further provided positive evidence towards using ICOMP type model selection criteria that guard against correlations of parameter estimates when choosing the best model among a portfolio of IRT models. Sample independent parameter estimates obtained using Rasch and IRT approaches in this study open an avenue to develop customizable yet psychometrically sound statistical research methodology assessments. v TABLE CONTENTS CHAPTER ONE: INTRODUCTION AND GENERAL INFORMATION .................................. 1 Problem Statement ...................................................................................................................... 3 Purpose of the Study ................................................................................................................... 8 Significance of the Study .......................................................................................................... 10 Limitations of the Study ............................................................................................................ 12 Definitions of Key Terms .......................................................................................................... 12 Organization of the Study ......................................................................................................... 13 CHAPTER TWO: LITERATURE REVIEW ............................................................................... 15 Graduate Students and Statistics Education .............................................................................. 15 Why Statistics is Taught at the Graduate Level? ................................................................... 15 Importance of Selecting an Appropriate Statistical Procedure/Test ..................................... 17 Statistics Education and Statistics Education Research ............................................................ 18 Importance of Assessing Cognitive Outcomes ......................................................................... 20 Assessing Statistical Constructs ................................................................................................ 21 Defining the Cognitive Constructs Associated with Statistics Education Research ................. 22 Review of Previous Studies on Developing Statistics Skills Assessment Scales ..................... 24 Statistical Reasoning Assessment (SRA) (1998) .................................................................... 24 Quantitative Reasoning Quotient (QRQ) (2003) ................................................................... 26 Instruments in (ARTIST) Project and its Relatives, CAOS (2002) ........................................ 27 Instruments in (ARTIST) Project and its Relatives, ARTIST Topic Scales (2002) ................ 29 Goals and Outcomes Associated with Learning Statistics (GOALS) (2012)......................... 30 Basic Literacy in Statistics (BLIS) (2014) ............................................................................. 31 Statistics Concept Inventory (SCI) (2002) ............................................................................. 32 Statistical Literacy Survey (SLS) ........................................................................................... 33 Development of the SAGS Instrument...................................................................................... 34 Brief Review of Classical Test theory (CTT) ........................................................................... 41 Item Response Theory (IRT) against Classical Test Theory (CTT) ......................................... 44 Review Basics of Item Response Theory .................................................................................. 45 IRT Parameters...................................................................................................................... 46 vi Assumptions of IRT ................................................................................................................ 48 Three Common IRT Models for Dichotomous Data .............................................................. 49 Goodness of Fit of IRT Models .............................................................................................. 52 Reliability in IRT ................................................................................................................... 53 Rasch Modeling as a Subset of IRT: Mathematically............................................................ 56 Differences in IRT and Rasch Modeling: