Purdue University Purdue e-Pubs Open Access Dissertations Theses and Dissertations Spring 2015 Difficulty as a concept inventory design consideration: An exploratory study of the concept assessment tool for statics (CATS) Dana L. Denick Purdue University Follow this and additional works at: https://docs.lib.purdue.edu/open_access_dissertations Part of the Educational Assessment, Evaluation, and Research Commons, and the Engineering Education Commons Recommended Citation Denick, Dana L., "Difficulty as a concept inventory design consideration: An exploratory study of the concept assessment tool for statics (CATS)" (2015). Open Access Dissertations. 448. https://docs.lib.purdue.edu/open_access_dissertations/448 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. Graduate School Form 30 Updated 1/15/2015 PURDUE UNIVERSITY GRADUATE SCHOOL Thesis/Dissertation Acceptance This is to certify that the thesis/dissertation prepared By Dana L. Denick Entitled Difficulty as a Concept Inventory Design Consideration: An Exploratory Study of the Concept Assessment Tool for Statics (CATS) For the degree of Doctor of Philosophy Is approved by the final examining committee: Ruth Streveler Chair Monica Cardella Senay Purzer Paul Steif To the best of my knowledge and as understood by the student in the Thesis/Dissertation Agreement, Publication Delay, and Certification Disclaimer (Graduate School Form 32), this thesis/dissertation adheres to the provisions of Purdue University’s “Policy of Integrity in Research” and the use of copyright material. Approved by Major Professor(s): Ruth Streveler Approved by: Ruth Streveler 4/9/2015 Head of the Departmental Graduate Program Date i DIFFICULTY AS A CONCEPT INVENTORY DESIGN CONSIDERATION: AN EXPLORATORY STUDY OF THE CONCEPT ASSESSMENT TOOL FOR STATICS (CATS) A Dissertation Submitted to the Faculty of Purdue University by Dana L. Denick In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2015 Purdue University West Lafayette, Indiana ii This dissertation is dedicated to the memory of my father David J. Denick (1942-2012) Your commitment to creativity and education continues to inspire me iii ACKNOWLEDGEMENTS I would like to express my sincere gratitude to Dr. Ruth Streveler for her guidance and support throughout my graduate studies at Purdue. I can't imagine experiencing such a valuable, respectful and caring advising relationship with anyone else. I would also like to thank my committee, Dr. Paul Steif, Dr. Senay Purzer, and Dr. Monica Cardella for their counsel and encouragement while working on my dissertation. I am grateful for my writing group and extended research family, especially Nicole Pitterson, Farrah Fayyaz, Natasha Perova, and Aidsa Santiago-Román. Each of you has helped me strengthen my work on this study and grow as a researcher. A special thanks to Emily Dringenberg, whose enthusiasm and friendship kept me going. I would not have been able to complete this dissertation without the support of my family. Most of all, I want to thank my mother for always believing I could get it done. You're my source of inspiration and perseverance; thank you for everything. This work was supported by the National Science Foundation under grant DRL 0918531. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author do not necessarily reflect the views of the National Science Foundation. iv TABLE OF CONTENTS Page LIST OF TABLES ........................................................................................................... viii LIST OF FIGURES ............................................................................................................ x ABSTRACT ...................................................................................................................... xii CHAPTER 1. INTRODUCTION .................................................................................... 1 1.1 Background Information ........................................................................................... 1 1.2 Problem Statement .................................................................................................... 1 1.3 Statement of Purpose ................................................................................................. 2 1.4 Research Question ..................................................................................................... 3 1.5 Overview of Methodology ........................................................................................ 4 1.6 Rationale and Significance ........................................................................................ 4 1.7 Organization of Dissertation ..................................................................................... 6 CHAPTER 2. LITERATURE REVIEW ......................................................................... 8 2.1 Introduction ............................................................................................................... 8 2.1.1 Development of Engineering Concept Inventories .......................................... 8 2.1.2 The Assessment Triangle ............................................................................... 10 2.2 Expanding upon the Cognitive Element of the Assessment Triangle ..................... 13 2.2.1 Nature of Conceptual Understanding ............................................................ 13 2.2.2 Statics Concepts and Conceptual Framework ............................................... 16 2.3 Expanding upon the Observation Element of the Assessment Triangle ................. 19 2.3.1 Concept Inventory Item Design ..................................................................... 20 2.3.2 CATS Item Design ........................................................................................ 21 2.3.3 Concept Inventory Development Process ...................................................... 22 2.3.4 Development of CATS .................................................................................. 23 v Page 2.4 Expanding upon the Interpretation Element of the Assessment Triangle ............... 26 2.4.1 Psychometric Evaluation ............................................................................... 27 2.4.2 Psychometric Evaluation of CATS ................................................................ 29 2.4.3 The Interpretation Vertex as a Validity Argument ........................................ 29 2.4.4 A Comprehensive Approach to Validity for Concept Inventories ................ 30 2.5 Addressing Gaps in Knowledge .............................................................................. 33 2.5.1 Cognitive Load Theory .................................................................................. 33 2.5.2 Evidence-Centered Design ............................................................................ 34 2.6 Summary ................................................................................................................. 36 2.6.1 Interconnectedness of the Assessment Triangle ............................................ 36 CHAPTER 3. METHOD................................................................................................ 38 3.1 Introduction ............................................................................................................. 38 3.2 Research Design ...................................................................................................... 38 3.2.1 Mixing Research Paradigms .......................................................................... 40 3.3 Data Sources ............................................................................................................ 41 3.3.1 CATS ............................................................................................................. 42 3.3.2 Source Material for Content Analysis ........................................................... 42 3.3.3 Interview Participants .................................................................................... 45 3.4 Data Collection ........................................................................................................ 47 3.4.1 Content Analysis Data Collection ................................................................. 47 3.4.2 Think-aloud Interviews .................................................................................. 47 3.4.3 Interview Materials ........................................................................................ 47 3.4.4 Interview Protocol ......................................................................................... 48 3.4.5 Use of Existent Data ...................................................................................... 48 3.5 Method of Analysis ................................................................................................. 49 3.5.1 CATS Item Grouping .................................................................................... 49 3.5.2 Content Analysis ............................................................................................ 51 3.5.3 Thematic Analysis ......................................................................................... 56 3.6 Theme Definitions ................................................................................................... 60 vi Page 3.6.1 Direction of Problem Solving