Scientific Reasoning: Research, Development, and Assessment
Total Page:16
File Type:pdf, Size:1020Kb
SCIENTIFIC REASONING: RESEARCH, DEVELOPMENT, AND ASSESSMENT DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Jing Han, B.S., M.S. ***** The Ohio State University 2013 Dissertation Committee: Approved by Professor Lei Bao, Adviser Professor Fengyuan Yang _______________________ Professor Andrew F. Heckler Adviser Physics Graduate Program Professor Evan R. Sugarbaker © Copyright by Jing Han 2013 ABSTRACT Education in Science, Technology, Engineering, and Math (STEM) is emphasized worldwide. Reports from large-scale international studies such as TIMSS and PISA continually rank U.S. students behind many other nations. As a result, the U.S. has increased its emphasis on the implementation of a more extensive science and mathematics curriculum in K-12 education. In STEM education, widely accepted teaching goals include not only the development of solid content knowledge but also the development of general scientific abilities that will enable students to successfully handle open-ended real-world tasks in future careers. One such ability, scientific reasoning, is closely related to a wide range of general cognitive abilities such as critical thinking and reasoning. Existing research has suggested that scientific reasoning skills can be trained and transferred. Training in scientific reasoning may also have a long-term impact on student academic achievement. In the STEM education community, it has been widely agreed that student development of transferable general abilities is at least as important as certain learned STEM knowledge. Therefore, it is important to investigate how to implement a STEM education program that can help students develop both STEM content knowledge and scientific reasoning. ii In order to develop such a knowledge base and to assess and evaluate the impact and effectiveness of education methods and resources, we need good assessment tools that can be easily applied in large scale and produce valid results comparable across a wide range of populations. In the area of scientific reasoning, there exists a practical tool, the Lawson’s Classroom Test of Scientific Reasoning, and since its initial development in the late 1970s and early 1980s, the test has undergone several revisions with the current version released in 2000. Although the Lawson’s test has provided much useful information for research and assessment purposes, the test itself hasn’t been systematically validated and several issues have been observed in large scale applications concerning item designs and scalability of the results (details will be provided in later sections). Therefore, there is an urgent need for systematic research on validating the Lawson’s test and further development of validated standardized assessment tools on scientific reasoning for K-18 students. This dissertation project establishes a first step to systematically improve the assessment instrumentation of scientific reasoning. A series of studies have been conducted: (1) A detailed validation study of the Lawson’s test, which has identified a number of validity issues including item/choice design issues, item context issues, item structure and wording issues (e.g. two-tier design), the limited scale of measurement range, and the ceiling effect for advanced students. iii (2) A study to determine the basic measurement features of the Lawson’s test with large scale data. (3) A data-mining study of Lawson’s test data, which helps identify learning progression behaviors of selected scientific reasoning skills. The results also provide evidence for researchers to evaluate and model the scoring methods of two-tiered questions used in the Lawson’s test. (4) A study with randomized testing to investigate the learning progression of the skill of control of variables (COV), which showed a series of fine grained intermediate levels of COV skills. This project produces rich resources for sustained research and development on scientific reasoning. It establishes a valuable baseline for teachers and researchers to apply the Lawson’s test in research and teaching and a solid foundation for researchers to further develop the next generation assessment instruments on scientific reasoning. iv ACKNOWLEDGMENTS I am extremely grateful to my adviser, Prof. Lei Bao, for all his support since my initial meeting to find out about working with the physics education research group. I would like to thank Prof. Andrew Heckler, Prof. Fengyuan Yang, and Prof. Evan Sugarbaker for serving on my committee. I also would like to thank fellow graduate students in the physics education research group for all the helpful discussions and support. I thank my family and friends for their unconditional support. Most importantly, I would like to thank the students for their participation and cooperation with this project. v VITA 2008-Present…………………………… Research Assistant, Department of Physics, The Ohio State University 2010…………………………………..... M.S. Physics, The Ohio State University 2006…………………………………..... B.S. Physics, Capital Normal University, Beijing, China PUBLICATIONS Jing Han, Li Chen, Yibing Zhang, Shaona Zhou,Lei Bao, “Seeing what students see in doing the Force Concept Inventory”, Am. J. Phys., In Press. Lei Bao, Amy Raplinger, Jing Han, Yeounsoo Kim, “Assessment of Students’ Cognitive Conflicts an Anxiety” , Journal of Research in Science Teaching, Submitted. Li Chen, Jing Han, Lei Bao, Jing Wang, Yan Tu, “Comparisons of Item Response Theory Algorithms with Force Concept Inventory Data”, Research in Education Assessment and Learning, 2 (02), 26-34, (2011). vi Shaona Zhou, Jing Han, Nathaniel Pelz, Xiaojun Wang, Liangyu Peng, Hua Xiao, Lei Bao, Inquiry Style Interactive Virtual Experiments: A Case on Circular Motion, Eur. J. Phys. 32 1597, 2011 Tianfang Cai, Hongmin Zhao,Jing Han, “Clicker in introductory physics class of Chinese University,” Physics and engineering, Vol.2, pp51-53, 2010( in Chinese). Lei Bao, Tianfang Cai, Kathy Koenig, Kai Fang, Jing Han, Jing Wang, Qing Liu, Lin Ding, Lili Cui, Ying Luo, Yufeng Wang, Lieming Li, Nianle Wu, “Learning and Scientific Reasoning”, Science, Vol. 323. no. 5914, pp. 586 – 587 (2009). Lei Bao, Kai Fang, Tianfang Cai, Jing Wang, Lijia Yang, Lili Cui, Jing Han, Lin Ding, and Ying Luo “Learning of Content Knowledge and Development of Scientific Reasoning Ability: A Cross Culture Comparison,” Am. J. Phys., 77 (12), 1118-1123 (2009). Jing Han, Dan Li, “Research in Moral and Law Education in XuanWu District”, Excellent Thesis of Special Undergraduate Grant for Research CNU, 2006. ( in Chinese) Jing Han, Wu Zheng “Enhancing the Gauss Theorem Education through Analysis of Universal Gravitational Field Intensity” Discussion of Physics Education, Vol. 25, 2006 (in Chinese). vii FIELDS OF STUDY Major Field: Physics viii TABLE OF CONTENTS Page Abstract …………………………………………………………………………………. ii Acknowledgments ……………………………………………………………………….. v Vita …………………………………………………………………………………..…...vi List of Figures ………………………………………………………………………….. xii List of Tables ………………………………………………………………….............. xiv Chapters: Chapter 1.Introduction to Research on Scientific Reasoning……………………………. 1 1.1 What is Scientific Reasoning? …………………………………………………... 1 1.2 Why is Scientific Reasoning Important? ………………………………………... 9 1.3 How is Scientific Reasoning Learned? …………………………………………12 1.4 How is Scientific Reasoning Assessed? ……………………………………….. 13 1.5 Scientific Reasoning, an Important Component of the 21stCentury Skills…….. 15 1.5.1 Skills Gap between Schools and Workplaces…………………………......17 1.5.2 What are 21st Century Skills? …………………………………………… 19 1.6 Outline of the Thesis…………………………………………………………….22 Chapter 2.Research on Assessment of Scientific Reasoning…………………………... 24 2.1 Theoretical Background of Scientific Reasoning………………………………. 24 2.2 Existing Research and Tools on Assessment of Scientific Reasoning…………. 28 2.2.1 Group Assessment of Logical Thinking (GALT) ………………………...29 2.2.2 The Test of Logical Thinking (TOLT) …………………………………... 31 2.2.3 Lawson’s Classroom Test of Scientific Reasoning (LCTSR) …………… 32 2.3 Expanding the Dimensions of Skills for Assessment of Scientific Reasoning….36 2.3.1 Control of Variables……………………………………………………….37 2.3.2 Proportions and Ratios…………………………………………………….42 ix 2.3.3 Probability……………………………………………………………….. 45 2.3.4Correlational Reasoning………………………………………………….. 47 2.3.5 Deductive Reasoning…………………………………………………….. 50 2.3.6 Inductive Reasoning………………………………………………………54 2.3.7 Causal Reasoning…………………………………………………….…... 56 Chapter 3. Validity of the Lawson’s Classroom Test of Scientific Reasoning.…..…… 60 3.1 A Historical Review on the Development of the Lawson’s Test………………. 60 3.2 Content Evaluation of Lawon’s Test – Item Context Issues…………………… 66 3.3 A Data-Driven Study on the Validity of the Lawson’s Test…………………… 67 3.4 Quantitative Results – Item Score Analysis…………………………………….69 3.5 Quantitative Results – Analysis of Two-Tier Score Patterns………………….. 72 3.6 Qualitative Results – Analysis of Student Interviews…………………………..75 3.7 Consideration on Two-Tier Question Structure……………………………….. 84 3.8 The Ceiling Effect and Measurement Saturation of the Lawson’s Test……….. 85 3.9 Conclusions…………………………………………………….………………. 88 Chapter 4.The Developmental Metric of Scientific Reasoning………………………... 89 4.1 Context of the Study…………………………………………………….……... 89 4.2 Data Collection…………………………………………………….…….…….. 90 4.3 The Developmental Scales of the Lawson’s Test Scores……………………… 93 4.3.1 The Learning Evolution Index