PHYSICS COMMUNICATION and PEER ASSESSMENT in a REFORMED INTRODUCTORY MECHANICS CLASSROOM a Dissertation Presented to the Academi
Total Page：16
File Type：pdf, Size：1020Kb
PHYSICS COMMUNICATION AND PEER ASSESSMENT IN A REFORMED INTRODUCTORY MECHANICS CLASSROOM A Dissertation Presented to The Academic Faculty By Scott S. Douglas In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the School of Physics Georgia Institute of Technology August 2018 Copyright c Scott S. Douglas 2018 PHYSICS COMMUNICATION AND PEER ASSESSMENT IN A REFORMED INTRODUCTORY MECHANICS CLASSROOM Approved by: Dr. Michael Schatz, Advisor Dr. Wendy Newstetter School of Physics College of Engineering Georgia Institute of Technology Georgia Institute of Technology Dr. Richard Catrambone Dr. Edwin Greco School of Psychology School of Physics Georgia Institute of Technology Georgia Institute of Technology Dr. Flavio Fenton Date Approved: May 25, 2018 School of Physics Georgia Institute of Technology On two occasions I have been asked, — “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” ...I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. Charles Babbage [We] should not assume that experts are the only ones who have a right to express themselves... I am very conscious of the fact that our feelings and strivings are often contradictory and obscure and that they cannot be expressed in easy and simple formulas. Albert Einstein This thesis is dedicated to Anat and to the new person who will shortly join our family. The former I have loved for many years, together and apart, and has taught me innumerable things—the latter, I expect, will teach me even more. ACKNOWLEDGEMENTS I would like to thank Mike Schatz and Ed Greco for their continual support and en- couragement. The introductory physics program they helped to build at Georgia Tech has been the testing-ground for all the research in this thesis, and their personal concern for the welfare and development of their students has been continually demonstrated by the time and effort they dedicate to teaching. I would like to thank John Aiken, Shih-Yin Lin, Emily Alicea-Munoz,˜ and Danny Ca- ballero for the challenges they both posed to me and helped me to overcome. They have all been professional, personal, and academic inspirations to me, as well as indispensable companions at our conferences. Anat Fintzi has been my partner and my guide for many years, including some rather difficult ones full of long-distance separation and uncertain reward. Her own academic work has informed mine, her statistical expertise has saved me from some grievous analyt- ical errors, and without her loving support, it would have been impossible for me to write a single line of this thesis. I would like to name the talented animators who helped to create our lecture videos, without whom our lecture series would not exist. They are Abhinav Uppal, Megan Llewellyn, Anthony Bandoly, Alana Lin, Olivia Ginn, Christina Chung, Nathan Olivares, Chris Syed, Zachary Wilken, and Evan Seitz. Special thanks goes to Evan for taking the lead on the Physics II video production team. The developers and designers on the VIP team deserve credit for helping to build the SWAPR system for peer assessment. They are Stuart Freeman, Rachel Golding, Manley Roberts, Brad Reardon, Joseph Lesniak, Ayan Das, Christopher Wang, Titus Woo, Aadil Omar, Julian Barnes, and Austin Duncan. Without their help and that of the staff and faculty on the Georgia Tech VIP program, our peer assessment system would have likely remained unknown outside of our own small lab. v Finally, I would like to acknowledge all the colleagues, comrades, and creatures, known and unknown, on whose work I relied and from whose intellectual heritage I drew. My own efforts, as I see them, are just a single contribution to the vast social enterprise we undertake together. Our work presented in this thesis was funded in part by the Bill and Melinda Gates Foundation, The Governor’s Office of Student Achievement (Georgia), and the National Science Foundation (DUE-0942076). Our research methods were approved by the Georgia Tech Institutional Review Board (protocols H09382, H13124, H13302, and H13308). vi TABLE OF CONTENTS Acknowledgments . v List of Tables . xiii List of Figures . xiv Chapter 1: Introduction . 1 1.1 Research Questions . 3 Chapter 2: Background and Methods . 5 2.1 Introduction . 5 2.2 Novice and Expert Comparisons . 5 2.3 The Motivations and Forms of Peer Assessment . 8 2.4 Previous Peer Assessment Research . 10 2.5 Laboratory Design . 11 2.5.1 Observational Data . 11 2.5.2 Motion Analysis . 13 2.5.3 Computational Modeling . 13 2.5.4 Video Lab Reports . 15 2.5.5 Rubric . 15 vii 2.6 Classroom Practices . 16 2.6.1 Spring 2014—On-campus . 16 2.6.2 Summer 2016 & 2017—Online . 18 2.7 Our Peer Assessment System . 18 2.7.1 Type and Motivation . 18 2.7.2 Our Peer Assessment Process . 20 2.8 Quantitative & Qualitative Research Methods . 21 2.8.1 Statistical Tests for Quantitative Analysis . 23 2.8.2 Significance Testing . 25 2.8.3 Parametric and Nonparametric Tests . 26 2.8.4 Effect Size . 26 2.8.5 Tests Used in This Thesis . 27 Chapter 3: Validity of Our Peer Assessment System . 31 3.1 Introduction . 31 3.2 Peer Assessment and Validity . 31 3.3 Our Peer Assessment System Compared to Others . 33 3.3.1 Effect Size . 33 3.3.2 Correlation . 34 3.3.3 Reasons for Relatively Low Agreement . 36 3.4 Calibration . 38 3.4.1 SWAPR 1.0 . 39 3.4.2 SWAPR 2.0 . 39 viii 3.4.3 Algorithm Development . 39 3.4.4 Algorithm Optimization . 44 3.5 Conclusion . 44 Chapter 4: Student Rating Changes Over Time . 46 4.1 Introduction . 46 4.2 Peer Ratings of Peer Videos . 47 4.2.1 Peer and Expert Grades . 47 4.2.2 Peer and Expert Grade Differences . 49 4.2.3 Peer and Expert Ratings . 51 4.3 Traditional Measures of Physics Conceptual Development . 55 4.4 Ratings by Subpopulation . 57 4.4.1 Ratings by Gender . 58 4.4.2 Ratings by Incoming GPA . 59 4.5 Generalizability and Repeatability . 60 4.6 Summary . 63 Chapter 5: Student Attitudinal Changes Over Time . 64 5.1 Interview Protocol . 65 5.2 Initial Findings . 66 5.3 Qualitative Themes . 67 5.3.1 Theme: Being ‘Nice’ or ’Harsh’ . 68 5.3.2 Theme: Expertise and Criticality . 70 5.3.3 Other Themes and Behaviors . 73 ix 5.4 Summary . 74 Chapter 6: Student Communication Changes Over Time . 76 6.1 Classifying Communicative Content . 76 6.1.1 The Prediction vs. Observation Discussion . 77 6.1.2 Prediction vs. Observation Coding Scheme . 78 6.1.3 Explanation Completeness . 84 6.2 Completeness vs. Student Ratings of Physics Content . 87 6.2.1 Completeness vs. Expert Ratings of Physics Content . 88 6.3 Discussion . 90 6.3.1 Student & Expert Rating Trends . 90 6.3.2 Student Communicative Trends . 93 6.4 Summary . 95 Chapter 7: Summary & Future Work . 97 7.1 Summary . 97 7.1.1 Background, Motivation, Design, and Validity of our Peer Assess- ment System . 97 7.1.2 Student Rating Changes Over Time . 98 7.1.3 Student Attitudinal Changes Over Time . 99 7.1.4 Student Communication Changes Over Time . 101 7.2 Future Work . 101 7.2.1 Validity of Our Peer Assessment System . 102 7.2.2 Student Rating Changes Over Time . 103 x 7.2.3 Student Attitudinal Changes Over Time . 104 7.2.4 Student Communication Changes Over Time . 105 7.2.5 New Exploratory Research . 106 7.3 Final Remarks . 107 Appendix A: Rubric . 110 Appendix B: How Peer Grading Works . 112 Appendix C: Comparison Codes . 115 Appendix D: Attribution Codes . 117 Appendix E: Pre/Post Interview Summaries . 119 E.1 Student A . 119 E.2 Student B . 120 E.3 Student C . 120 E.4 Student D . 121 E.5 Student E . 123 Appendix F: Interview Transcripts . 126 F.1 Student 1, Spring 2014 . 127 F.2 Student 2,.