Playground Physics Impact Study
Total Page：16
File Type：pdf, Size：1020Kb
Playground Physics Impact Study Lawrence B. Friedman, Ph.D. Jonathan Margolin, Ph.D. Andrew Swanlund, Ph.D. Sonica Dhillon Feng Liu, Ph.D February 2017 Playground Physics Impact Study September 2015 February 2017 Lawrence B. Friedman, Ph.D Jonathan Margolin, Ph.D. Andrew Swanlund, Ph.D. Sonica Dhillon Feng Liu, Ph.D. 1000 Thomas Jefferson Street NW Washington, DC 20007-3835 202.403.5000 | TTY 877.334.3499 www.air.org Copyright © 2017 American Institutes for Research. Xxxx_xx/17 Contents Executive Summary .........................................................................................................................1 Student Outcomes ......................................................................................................................1 Program Fidelity of Implementation and Use ............................................................................2 Conclusions ................................................................................................................................2 Chapter 1: Introduction ....................................................................................................................4 Playground Physics Program .....................................................................................................4 Research Questions and Study Design ......................................................................................8 Chapter 2: Study Design, Sample and Data Sources .....................................................................10 Sample......................................................................................................................................10 Sample Characteristics .......................................................................................................12 Data Sources ............................................................................................................................15 Student Science Knowledge Assessment...........................................................................15 Pre- and Posttest Student Survey. ......................................................................................16 Key Program Components and Criteria for Fidelity of Implementation .................................19 Chapter 3: Student Outcomes ........................................................................................................21 Baseline Equivalence ...............................................................................................................21 Playground Physics Impact on Students ..................................................................................22 How does participation in Playground Physics influence middle school students’ knowledge of physics concepts? ......................................................................................23 How does participation in Playground Physics influence middle school students’ engagement in science class? ...........................................................................................23 How does participation in Playground Physics influence middle school students’ motivation, attitudes towards learning science and science-related aspirations careers? .............................................................................................................................24 Does the Effect of Playground Physics Differ as a Function of Gender or Ethnicity? ............25 Summary of Student Outcomes and Limitations .....................................................................26 Chapter 4: Implementation of Playground Physics and Its Relationship to Student Outcomes ......................................................................................................................................27 To what extent were Playground Physics components implemented with fidelity? ...............27 How extensively did teachers use Playground Physics to teach physics? ...............................29 Summary of Fidelity of Implementation and Relationship to Student Outcomes ...................31 Chapter 5: Discussion ....................................................................................................................32 Playground Physics Promotes Greater Knowledge and Understanding of Physics Concepts .................................................................................................................................32 Playground Physics was Not Associated with Differences in Engagement or Attitudes ........33 References ......................................................................................................................................35 Appendix A. Playground Physics Curriculum Activities ..............................................................38 Appendix B. Student Outcome Measures ......................................................................................42 B.1. Playground Physics Student Survey Aligned to Student Affect Constructs ....................42 B.2. Playground Physics Pretest Knowledge Assessment .......................................................46 B.3. Playground Physics Posttest Knowledge Assessment ..... Error! Bookmark not defined. Appendix C. Teacher Survey .........................................................................................................62 C.1. Playground Physics Teacher Survey – Impact Study, Treatment Condition ...................62 C.2. Physics instruction Teacher Survey – Impact Study, Control Condition .........................71 Appendix D. Impact Analysis Technical Approach ......................................................................74 Establishing Baseline Equivalence ..........................................................................................74 Evaluating Group Differences .................................................................................................74 Appendix E. Output from Statistical Models .................................................................................76 Appendix F. Knowledge Assessment Responses and Standards Alignment .................................79 Appendix G. 2014-15 Fidelity of Implementation Analysis .........................................................80 Appendix H. Supplemental Analysis .............................................................................................84 Executive Summary Playground Physics is a technology-based application and accompanying curriculum designed by New York Hall of Science (NYSCI) to support middle school students’ science engagement and learning of force, energy and motion. The program includes professional development, the Playground Physics app, and a curriculum aligned with New York State Learning Standards, Common Core State Standards, and the Next Generation Science Standards. The iOS app allows students to record and review videos through three “lenses:” motion, force (Newton’s third law), and energy, and the curriculum integrates informal and formal inquiry-based learning strategies to promote greater student knowledge and understanding of physics. The program was designed to be implemented in a formal school setting during the regular school day. This report describes the results of an experimental study of the Playground Physics program’s impact on learning of physics concepts, student engagement, and science-related attitudes. Sixty New York City middle grade teachers were randomly assigned to treatment or control conditions. Treatment teachers were asked to participate in Playground Physics professional development and use Playground Physics as part of their physics instruction during the 2015-16 academic year; control teachers were asked to use their regular instruction. In total, 15 teachers left the study. The final sample included student data from 24 treatment teachers and 21 control teachers. Student Outcomes Student knowledge of physics concepts, engagement in science class, and science-related attitudes were measured at two points in time: once prior to teacher implementation of Playground Physics (fall 2015) and once after teacher completion of physics instruction (winter/spring 2016). Students completed a knowledge assessment comprising 20 multiple- choice questions aligned to four New York science standards related to the content covered in the Playground Physics program. Students also completed a survey with groups of items addressing engagement in science class and the following constructs of science-related attitudes: interest in science, science self-concept, intrinsic motivation, and educational and career plans relevant to science. A two-level hierarchical linear model with students nested within teachers was employed to examine differences in these outcomes between students of teachers who used Playground Physics as part of their physics instruction and those who did not. Means and differences were regression adjusted to account for student grade level, demographic characteristics, and performance on pretest measures, as well as teacher years of instructional experience. Impact results for each outcome measure were calculated separately. Students of treatment teachers showed significantly greater physics knowledge at posttest than students of control teachers. No differences were noted for student engagement in science class or the four constructs of science- related attitudes. Teachers in the two conditions did not differ with respect to the total number of days spent teaching these physics topics,