An Evaluation Report: I3 Development Grant Dev07 – Sammamish High School
Total Page:16
File Type:pdf, Size:1020Kb
An Evaluation Report: i3 Development Grant Dev07 – Sammamish High School “Re-imagining Career and College Readiness: STEM, Rigor, and Equity in a Comprehensive High School” Randy Knuth, Ph.D. Knuth Research, Inc. [email protected] Paul S. Sutton, Ph.D. Pacific Lutheran University [email protected] Sheldon Levias, Ph.D. University of Washington, Seattle Annie Camey Kuo, Ph.D. Stanford University Matthew Callison Indiana University March 1, 2016 Knuth, et al.: An Evaluation Report: Investing in Innovation (i3) Development Grant Dev07 Acknowledgements The findings reported herein were supported by funding from an Investing in Innovation Grant (i3), PR/Award # U396C100150, as awarded by the Department of Education. The authors would like to thank the teachers and school leaders at the awarded institution who gave generously of their time and insights to help us complete this evaluation. The authors acknowledge, as well, the assistance of Andrew W. Shouse and Elizabeth Wright who made contributions to this project and, in some cases, early support towards this evaluation. 2 Knuth, et al.: An Evaluation Report: Investing in Innovation (i3) Development Grant Dev07 Table of Contents Executive Summary ................................................................................................................................... 1 Chapter 1: The School Context ............................................................................................................... 7 District Impact on Teacher-Centered Curriculum Design ..................................................................... 8 Why PBL? ..................................................................................................................................................... 10 Chapter 2: Fidelity of Implementation ............................................................................................... 12 Scoring .......................................................................................................................................................... 22 Analysis ......................................................................................................................................................... 23 Chapter 3: Qualitative Methods ........................................................................................................... 25 Chapter 4: Increasing Teacher Pedagogical Expertise ................................................................. 30 Summer Institute of Learning and Teaching (SILT) ............................................................................. 32 Methodology ................................................................................................................................................ 32 Findings ........................................................................................................................................................ 34 PBL Curriculum Design Teams ................................................................................................................ 38 Methodology ................................................................................................................................................ 41 Findings ........................................................................................................................................................ 43 Chapter 5: School Leadership Structure ............................................................................................ 51 Purpose of Leadership Team..................................................................................................................... 51 Methodology ................................................................................................................................................ 51 Leadership Team Membership .................................................................................................................. 52 Peripheral Members of Leadership Team ............................................................................................... 53 Leadership Team Roles .............................................................................................................................. 53 Leadership Team Meetings ........................................................................................................................ 60 Advisory Board Meetings ........................................................................................................................... 60 Findings ........................................................................................................................................................ 61 Discussion .................................................................................................................................................... 65 Chapter 6: Exploratory Studies 1 and 2 .............................................................................................. 66 Assessing the Impact of Problem Based Learning (PBL) on Students’ Career and College Readiness ...................................................................................................................................................... 66 Stated Goals and Student Populations ..................................................................................................... 67 PBL Redesigned Courses, Student Cohorts, and Student Populations ............................................... 67 How We Describe Groups of Students ................................................................................................... 71 Methodology: Exploratory Study #1 ....................................................................................................... 73 Findings: Exploratory Study #1: Comparison of Student Performance in AP Coursework ........... 78 Impact on Special Populations: English Language Learners ................................................................ 99 PBL Adoption and Comparative AP Scores by Department ............................................................. 107 Adoption of PBL Within Departments ................................................................................................. 109 Comparison of Mean AP Scores by All Students Disaggregated by Department ........................... 122 College and Career Readiness Outcomes for Students Not Participating in AP Coursework ...... 130 i Knuth, et al.: An Evaluation Report: Investing in Innovation (i3) Development Grant Dev07 Findings: Exploratory Study #2: Impact of Starting Strong/Sammamish Leads on Students’ Career and College Readiness ................................................................................................................. 132 Methodology: Exploratory Study #2 ..................................................................................................... 132 Chapter 7: Discussion and Conclusions .......................................................................................... 140 References ................................................................................................................................................ 142 Appendix A. Implementing PBL Classroom Observation Protocol ................................................. 145 Appendix B: Key Element Classroom Observation Protocol ............................................................ 148 Appendix C: Levels of Use (LOU) Teacher Interview Protocol ....................................................... 151 ii Knuth, et al.: An Evaluation Report: Investing in Innovation (i3) Development Grant Dev07 List of Tables and Charts Tables: Table 1: Sammamish High School’s Free and Reduced Lunch Population Table 2: Teacher Population by Year Table 3: Levels of Use (LOU) Table 4: Establishing the Threshold for PBL Implementation and PBL Readiness Table 5: Sammamish High School Professional Learning Infrastructure Table 6: Measuring Relevance at SILT Table 7: Teacher Participation in SILT Table 8: Fidelity Rating: SILT Table 9: SILT Data Collection Summary Table 10: Overall SILT Effectiveness of Relevance Table 11: Teacher Design Team Data Summary Table 12: Instances of Sharing in AP Human Geography Design Team Meetings Table 13: Instances of Sharing in the Junior English Design Team’s Satire Unit Table 14: Instances of Sharing in Geometry Design Team Meetings Table 15: Overview of Teacher Leaders, Department Affiliation, Major Responsibilities, and FTE Table 16: Advisory Board Meetings Table 17: Courses Targeted for PBL Redesign Table 18: Description of Cohort and Graduation Year Table 19: Comparison of Mean Number of PBL and All Courses Taken by Cohort Table 20: Description of PBL Dosage by Cohort Table 21: Sammamish High School Changing Demographics Over Time Table 22: Number of Students in Each Cohort Matched by Number of SHS Years and AP Test Table 23: AP Tests taken by Course Table 24: Statistically Significant Gains in AP Scores by Course Table 25: Statistically Significant Gains by Students Who Speak a First Language Other Than English at Home (EngNotFirst) in Mean AP Scores by Course Table 26: Statistically Significant Gains by Students Who Receive Free and Reduced Lunch (FRL) Services in Mean AP Scores by Course Table 27: Statistically Significant Gains by Students with Disabilities (SWD) in Mean AP Scores by Course Table 28: Comparison of Mean AP Scores Between All Students in Exploratory Study #1: Mean AP Scores