The School District of Philadelphia The Office of Research and Evaluation Race to the Top The Educator Effectiveness Project Year 2 September 2014 The School District of Philadelphia Race to the Top The Educator Effectiveness Project, Year 2 Prepared for: The Office of Human Resources Naomi Wyatt Chief Human Resources Officer Tennille Peeler Program Director, Race to the Top By: The Office of Research and Evaluation James Jack, M.S. Research Assistant Katherine Stratos, M.S. Senior Research Associate Tonya Wolford, M.Ed., Ph.D Deputy September 2014 Contents EXECUTIVE SUMMARY ........................................................................................................................................... 1 BACKGROUND AND INTRODUCTION.................................................................................................................... 10 RACE TO THE TOP GRANT .............................................................................................................................................. 10 PENNSYLVANIA CONTEXT AND LEGISLATION....................................................................................................................... 11 LOCAL CONTEXT AND SDP PROGRAM DESIGN ................................................................................................................... 18 EVALUATION RESEARCH ...................................................................................................................................... 22 RESOURCES ................................................................................................................................................................. 22 TEACHER OBSERVATIONS AND OVERALL RATINGS ............................................................................................................... 23 Aligning Professional Growth System with new Act 82 mandate ....................................................................... 23 Identifying and Implementing a New Observation Rubric .................................................................................. 25 Identifying the Summative Rating Score for Temporary Professional Employees .............................................. 25 Incomplete or Missing Observation Data............................................................................................................ 26 Incorporating Building Level Data into Teacher Ratings ..................................................................................... 27 Summative Evaluation Results ............................................................................................................................ 28 TRAINING ON MULTIPLE MEASURES RATING SYSTEM AND DANIELSON FRAMEWORK ................................................................ 32 Principals ............................................................................................................................................................. 33 Teachers .............................................................................................................................................................. 33 Non-Teaching Professionals ................................................................................................................................ 36 All Educators ....................................................................................................................................................... 37 ESTABLISH STUDENT LEARNING OBJECTIVES (SLOS) ............................................................................................................ 38 District-wide SLO ................................................................................................................................................. 39 Teacher-created SLOs ......................................................................................................................................... 40 ESTABLISH PROCESS FOR PERFORMANCE IMPROVEMENT PLAN (PIP) ..................................................................................... 44 ESTABLISH PVAAS AND TEACHER SPECIFIC DATA PROCESSES ............................................................................................... 45 EVALUATION MONITORING SOFTWARE ............................................................................................................................ 48 INTER-RATER AGREEMENT AND RELIABILITY ...................................................................................................................... 50 IMPLEMENTATION SUPPORT STAFF .................................................................................................................................. 53 DISCUSSION ......................................................................................................................................................... 55 IMPLEMENTATION CONCERNS ......................................................................................................................................... 55 CAPACITY ISSUES .......................................................................................................................................................... 59 POSITIONING IMPLEMENTATION OF THE EDUCATOR EFFECTIVENESS SYSTEM ............................................................................ 61 CONCLUSION ....................................................................................................................................................... 62 WORKS CITED ...................................................................................................................................................... 63 APPENDIX A ......................................................................................................................................................... 65 APPENDIX B ......................................................................................................................................................... 73 APPENDIX C ......................................................................................................................................................... 81 APPENDIX D ......................................................................................................................................................... 89 i APPENDIX E ......................................................................................................................................................... 97 APPENDIX F ....................................................................................................................................................... 102 APPENDIX G ....................................................................................................................................................... 105 APPENDIX H ....................................................................................................................................................... 106 APPENDIX I ........................................................................................................................................................ 107 APPENDIX J ........................................................................................................................................................ 111 APPENDIX K ....................................................................................................................................................... 118 APPENDIX L ........................................................................................................................................................ 122 ii Executive Summary Background The primary objective of Pennsylvania’s Race to the Top (RTTT) Local Education Agency (LEA) Grant program is for participating local entities to adopt and implement Pennsylvania’s Educator Effectiveness System and use the evaluation process and results to inform local decisions in support of student achievement. The School District of Philadelphia (the District) applied for and received $11,112,128 for a three-year period (2012-2013, 2013-2014, 2014-2015) to adopt and implement the new Educator Effectiveness System. Methods The District’s Office of Research and Evaluation (ORE) conducted an implementation study (formative) and an impact evaluation (summative) of the RTTT work in 2013-2014. Data sources include program documents, observations, feedback surveys, as well as observation and evaluation rating data. In Year 2, the evaluation focused on how and to what degree the District has been successful in implementing the first stage of the new Educator Effectiveness System, and how the District is working to prepare for later stages of implementation. Findings 1. Through the first two years of the grant, the District has expended less than one-third of its award amount, and used RTTT funds as follows: Year 1: $1,171,413.10 Year 2: $2,135,491.99 Total Amount Years 1 and 2: $3,306,905.09 Amount Left to be expended in Year 3: $7,805,222.91 For the first two years, the top three expenditures were school computer equipment ($730,871.70), professional education and/or consulting services ($723,383.90), and expenses related to management and administrative costs ($710,171.70). According to the Pennsylvania Department of Education (the State), all funds related to the RTTT grant must be spent by June 30, 2015 (Pennsylvania Department of Education, 2012b). Any funds not obligated (e.g.