A Great Evaluator For Every Teacher

Spring 2012

5 Ways to Ensure That Teacher Evaluations Are Fair, Reliable, and Effective

Introduction: We are a group of teachers from Greater district Our 5 Recommendations and charter schools who have come together to ensure that teachers’ perspectives are included as our state implements changes to its teacher evaluation system. We 1. Ensure that teachers understand the know that policy leaders in and across the purpose and structure of the evaluation country are currently rethinking teacher evaluation and system before they are evaluated. we have specifi c suggestions to off er. 2. Require robust training of all evaluators. We support many of the principles underlying the new evaluation system. It has the potential to identify and 3. Require rigorous certifi cation & celebrate high-performing teachers, as well as identify, recertifi cation for all evaluators. support, and potentially dismiss teachers who are repeatedly deemed ineff ective. Th e new evaluation system 4. Ensure that evaluators have the capacity, also provides teachers with regular opportunities to time, and support necessary to complete their receive feedback and improve their practice. All of this evaluations. will benefi t our most important constituent: our students. 5. Collect and use teacher feedback throughout the evaluation process. However, it is important that the new evaluation regulations are implemented consistently and thoroughly to allow for all teachers to experience the potential benefi ts. We have implementation and are not benefi tting from the intended surveyed 112 teachers from across Massachusetts who purpose of the new system. We want the new system to have already experienced the new evaluation system this succeed, both for the sake of teachers and of students. school year (those in Level 4 or “Early Adopter” schools). We recognize that our data refl ect a relatively small sample of teachers, but we still see great value in sharing Th e recommendations that follow are based on our survey the results because they provide a window into teachers’ results, as well as recent research on teacher evaluation real-time experiences. And, they provide more collective and our own diverse experiences with evaluation. teacher-level data than we have seen anywhere else. Much of our focus is on the quality of evaluators. We feel Th e data show that there is a high degree of variability: that this is an essential component of a fair, reliable, and many teachers describe their experience as frustrating eff ective evaluation system that is not garnering much and ineff ective, many others paint a more neutral picture, attention right now. We welcome greater accountability for and some are quite positive (even given that this year teachers based on whether or not our students are learning is essentially a pilot year). We have concerns for the from our instruction; however, we implore policymakers large number of teachers who are experiencing weak to hold equally high expectations for evaluators. Our Recommendations: A poor, confusing introduction of the evaluation system has the potential to undermine teachers’ buy-in to that system. Indeed, a National Institute for Excellence in Teaching study on teacher evaluation in Chicago noted 1. Ensure That Teachers Understand that an eff ective evaluation system “requires much more the Purpose and Structure of the than just new evaluation tools, procedures, and training. Evaluation System Before They Are It requires deliberate and ongoing eff orts to help teachers Evaluated understand why professional standards have to be raised and how they and their students will benefi t and be supported, etc.”i Th e new Massachusetts educator evaluation system is detailed and complex. Teachers must have a reasonable Table 2. Overall, how satisfied were opportunity to learn about the purpose and structure of you with your introduction to the the new evaluation system before they are evaluated. new evaluation system?

Unfortunately, our recent survey of Massachusetts Very Satisfi ed 5 2% teachers in Level 4 schools and Early Adopter districts suggests that this has oft en not been the case. Our survey 4 20% data found that: 3 29%

• 38% of teachers surveyed had no knowledge of the 2 28% new evaluation system until it was implemented in their schools in the Fall of 2011. (Table 1) Not Satisfi ed 1 22% • When asked to rate their satisfaction regarding their introduction to the new evaluation system on a 1-5 06 1218 24 36 n=112 scale (1 = very dissatisfi ed, 5 = highly satisfi ed), 50% Number of Teachers of teachers selected a rating of 1 or 2. 22% selected a rating of 4 or 5. (Table 2) In order to ensure that teachers understand the purpose Table 1. When did you first learn and structure of the evaluation system before they are about the new evaluation system? evaluated, we recommend that the state and districts:

• Provide a clear, coherent introduction to the Winter 2010- 27% evaluation system that focuses upon its most 2011 important aspects – such as its purpose, the evaluation cycle, and the evaluation rubric. Spring 2011 21% • Ensure that school administrators understand the purpose and structure of the new evaluation system Summer 14% before it is introduced in their schools. 2011 • Ensure that teachers have a reasonable opportunity Fall 2011 38% to ask questions about the evaluation system during and/or aft er the introduction. 0102030 40 50 • Help teachers understand the rubric by providing n=112 Number of Teachers model videos with accompanying rubrics.

Several respondents described their introduction to the • Use end-of-session surveys to collect teachers’ new evaluation system as a frustrating experience. One questions and comments and assess their respondent explained, “Administrators were unclear about understanding of the evaluation system. Create the new evaluation themselves, so they could not present plans to follow up if those surveys suggest further it clearly and eff ectively.” Another noted that “there was explanation and clarifi cation is needed. a great deal of confusion in our building surrounding the new evaluation system.”

2 Teachers rated the effectiveness 2. Require Robust Training of all of the following ways they were Evaluators introduced to the new evaluation system: Evaluators must have a clear and thorough understanding Table 3. School-based Meeting of the evaluation regulations before they are allowed to evaluate teachers. A poor evaluator undermines the reliability and eff ectiveness of the entire evaluation Not Applicable 10% process. Unfortunately, our survey data suggests that many teachers view their evaluator as unprepared or unable to perform their job well as evaluator. Very Eff ective 7%

Eff ective 29% • While 59% of survey respondents gave their evaluators an overall rating of good or excellent, Somewhat 41% gave their evaluators an overall rating of fair or 38% poor. (Table 6) Eff ective Not Eff ective 17% • While 52% of respondents rated their evaluator’s content area knowledge as good or excellent, 45% rated it as poor or fair. (Table 7) 0816 24 32 40 48 n=112 Number of Teachers • While 60% of respondents rated the quality of their Table 4. Districtwide Meeting evaluator’s feedback as good or excellent, 35% of respondents rated it as fair or poor. (Table 8)

Not Applicable 54% • 34% of respondents rated their evaluator’s understanding of the evaluation process as poor or Very Eff ective 3% fair. (Table 9)

Eff ective 17% We must ensure that evaluators are fully trained – and this will take time. Research conducted by the Teacher Somewhat Advancement Program (TAP) suggests it may take up 13% Eff ective to 40 hours to properly train an evaluator.ii Th e Gates Measures of Eff ective Teaching (MET) project advises Not Eff ective 14% 17-25 hours of training.iii Th ese robust evaluator training programs help to ensure that all evaluators – not just some 01224 36 48 60 – are reliable and eff ective. n=112 Number of Teachers Table 6. Overall, how would you rate Table 5. Individual Meeting With your primary evaluator? Administrator or Mentor

Not Applicable 44% Excellent 21%

Very Eff ective 10% Good 38%

Eff ective 18% Fair 23% Somewhat 21% Eff ective Poor 18% Not Eff ective 8% 0102030 40 50 20 50 010 30 40 n=112 Number of Teachers n=112 Number of Teachers “I truly believe that this evaluation procedure Teachers were asked to rate their could improve my teaching; I know that I am primary evaluator in the following not reaching all of my students at the mo- areas: ment. However, when I had a diffi cult time writing my goals, my evaluator did not seem Table 7. Knowledge in the Content to understand how to write these much bet- ter than I did. I felt if I had written better goals Area I could have seen more success this year. I have colleagues who had a different evaluator Poor 18% and they seemed to benefi t a great deal from the process” Fair 27%

-Teacher from a Massachusetts Level 4 Good 32% School Excellent 20% “Everything I presented for my plan, my evalu- ator could not approve it without asking and Not Applicable/ Not Sure 4% sending for outside help. I am very interested in improving my practice, and I feel this is do- 07 1421 28 35 42 n=112 ing the opposite.” Number of Teachers -Teacher from a Massachusetts Level 4 Table 8. Quality of Feedback School Poor 19%

We recommend that a robust evaluator training program Fair 18% in Massachusetts should: • Ensure that evaluators have an accurate Good 32% understanding of the new evaluation system – especially its purpose and the steps in the evaluation cycle. Excellent 28% Not Applicable/ • Norm evaluators’ scoring by providing model Not Sure 5% videos and rubrics. 07 1421 28 35 42 n=112 • Consider creative, cost-eff ective options for Number of Teachers comprehensive training, including both on-line and in-person training. Table 9. Understanding the Year-Long Evaluation • Provide evaluators with ongoing opportunities to network and communicate with each other. Poor 13%

Finally, one critical – but frequently overlooked – Fair 21% component of the evaluation system is the relationship between the evaluator and the person being evaluated. Good 36% Trust is a foundation for that relationship to fl ourish. In order for this relationship to “work,” the teacher must trust that the evaluator has been trained thoroughly and Excellent 25% has knowledge and skills that will inform the teacher on how to improve his or her practice. Not Applicable/ 4% Not Sure 07 14 21 28 35 42 Th us, a robust evaluator training program must prepare n=112 evaluators to do more than just use the evaluation tools Number of Teachers

4 – it must prepare them to develop caring, supportive One respondent to our survey noted, “Th is evaluation relationships with educators. As Anthony Bryk, President is totally subjective. What one evaluator sees is not what of the Carnegie Foundation for the Advancement of another sees.” Teaching, recently noted in his research on trust in schools, “It must be made clear what it means to mentor well.”iv It is essential that we reverse this trend. We must acknowledge that evaluating classroom teaching can be as complex as teaching itself. Th us, as part of the certifi cation 3. Require Rigorous Certification & and recertifi cation process, we must: Recertification for All Evaluators • Use model videos and/or other performance-based assessments to certify that evaluators can evaluate While initial training for evaluators is essential, training teaching reliably and eff ectively. alone is not enough to ensure that evaluators are reliable and eff ective. Evaluators must also pass certifi cation • Create clear expectations for evaluators related to and recertifi cation assessments to demonstrate their mentoring and providing feedback and ensure that evaluators meet these expectations. competency. A rigorous certifi cation and recertifi cation system is necessary to ensure that evaluators are eff ective • Hold evaluators accountable for their work even and reliable, and thereby foster trust between evaluators aft er they have been certifi ed by requiring periodic and teachers. recertifi cation. Th e recertifi cation process could include a variety of: Our survey data suggests that even under the new -feedback from teachers with whom the evaluation regulations, some Massachusetts teachers are evaluator has worked. rarely being observed, and when they are observed, they -additional videos and/or other frequently do not trust the feedback they receive. Our performance-based assessments to data revealed: ensure that evaluators are still reliable and eff ective. • In schools required to formally evaluate at least half -a portfolio of the evaluator’s work of educators in year one of implementation, 18% of which can be submitted for recertifi cation. teachers were not observed at all this year and 24% were observed only once. (Table 11) 4. Ensure That Evaluators Have • 39% of teachers rarely or never use the feedback given by evaluators. Th at number increases to 56% the Capacity, Time, and Support if teachers who incorporated feedback just once a Necessary to Complete Their month are included. (Table 13) Evaluations Table 10. Rate your evaluator’s availability to meet with you. An eff ective evaluation system must ensure that evaluators have the capacity, time, and support necessary Poor 10% to fulfi ll their responsibilities. Our survey results suggest that evaluators are primarily principals and assistant principals, and that they oft en lack the time and/or Fair 25% support to complete their evaluations. More specifi cally, our survey found: Good 30% • Th e Massachusetts model system suggests that each Excellent 32% teacher should have one announced observation and multiple brief unannounced observations with feedback between October and May. However, Not Applicable/ 3% Not Sure more than half way through this fi rst year, 42% of teachers in our survey had been evaluated only 0 or 07 1421 28 35 42 1 time. (Table 11) n=112 Number of Teachers Table 11. How many times have you been observed this school year? • 85% of teachers reported that their observations were performed only by their principals or assistant principals. 0 18% Th e data above suggest that it may be extraordinarily 1 24% diffi cult for many principals and assistant principals to complete high-quality evaluations for all teachers while 2 11% still fulfi lling their other important responsibilities. One respondent to our survey noted that “time seems to be the limiting factor in the process.” Another shared that her 3-4 30% evaluator “barely has time to meet with me and give me feedback on the class he visited.” Th is may be a particularly 5+ 18% salient concern for large, comprehensive high schools.

07 1421 28 35 n=112 We strongly advise that districts use individuals from a Number of Teachers variety of roles as evaluators. Besides principals and Table 12. Who observed you? assistant principals, evaluators could include:

Principal 38% • School-based Master Teachers

Assistant 47% • Teacher Mentors Principal • District-based Peer Evaluators Department 28% Head • Instructional Coaches

Teacher 19% • Department Heads

Other 29% All qualifi ed evaluators should be assigned a manageable number of teachers so that they can dedicate the 01122 33 44 55 appropriate time necessary to complete high-quality n=112 evaluations. Furthermore, we encourage Massachusetts Number of Teachers and its districts to explore other creative ways to address Table 13. How often have you the amount of time needed to complete teacher evaluations reliably and eff ectively. For example, in-person classroom incorporated evaluator feedback into observations can be supplemented with the use of digital your teaching? video recordings of teacher instruction which can be viewed outside the school day. Daily 20% 5. Collect and Use Teacher Feedback Weekly 25% Throughout the Evaluation Process

Monthly 17% Teachers themselves have a wealth of knowledge concerning the eff ectiveness of their evaluators. Th ey Rarely 26% are the only ones who will interact with evaluators during every stage of the process, and as such, are the only people with a global view of how well evaluators Never 13% are doing their job. Our survey contained 16 questions and took approximately 5-10 minutes to complete. Only 06 12 18 24 30 a small portion of this survey directly concentrated on n=112 Number of Teachers evaluator performance, and yet, we garnered information

6 about evaluators’ frequency of visits, quality of feedback, A Final Word dedication to improving teacher instruction, availability to meet with teachers, and understanding of the year-long process. Th e new Massachusetts evaluation system has the potential to transform the teaching profession and improve student achievement throughout the state. However, our survey Data collected at a Boston T+ Network event held data suggests that unless more is done to ensure that all in the fall of 2011 suggests that many teachers evaluators are reliable and eff ective, it will fail to live up are willing to provide feedback to their school to that potential. Some good evaluators are not enough leaders. – all evaluators must have the support, training, and certifi cation they need to ensure that all students have the great teachers they deserve. Data from that event showed that: • Teachers are amenable to evaluating their principals 2-3 times a year. To get in touch with us, please contact Maria Fenwick, • 54% of teachers would spend up to 60 Teach Plus Boston Executive Director, at minutes a year evaluating their principals. [email protected]. • 58% of teachers favored a combination of evaluation methods including online surveys, focus groups, and to a lesser extent, paper –based surveys. Methodology Th e survey data included in this memo was collected Ninety-three Greater Boston teachers were from 112 teachers who teach in Level 4 or Early Adopter present at the event. schools across Massachusetts. We designed the survey ourselves with input from a Strategic Data Fellow from Harvard’s Center for Education Policy and Research. We disseminated the survey via email to as many teachers in Th is type of feedback is indispensible to support evaluators’ Level 4/Early Adopter schools as possible. professional development, as well as to hold evaluators accountable for their work. It also values teachers as professionals who can provide important feedback to References: their co-workers in the education fi eld. i Jerald, Craig D. and Van Hook, Kristin. (2011). More than Measurement: Th e TAP System’s Lessons for Designing Better We recommend: Teacher Evaluation Systems. Chicago, IL: National Institute for Excellence in Teaching. • Teachers should be called upon to give formal ii. Ibid. feedback related to their evaluator’s ability to iii. Bill and Melinda Gates Foundation, Measures of Eff ective Teaching provide eff ective and reliable evaluations. Project. (2012). Gathering Feedback for Teaching: Combining High- Quality Observations with Student Surveys and • Teachers should be asked to give feedback Achievement Gains. on multiple dimensions of evaluation. Th ese dimensions could include: iv Bryk, A., Harding, H., and Greenberg, S. Contextual Infl uences on -Quality of feedback Inquiries into Eff ective Teaching and Th eir Implications for Improving Student Learning, Spring 2012, Harvard Educational Review -Ability to support teachers Volume 82, Number 1 / Spring 2012. -Availability to meet -Frequency of observations -Ability to connect teacher to resources -Depth of pedagogical knowledge -Depth of content knowledge -Knowledge of the evaluation system

Th is feedback should be used as one part of the evaluator’s overall job performance evaluation. Greater Boston 2012 Teaching Policy Fellows

Aadina Balti – Mason Pilot Elementary School, Roxbury Nicholas Churchill – UP Academy, South Boston Jennifer DiSarcina- Eliot K-8 School, Boston *Christopher Doss – , Boston Erin Dukeshire – Orchard Gardens K-8 Pilot School, Roxbury Nicholas Gesualdi – Orchard Gardens K-8 Pilot School, Roxbury Lisa Goncalves – Blackstone Elementary, Boston Riana Good – Boston Teachers Union Pilot School, Katie Gribben – Cambridgeport School, Cambridge Audrey Herzig – Manning Elementary School, Jamaica Plain Phuong Luong – Haggerty Elementary School, Cambridge *Marta Magnus – Wang Middle School, Lowell Lillian Marshall – , Dorchester *Suzanne McGlone – Orchard Gardens K-8 Pilot School, Roxbury Michelle Morrissey – Academy of the Pacifi c Rim, Hyde Park *Deb Opar – Prospect Hill Academy Charter School, Somerville Najia Polad – Mattahunt Elementary, Mattapan Geoffrey Rose – Dever-McCormack K-8 School, Dorchester Gene Roundtree – Madison Park Technical Vocational High School, Roxbury Xavier Rozas – The English High School, Jamaica Plain Emily Salander – Excel Academy Charter School, East Boston Kelly Tynan – UP Academy, South Boston Irischa Valentin – Dever-McCormack K-8 School, Dorchester Andrew Vega – Orchard Gardens K-8 Pilot School, Roxbury *Jeff Vogel – Prospect Hill Academy, Somerville Rebecca Westlake – Phoenix Charter Academy, Chelsea

Maria Fenwick, Teach Plus Greater Boston Executive Director Heather Peske, Teach Plus National Director of Programs

*Denotes lead author