MATENEK - TEACHER PROFESSIONAL DEVELOPMENT Final Report Friday, November 22, 2019
Total Page：16
File Type：pdf, Size：1020Kb
MATENEK - TEACHER PROFESSIONAL DEVELOPMENT Final Report Friday, November 22, 2019 Table of Executive Summary 3 Results and policy lessons 3 1. Introduction 4 1.1 Document purpose ……………………………………………………………………………………………………………………………….4 1.2 Activity background…………………………………………………………………………………………………………………………….. 4 The importance of ‘effective teachers’ 4 Bilateral education reforms 5 Shifting to job-embedded professional development 5 Leveraging digital technologies 6 1.3 Matenek program design overview ……………………………………………………………………………………………………..6 2. The Matenek pilot 8 2.1 Program design ……………………………………………………………………………………………………………………………………8 1. Formative research 8 2. Product development research 9 3. Rapid prototyping and user-testing 9 4. Identifying appropriate professional learning and development content 10 2.2 Program Implementation …………………………………………………………………………………………………………………….11 Ensuring fidelity of implementation 11 Advantages and disadvantages 12 Challenges 13 3. Monitoring, evaluation and learning 14 3.1 Methodology……………………………………………………………………………………………………………………………………… 14 Difference-in-differences 16 3.2 Pilot population and sample ……………………………………………………………………………………………………………….16 Teachers and school leaders 16 3.3 Data collection methods ……………………………………………………………………………………………………………………..17 Teachers 17 School leaders 17 MoEYS staff 18 4. Findings 19 Outcome 1: More teachers are knowledgeable about content and pedagogy, which results in a change in their classroom practice.………………………………………………………………………………………………………………………….. 19 Outcome 1.1 Preparation for class 19 Outcome 1.2 On-pace delivery of lessons and subjects 24 Outcome 1.3 Fidelity of lesson implementation 25 Outcome 1.4 Content and pedagogy skills 26 Outcome 1.5 Attitudes and behaviours 28 Outcome 2: More school leaders are able to understand teachers’ needs and provide better support ……29 Outcome 2.1 Understanding of needs and support 29 Outcome 3: More MoEYS staff understand, value, and use Matenek and other ICTs. ………………………………..30 Outcome 3.1 MoEYS understand and value using ICTs and Matenek to achieve education outcomes 30 Outcome 3.2 MoEYS use of ICTs and Matenek to achieve education outcomes 32 Outcome 3.3 MoEYS feel confident using ICTs and Matenek to achieve education outcomes 33 5. Cost effectiveness 34 Value for Money 36 6. Pathways to scale 39 Model 1: Grade One expansion plus ‘over-the-top’ 39 Model 2: Grade 1 & 2 and content expansion 40 Model 3: Eskola integration & Grade 1 Expansion 40 Potential barriers to scaling 40 Beyond 2020 & Exit Strategy 41 7. Conclusion 42 8. Appendices 43 Matenek - Final Report - Catalpa International Ltd 2 Executive Summary Matenek is a digital learning platform and program designed to improve student outcomes by improving teacher abilities. When teachers learn more, their students do too. Matenek is a first-of-its-kind program and web platform for Timor-Leste, built through a successful collaboration with the Ministry of Education, Youth and Sport (MoEYS). It is aligned with the National Teaching Curriculum and Australian Embassy goals. Matenek was designed specifically for teachers with low digital literacy and limited access to reliable internet. Matenek’s design and launch happened against the background of a profound national education reform. This reform is rolling out a new child-centred primary education curriculum and changing the language of instruction. Matenek aims to support the MoEYS’ effort to implement this reform, especially the curriculum roll-out. The Matenek design took place from April 2018 to February 2019. Implementation of the pilot program, involving 14 teachers across 14 schools and over 500 students, is running from March - December 2019. Results and policy lessons Impact on education: Teachers used the Matenek program to prepare for class for 45 minutes each day, more than double the amount of preparation time pre-pilot. In addition to usage, initial research suggests that this converted into tangible and observable changes in their classroom behaviour, improving teacher effectiveness and enabling them to implement the national curriculum with greater fidelity. The additional preparation time enabled Matenek teachers to teach more effectively compared to teachers in the control group (second grade teachers not in the program) improving their readiness to teach, student readiness, and preparation of didactic materials. Matenek teachers also taught their lessons more on pace and with more fidelity and implemented more classroom practices and strategies compared to control teachers. This was especially true in the area of responding to student questions, praising both male and female students, and putting student work up on the wall. There was also a reduction in all teachers’ use of verbal and physical violence. The research also showed the success of the pilot’s impact on school leaders and MoEYS staff. School leaders reported Matenek teachers were more prepared for class, and school leaders were better able to see their teaching progress as it was clearly and visually tracked in Matenek. Cost effectiveness: Based on our estimates, Matenek affords the MoEYS a cost effective, efficient, and impactful delivery mechanism for teacher training. These benefits are also seen across distribution and updates to curricular materials and teacher support, when compared to more traditional models. By using Matenek, the MoEYS can save money on training and distribution of materials, as well as save time by making it easier for school leaders to support teachers and track their teaching progress. Recommendations: Catalpa suggests the MoEYS and DFAT work in close partnership to take immediate steps to scale Matenek to more grades, more teachers, or both. This plan for scale should be both financially viable and strategic. Integrating with Eskola should be a path further explored in the future. Matenek - Final Report - Catalpa International Ltd 3 1. Introduction 1.1 Document purpose This report provides an overview of the Matenek program, outlining the inception, design and implementation processes. It presents key implementation findings from the program’s pilot, analysing its impacts from the perspective of teachers, school leaders and the broader MoEYS (specifically those departments engaged in the design and implementation of the program). It also provides a set of recommendations and next steps, followed by appendices. The Matenek program is a partnership between the MoEYS, the Australian Department of Foreign Affairs and Trade (DFAT) and Catalpa International. 1.2 Activity background The importance of ‘effective teachers’ While the importance of good teachers may be obvious, over the past decade education research has begun to quantify the relationship between teacher quality and student outcomes. Research has shown that effective teachers have a substantial impact on increasing student learning outcomes.1 The effect has been measured in Indonesia, in a study that demonstrated that for every 0.29 units of improvement in a teacher’s knowledge, his or her students’ performance will improve by one unit - a threefold return.2 Moving from a low-performing teacher to a high-performing teacher increases student learning dramatically. Research also demonstrates that investing in teachers is more effective than simply decreasing the teacher-student ratio.3 And ultimately, effective teachers have a substantial impact on the long-term well-being of students, affecting not only their academic achievement and how far they will study, but their income once they enter the labour market. “Effective teachers are teachers who combine deep content knowledge, high-quality practices, creativity and empathy to improve student learning today and in their lifelong learning trajectory.” Understanding the crucial role of the teacher is particularly pertinent to the Timor-Leste context. Following independence in 1999, the nature of schools and the function of teachers changed significantly with the departure of Indonesian school teachers, who made up the majority of the workforce. When schools were reopened, most replacement teachers were unqualified. Research conducted in 2016 found that only about 60% of permanently employed primary teachers held a recognised bachelor level qualification, while many contract/volunteer teachers had only completed secondary school.4 1 Evans, David K. and Yuan, Fei. “The Working Conditions of Teachers in Low and Middle-income Countries”. World Bank Background Paper. 2018. Buhl-Wiggers, Kerwin, Jason T., Smith, Jeffrey A., and Thornton, Rebecca. “The Impact of Teacher Effectiveness on Student Learning in South Africa” Working Paper, 2017. Bau, Natalie, and Das, Jishnu. “Teacher Value-Added in a Low-income Country” American Economic Journal, (forthcoming). 2 ibid 3 Ree, Joppe De. “How Much Teachers Know and How Much It Matters in Class: Analyzing Three Rounds of Subject-Specific Test Score Data of Indonesian Students and Teachers” World Bank Policy Research Working Papers, 2016. 4 DFAT Timor-Leste Sector Overview 2019 Matenek - Final Report - Catalpa International Ltd 4 Understandably, student learning outcomes remained low — in a 2009 Early Grade Reading Assessment of students’ reading ability, the Government of Timor-Leste found that 70% of Grade 1 students were not able to read a single word of the national language of Tetun5. The 2011 Early Grade Mathematics Assessment found that only one-third of students in Grade 1 were able to solve simple additions6. Bilateral education reforms To address this and other learning challenges,