Evaluation of Formative Continuous Assessment Pilot in Amhara and Tigray Regions

Evaluation of Formative Continuous Assessment Pilot

in Amhara and Tigray Regions

September 2018

A partnership with American Institutes for Research

Contract No. AID 663-C-15-00001

USAID Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation (READ M&E)

Evaluation of Formative Continuous Assessment Pilot in Amhara and Tigray Regions

Main Authors: Belay Tefera, Belen Mekonnen, Daniel Tefera, Solomon Areaya Additional Contributors: Mohammed Elmeski, Rebecca Stone, Mariela Taylor

Prepared by: American Institutes for Research 1000 Thomas Jefferson Street, NW Washington, DC 20007

Submitted to: U.S. Agency for International Development/Ethiopia Addis Yigzaw, COR

Under Contract No. AID-663-C-15-00001

Disclaimer: This report was made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the American Institutes for Research (AIR) and do not necessarily reflect the views of USAID or the United States Government. Table of Contents

Executive Summary ...... 1 Introduction ...... 1 Findings and Discussions ...... 1 Conclusions and Recommendations ...... 4 I. Introduction ...... 5 1.1. Background ...... 5 1.2. The FCA Project ...... 5 1.3. Description of FCA Tools ...... 6 II. Evaluation Methods ...... 10 2.1. Research Objectives ...... 10 2.2. Study Design ...... 10 2.3. Sampling of Respondents ...... 12 2.4. Data Collection Procedures ...... 13 2.5. Coding and Data Analysis ...... 14 2.5.1. Qualitative Data Analysis ...... 14 2.5.2. Quantitative Data Analysis ...... 14 III. Findings and Discussions ...... 14 3.1. FCA Tools: Availability, Training and Use ...... 14 3.1.1. Tool Availability and Personnel Training ...... 15 3.1.2. Use of FCA Tools ...... 17 3.1.3. Assessment, Record-keeping, Feedback, Follow-up, and Remedial Measures ..... 20 3.2. Contributions of FCA Training and Tools ...... 26 3.2.1. Teachers’ Improved Understanding of FCA’s Role ...... 27 3.2.2. Changes in Students’ Learning and Achievement ...... 28 3.2.3. Cascading and Sustainability...... 30 3.3. Challenges, Opportunities and Measures ...... 31 3.3.1. Challenges ...... 31 3.3.2. Opportunities ...... 34 3.3.3. Participants’ Suggestions for Enhancing FCA ...... 36 IV. Conclusions and Recommendations ...... 39 4.1. Conclusions ...... 39 4.2. Recommendations ...... 40 References ...... 42

Annexes

Annex 1: List of Names of Regions, Woredas and Schools ...... 45 Annex 2: Common Themes of the Interview Protocols ...... 48 Annex 3: Tool 1: Classroom Observation Guide for READ M&E’s FCA ...... 49 Annex 4: Tool 2: Classroom Observation Checklist for READ M&E’s FCA Intervention; for grades 3 &4 54 Annex 5: Tool 3: Group discussion with Teachers ...... 59 Annex 6: Tool 4: Group Discussion Questions for Supervisors and Principals, ...... 61 Annex 7: Tool 5: Interview Guide for Woreda Education Experts, who took the FCA training delivered by READ M&E ...... 63 Annex 8: FCA feedback communication form ...... 65

Tables

Table 1: Key Evaluation Questions ...... 11 Table 2: Sampling of Zones, Woredas, and Schools ...... 13 Table 3: Availability of FCA Tools in Classrooms, Grades 1 to 4 ...... 15 Table 4: Availability of FCA Tools in Classrooms, Grades 3 and 4 ...... 15 Table 5: Number of FCA Trained Teachers in First Cycle Primary Schools ...... 17 Table 6: Use of FCA Assessment Tools in Classrooms, Grades 1 and 2 ...... 18 Table 7: Use of FCA Assessment Tools in Classrooms, Grades 3 and 4 ...... 18 Table 8: Types of Skills Assessed by FCA Tools in Grades 1 and 2 ...... 20 Table 9: Types of Skills Assessed by FCA Tools in Grades 3 and 4 ...... 21 Table 10: Use of Record-keeping Tools during Observation of Tigrigna and Amharic Speaking Classrooms ...... 21 Table 11: Teachers’ Feedback during Classroom Observations ...... 23 Table 12: Teacher Remedial Strategies Used during Classroom Observation ...... 24

Figures

Figure 1. Sample Afaan Oromo flashcards ...... 7 Figure 2. Example of KWL ...... 8 Figure 3. Example of Mind Map ...... 8 Figure 4. Example of Web Map ...... 9 Figure 5. Example of sequential graphic organizer ...... 9

Acronyms

AIR American Institutes for Research AMPS Amhara Principal/ Supervisor AMT Amhara Teacher AMWE Amhara Woreda Office Expert CA Continuous Assessment EGRA Early Grade Reading Assessment ESDP Education Sector Development Program FCA Formative Continuous Assessment KWL Know, Want to Know, Learned MoE Ministry of Education MT Mother Tongue MTL Mother Tongue Language TGPS Tigray Principal/ Supervisor TGT Tigray Teacher TGWE Tigray Woreda Office Expert USAID United States Agency for International Development

Evaluation of Formative Continuous Assessment

Executive Summary Introduction Background: The Ethiopian education sector has documented remarkable strides in the last few decades. However, it has experienced equally significant challenges with education quality. These concerns of quality appear to be critically important for education during the formative years, where foundations are laid down for lifelong learning. The National Learning Assessment (NLA) results consistently show that Ethiopian children are unable to read at a grade-appropriate level because they did not learn to read by the time, they complete the first cycle primary school (MoE, 2008; RTI, 2010; Seid, 2015; Belay & Belay, 2018). Different actors have been working to address this problem. READ M&E envisioned addressing the critical need of improving literacy instruction in the early grades by reworking the goals, strategies, and tools of continuous classroom assessment. READ M&E developed and implemented a Formative Continuous Assessment (FCA) pilot that consisted of training principals and mother-tongue (MT) teachers in Grades 1-4 and distributing FCA tools. The pilot project was implemented in Amhara, Oromia and Tigray regions. However, the Oromia regional state education bureau declined to participate in the evaluation because of a dispute it had with the Mother Tongue (MT) textbooks. The pilot also aimed to hand over best practices, achievements, and research results to key regional and national stakeholders so that the project could be sustained.

Purpose: The purpose of this evaluation is to examine the implementation of FCA training and use of FCA tools in Amhara and Tigray regional states. It specifically evaluates the presence of FCA tools delivered to various schools, the involvement of trained MT teachers in FCA implementation, the extent of FCA tool usage for student assessment, contributions of FCA training and tools, challenges experienced in the process, opportunities available for future use and the way forward for FCA scalability and sustainability.

Methods: The study sites were the Amhara and Tigray regional states. The research team collected 112 classroom observations and conducted interviews with 15 woreda education experts, 52 school directors and vice directors, and 52 FCA-trained teachers.

Findings and Discussions FCA Tools Delivered: The FCA tools delivered to Grades 1 and 2 classrooms were A6 flash cards, A4 flash cards, index cards, pocket chart, and letter recognition recording sheet. FCA tools for Grades 3 and 4 classrooms were graphic organizers, writing rubrics, and index cards. Although each classroom is expected to have all the tools that correspond to their grade level, only 23% of classrooms contained all the five tools. Pocket charts were the most commonly available tools and were found in 76.3% of the classrooms, while letter recognition sheets were the least commonly available and found in only 42.4% of the classrooms. Grades 2 and 4 classrooms in Amhara had

Evaluation of Formative Continuous Assessment the least number of each tool. Evidence suggests that in most sites, FCA tools were highly valued, and teachers took the utmost care to protect them from damage, stealing, and related other losses.

FCA Trained Personnel: Almost all of the woreda experts interviewed indicated that they have heard about the FCA training and its implementation through informal conversations with teachers, school reports, or personal encounters during supervision. In contrast, 76% of the observed classrooms had FCA-trained teachers that understood how to use the FCA tools. FCA-trained teachers shared what they learned with untrained teachers and other personnel. While there are few FCA tools available in Grades 2 and 4 classrooms in Amhara, there is a higher proportion of teachers trained in FCA.

Utilization of FCA Tools: FCA tools served a variety of interrelated purposes, including classroom instruction, student assessment, and categorizing students into corresponding skill levels. Almost all interviewed teachers reported that they have been using FCA tools regularly in classroom instruction, assessments, feedback and record-keeping for MT language teaching. This was more common among teachers in Grades 1 and 2. Contrary to teachers’ responses, principals and woreda experts and classroom observations indicate that FCA tools are not consistently applied in all grades and that some teachers were not using these tools. Data collectors observed that teachers used FCA tools more frequently in Grades 1 and 2 than they did in Grades 3 and 4. They also found that A4 and A6 cards were used more frequently than others. Nearly half of the teachers observed used FCA recording forms provided to them during training, such as the writing rubric, letter recognition recording sheet.

Contributions: Researchers found that the FCA intervention enhanced teachers’ understanding of assessment, record-keeping and feedback. Participants agreed that the training has helped teachers understand and practice student-friendly pedagogy, and improved collaborative work among teachers. The intervention’s most striking impact is to shift instruction from summative- focused continuous assessment to formative-oriented approach. That is, instead of relying on general assessments, teachers used individual assessments for monitoring students’ learning progress. They consistently and continuously recorded assessments. Teachers learned how to provide positive, constructive feedback and used proper rewarding techniques. Researchers also noted encouraging changes in students’ learning and achievement, self-confidence, participation, and healthy competition among their peers.

Considering the reported benefits of FCA tools teachers recommended that they should be adapted to other subjects such as English, natural science, and mathematics; to higher grades; and to preschools. They further proposed that the local community should be involved in developing FCA tools by preparing local equivalents and duplicating those missing. FCA-trained teachers trained the untrained MT teachers, teachers of other subjects, teachers from other departments and schools.

Evaluation of Formative Continuous Assessment

Finally, untrained teachers expressed significant interest in the trainings because they believed that the FCA approach was inspirational and attractive.

Challenges: The following major challenges were identified:

▪ Design of FCA tools: Participants reported issues with the tools’ graphics, relevance, grade-appropriateness, and complexity (the detail is discussed in the body of this report). They also noted that there were discrepancies between the drawings and letters, materials, and textbooks. ▪ Teacher-Related Problems: Teachers reported struggling to cover a large amount of content in a short period of time. Additional challenges include teacher shortage, placing teachers in grades and subjects outside of their specialization, and teachers’ behavioral, motivational and attitudinal problems. ▪ Workload: Teachers struggled with a heavier work load, especially those with large classes, because FCA requires individualized assessment, follow-up, and support on a daily basis. ▪ Resources: Respondents reported challenges related to finding space for FCA storage, shortage of FCA tools, and resource constraints such as shortage of paper to duplicate recording sheets and absence of photocopier machine to make copies of some of the tools for use. ▪ Follow-up and Supervision: Local education actors complained that there was not enough follow-up and supervision from principals and local education offices

Opportunities: There are a number of potential opportunities for scaling up and sustaining FCA activities. Decisionmakers can build on the MoE’s policy on continuous assessment; the newly launched teachers’ continuous professional development program; and the Education Sector Development Program V’s (ESDP V) focus on improving education quality at all levels (with a major emphasis on basic literacy skills in the lower grades). In addition, decisionmakers can build partnerships with development organizations that work on education to identify and pursue opportunities that can be deployed in implementing FCA. At the grassroots level, decisionmakers can revise existing textbooks; use locally available resources to produce FCA tools; and foster enthusiasm among teachers and schools about the new FCA training and methods. Finally, the MoE’s new continuous assessment initiative and funders’ interest to support these efforts can be leveraged in favor of FCA.

Enhancing FCA in the Future: Respondents recommended scaling up and sustaining FCA activities. Decisionmakers should provide resources to ensure enhanced use of FCA tools in the future. They also need to address MT teachers’ training needs, assign trained teachers properly, provide opportunities for teachers to share their experiences, strengthen supervision and follow-

Evaluation of Formative Continuous Assessment up procedures, establish collaboration among stakeholders, conduct research and evaluation, and systematize lessons for best practices.

Conclusions and Recommendations Conclusions: Observers could not confirm that the FCA tools were available for use in all of the sampled classrooms. However, in the classrooms where the FCA tools were clearly available, the tools were properly utilized. In cases where FCA tools were not available, local stakeholders made attempts to fill in gaps by transferring knowledge to untrained teachers and creating and reproducing FCA tools. Teachers’ use of FCA training and tools were found to yield very encouraging results. For example, teachers’ motivations and attitudes improved, pedagogical methods have become more student-centered and collaborative, assessment practices have become more instrumental for facilitating learning, record-keeping was continuous and systematic, and providing feedback was a lot more constructive. These changes have translated into encouraging changes in student participations and achievements. The FCA approach reportedly increased student participation and learning achievement. However, these contributions were not without challenges, such as FCA tool quality and quantity, teacher-related problems, the FCA’s time consuming nature, and supervision problems. Researchers identified opportunities to overcome these challenges and effectively implement FCA practices. These opportunities require working with actors at the top policy level, the middle level (such as nonprofit organizations), as well as the schools at the ground level.

Recommendations: FCA needs to be scaled up. Decisionmakers need to determine a strategy to widen the scope and coverage of the FCA training, revise the FCA tools, and ensure that schools are ready to implement FCA activities.

Evaluation of Formative Continuous Assessment

I. Introduction 1.1. Background Ethiopia recognizes the role that education plays in achieving its vision of becoming a middle- income country by 2025. Ethiopia has made huge progress toward achieving this vision by improving access and quality of education in the last few years. Despite this progress, recent data indicate that learning levels remain low, drop out and repetition are still high, and current educational inputs are not being effectively converted into educational gain. For example, the 2018 Early Grade Reading Assessment (EGRA) indicated that 37.2% of students in Grade 2 and 3 were unable to read a single word of grade-level text and that only 6.2% of students demonstrated the ability to read fluently with full comprehension (American Institutes for Research, forthcoming). The National Learning Assessment, conducted as a bench mark for the Education Sector Development Program IV (ESDP IV) plan, also portrayed a gloomy picture. Education stakeholders originally predicted that 70% of grade 4 students will score at least 50%, while 25% will score at least 75%. Achievements were alarmingly low: only 25% of students scored at least 50%, while only 2.3% scored 75% or higher (ESDP V, 2015).

Cognizant of these problems, the Federal Ministry of Education prioritized education quality in its ESDP V program with an explicit emphasis on improving the overall literacy and numeracy level of the school-aged population. According to ESDP V, one of the education sector’s goals for 2015/2016–2019/2020 is to deliver quality education that meets Ethiopian children’s diverse learning needs. More specifically, it promises to focus on the core foundational skills in early grades, as these skills affect children’s ability to learn in all subsequent stages of the education system. As part of these efforts, the Ministry committed to improving teachers’ pedagogical skills related to children’ foundational literacy, numeracy and language skills through continuous professional development.

Various actors and development partners have joined hands with the government to address the need for improved instruction quality; none more so than USAID. The USAID-funded READ M&E project, implemented by the American Institutes for Research (AIR), is currently piloting the Formative Continuous Assessment (FCA) program. The FCA program focuses on improving the teaching and learning of literacy and helping teachers to diagnose, assess and remediate reading challenges. This report is a mini-evaluation of the pilot intervention’s implementation, with the aim of sharing lessons learned and identifying the next steps for ensuring the sustainability and use of FCA tools for assessment purposes. 1.2. The FCA Pilot The Federal Ministry of Education’s (MoE) Education and Training Policy of the Federal Democratic Republic of Ethiopia (1996) clearly stipulates that continuous assessment shall be implemented across all classrooms in the country. The MoE developed continuous assessment

Evaluation of Formative Continuous Assessment manuals1 and provided short-term training and in-service training programs for teachers. This approach demonstrated a significant shift from traditional summative assessment practices that are characteristic of Ethiopian classrooms. This new approach required trainers, teachers, and even students to shift their perceptions of the way that assessment works, because Ethiopian classrooms traditionally do not implement assessment FOR learning; only of assessment OF learning.

Small-scale investigations in Ethiopia have identified many issues with the way continuous assessment has been implemented. For example, continuous assessment is often synonymous with continuous testing (Abiy, 2013), more judgmental than developmental (Aytaged, 2010), inconsistent across different settings (Sintahehu, 2016), less aligned with student learning and curriculum plan (Fisseha, 2010), and does not sufficiently improve the teaching and learning process (Yeheyis & Getachew, 2014). These aforementioned qualities are in line with summative continuous assessment, which has been implemented to the exclusion of other, more instrumental forms of continuous assessment, such as formative continuous assessment.

READ M&E staff began the FCA initiative in 2016 with the purpose of integrating formative continuous assessment with existing summative assessment practices. Since then, READ M&E developed FCA flashcards, small and big pocket charts, graphic organizers and recording sheets. READ M&E trained 1,445 (704 male and 741 female) participants on the use of FCA tools and provided all participants with FCA tools. Training recipients included Mother Tongue Language (MTL) teachers, principals, supervisors, and zone, woreda, and regional education bureau experts. READ M&E also conducted the first round of FCA experience sharing workshop in April 2017. The trainings took place in Tigray, Amhara and Oromia and were attended by a total of 1,385 (532 male and 713 female) participants. READ M&E conducted this assessment as a follow-up to the first round of learning and experience sharing meeting. It also plans to meet and consult with decisionmakers at the federal and regional level to share findings from READ M&E’s FCA activities.

1.3. Description of FCA Tools READ M&E developed and purchased FCA tools. The materials developed and produced include A4 Flashcards, A6 Flashcards, Graphic Organizers (KWL2, Mind Map, Web Map, Context and Connection Map, and Sequence Map), Index Card, Letter Recognition Recording Sheet, and Writing Rubric. READ M&E purchased the Pocket Chart. The A4 and A6 Flashcards were designed to assess identification of letters, sight words, and target letters from a scrambled group

1 The present author has participated in the development of manuals as part of a bigger project organized by the National Organization for Examination, Ministry of Education, Ethiopia to develop different continuous assessment manuals for different subjects and grade levels (see Tigistu, Alemayehu and Belay Tefera, 2004). 2 A reading strategy in which the students list what they Know, what they Want to know, and what they have Learned after reading. 6

Evaluation of Formative Continuous Assessment of letters, recognition of word formation, and segmenting and blending skills of mainly Grade 1 and Grade 2 students.

Figure 1. Sample Amharic flashcards

The Graphic Organizers were designed to assess Grade 3 and Grade 4 students’ critical thinking skills, ability to visualize and organize information, and ability to identify new ways of connecting information. KWL is a chart with three columns designed to assess students’ ability to organize ideas about a topic before, during, and after discussion. The Mind Map visually represents hierarchically ordered information and is designed to assess skills like vocabulary development, concept naming, phrase construction, and sentence construction.

Web Map is one of the Graphic Organizers designed to assess students’ ability to categorize and write main and supporting concepts. Context and Connection Map helps to think in a context and assesses students’ ability to infer meaning from other words, form new words, and draw meaning from the connection. Sequence Map contains a series of boxes and arrows used to record the steps involved in certain activities. Below are some examples of FCA tools developed and produced;

Evaluation of Formative Continuous Assessment

Figure 2. Example of KWL

Figure 3. Example of Mind Map

Evaluation of Formative Continuous Assessment

Figure 4. Example of Web Map

Figure 5. Example of sequential graphic organizer

Evaluation of Formative Continuous Assessment

Students are expected to fill these boxes or arrows with distinct activities that are needed to accomplish a certain task. The Index Card is a reusable, plain A6 card used to assess letter and word recognition and also segmenting and blending skills among Grade 1 and Grade 2 students. Teachers can use this card to write a targeted letter of the week and ask students to identify the letter, or they can write a decodable word and ask a student to segment it.

The Pocket Chart is a plastic chart with many separate pockets to store flashcards or index cards with a word or letter written on it. It is useful way to assess the letter identification, word formation, and sentence construction skills of Grade 1 and Grade 2 students. The other FCA tools are the Letter Recognition Recording Sheet and the Writing Rubric. As the names indicate, these tools were designed to record students’ progress in their learning to read. The Letter Recognition Recording Sheet was designed to be used among Grade 1 students while the Writing Rubric was designed to be used with students in other grade levels. II. Evaluation Methods 2.1. Research Objectives The major objectives of this evaluation of the FCA implementation are to:

▪ Examine the availability and use of FCA tools ▪ Assess the FCA tools’ contributions to the assessment of learning to read ▪ Identify major challenges, potential opportunities and strategies while implementing FCA ▪ Suggest approaches to addressing the challenges, and ▪ Recommend activities to be completed before transferring the project to the MoE 2.2. Study Design To achieve these objectives, this evaluation primarily used qualitative methods to answer the research questions. Table 1 shows the list of research questions, along with the corresponding data collection instruments.

Evaluation of Formative Continuous Assessment

Table 1: Key Evaluation Questions

Key Evaluation Questions

Are the FCA tools distributed to teachers and Classroom observation (grades 1 and 2) supervisors available in the classroom, school, or Classroom observation (grades 3 and 4) 1 are they in the hands of the MT teachers? How trained are the education personnel involved in the Group discussion with teachers use of FCA tools? Group discussion with principals and supervisors Classroom observation (grades 1 and 2) Do MT teachers use FCA tools delivered by READ Classroom observation (grades 3 and 4) M&E for assessment purposes? Which of the FCA 2 Group discussion with teachers tools are frequently used in the classroom and how are they used? Group discussion with principals and supervisors Discussion with woreda experts Classroom observation (grades 1 and 2) How do teachers conduct FCA, keep records, give Classroom observation (grades 3 and 4) 3 feedback and conduct follow up to enhance students' learning? Group discussion with teachers Group discussion with principals and supervisors Classroom observation (grades 1 and 2) What are the remediation strategies frequently Classroom observation (grades 3 and 4) employed by teachers in supporting students with 4 Group discussion with teachers reading problems? What supports are provided to teachers? Group discussion with principals and supervisors Discussion with woreda experts Classroom observation (grades 1 and 2) Classroom observation (grades 3 and 4) What are the perception of teachers, supervisors 5 and woreda expert about the contribution of the Group discussion with teachers FCA tools in improving children's reading ability? Group discussion with principals and supervisors Discussion with woreda experts Classroom observation (grades 1 and 2) Classroom observation (grades 3 and 4) What were the major challenges encountered and 6 Group discussion with teachers opportunities available while using FCA tools? Group discussion with principals and supervisors Discussion with woreda experts Group discussion with teachers What suggestions were given to dealing with the 7 Group discussion with principals and supervisors challenges? Discussion with woreda experts Group discussion with teachers What should be done by READ M&E and the MoE 8 Group discussion with principals and supervisors to properly implement FCA in the future? Discussion with woreda experts

Evaluation of Formative Continuous Assessment

The research team primarily used qualitative instruments to collect narratives that respond to the research questions listed in Table 1. The READ M&E staff developed separate interview protocols for teachers, principals, cluster supervisors, and woreda education office experts. These interview protocols are included in Annex 2. The data collectors primarily conducted individual interviews with woreda experts, principals, and cluster supervisors. They also conducted individual and focus group discussions with teachers, depending on the FCA-trained teachers’ availability. The interviews addressed the availability and utilization of FCA tools, contributions, challenges, opportunities, and suggestions for future use of FCA tools.

The interview protocols were complemented by quantitative data collected from classroom observations, which contain a structured observation checklist. The observation checklist was used to gather information on the frequency on the use of different FCA tools, teachers’ activities in assessing learning, record keeping, giving feedback, conducting follow up, and taking remedial measures. Annex 4 includes the observation checklist.

2.3. Sampling of Respondents The evaluation used multi-level sampling design to collect the data from regions, zones, woredas, and schools from two regional states. Tigray and Amhara regional states were included in the evaluation by default because they were pilot intervention regions. The Oromia regional state declined to participate in the intervention because of a dispute it has with the Mother Tongue (MT) textbooks produced in the region.

The team randomly selected seven zones, three from Tigray and four from Amhara to participate in the pilot intervention. The team included all the pilot implementation woredas in the selected zones in the sampling frame. The number of woredas selected from each region corresponded with the number of implementation sites READ M&E had in each region: eight woredas from Amhara and six woredas from Tigray.

For the third level of sampling, the team consulted with the Zone Education Department and Woreda Education Offices to select the schools. All of the selected schools had trained mother- tongue teachers that taught in Grades 1-4 during the evaluation. READ M&E selected 32 schools from Amhara and 24 schools from Tigray. Table 2 shows the distribution of zones, woredas, and schools selected.

Evaluation of Formative Continuous Assessment

Table 2: Sampling of Zones, Woredas, and Schools

Number of Interventions Proposed Sample Regions Zones Woredas Schools # Zones # Woreda # Schools Tigray 3 3 84 3 6 24 Amhara 8 4 90 4 8 36 Total 11 174 7 14 56

READ M&E selected respondents based on their position and availability during the evaluation. READ M&E secured a list of MT teachers who were trained, received FCA materials and participated in the experience sharing workshop. The team interviewed 151 MT teachers (31 males and 120 females), 52 principals and vice principals, 26 cluster supervisors, and 14 woreda experts. The observed gender imbalance seems to have been emanated from the existing realty of the observed lower primary schools MT teachers distribution.

In addition, they selected 2 MT teachers from each school and conducted classroom observations on a total of 112 MT teachers. Annex 1 contains a list of regions, woreda and school names. This table also contains a list of codes for interviewees and respondents from corresponding sites. These codes are used to quote respondents’ ideas in the findings and discussion sections.

2.4. Data Collection Procedures READ M&E staff identified and recruited 28 experienced data collectors (22 males and 6 females). The technical team organized a two-day intensive training, during which they briefed data collectors and supervisors on the FCA pilot implementation in Tigray, Amhara, and Oromia. The training also included information about the FCA data collection tools, data collection procedures, roles and responsibilities, and the data collection timeline. Participants also learned about the evaluation questions and the sampling strategy. The facilitators emphasized the importance of ensuring data quality. They ensured that the data collectors were well-trained on the use of each evaluation protocol and that they understood expectations around high quality data collection. Data collectors were instructed to use audio recording if participants consented.

The READ M&E team then assigned data collectors to their partners; one data collector was responsible for facilitating the interview, while the other was responsible for taking notes and recording the interview. Six teams were sent to Tigray and eight teams were sent to Amhara, where they interviewed woreda education experts, cluster supervisors, principals and vice principals, and MT teachers. In addition, data collectors conducted 112 classroom observations, each of which took approximately 45 minutes. The classroom observations focused on collecting information about the availability of FCA tools, the ways that MT teachers use FCA tools, the types of feedback that they give, and the remedial measures they use.

Evaluation of Formative Continuous Assessment

2.5. Coding and Data Analysis 2.5.1. Qualitative Data Analysis The qualitative analysis used top-down and bottom-up approaches. The top-down approach included fitting data into themes identified in the research questions. The bottom-up approach included identifying emerging themes from the data.

The researchers integrated similar items and interview questions or themes that are presented in the interview guide and identified unique themes. Nine broad themes were initially identified from the qualitative data and were later restructured into six themes. The interview responses and narratives were classified according to these six themes (see Annex 3). The researchers scanned the descriptions under each heading to ensure text-relevance. Many narratives were identified that didn’t fit into the heading and had to be re-classified within existing themes or classified under new, bigger themes. The team excluded descriptions in which the respondent expressed contradictory views, such as, “It would be ideal if all MT language teachers could take the trainings and if the trainings were only given to language teachers” (AMT-14).3

Next, the team gleaned the narratives to identify important codes within each of the six main themes. The codes were synthesized into sub-themes, which served to structure this report.

2.5.2. Quantitative Data Analysis The research team analyzed quantitative data gathered through observation checklist using Statistical Package for Social Scientists version 21. The research team entered the data via a prepared template and subsequently cleaned it to ensure accuracy. Researchers used descriptive statistical analyses–mostly percentages–to support and triangulate findings from the qualitative data. III. Findings and Discussions 3.1. FCA Tools: Availability, Training and Use This section presents data about FCA tools, personnel training, and teachers’ performances. Data generated from interviewees and classroom observations of grades (1 to 4) were thematically integrated and presented in the following discussions. Themes addressed include the availability of FCA tools in classrooms, personnel trained in FCA, types of skills assessed, recording the assessment results, providing feedback to students, using remedial strategies with students, and teachers receiving follow-up and support.

3 All in-text citations given in parenthesis represent codes of interviewees. The letters ‘AMT’ stand for teacher in Amhara region and ‘AMPS’ stand for principal/ supervisor in Amhara region. In the same way, ‘TGT’ and ‘TGPS’ refer respectively to teacher and principal/ supervisor from Tigray region. 14

Evaluation of Formative Continuous Assessment

3.1.1. Tool Availability and Personnel Training Are the FCA tools distributed to trained teachers and supervisors available in the classroom, school, or in the hands of MT teachers? Interviewees indicated that they received a number of FCA tools during the FCA training. They frequently mentioned tools such as flash cards, graphic organizers (different maps), pocket charts, and recording forms. One principal stated, “This training held last time consisted of the provision of full materials that cost Birr 4,500.00. Every trained teacher has these materials (tools) at hand” (TGPS-17). READ M&E provided each teacher, supervisor, and principal who completed the three-day training with materials that are worth more than 4,500 ETB. The purpose of providing trained principals and supervisors with a package of FCA tools was to capacitate them to use during teachers’ supervision as well as conduct school level training for new and untrained teachers as appropriate.

The FCA tools were distributed in schools with the expectation that each classroom would have all the relevant FCA tools. However, as seen in Table 3, only 12 classrooms in Grades 1 and 2 had all five tools. Similarly, Table 4 shows that only 14 classrooms in Grades 3 and 4 had all three tools. Pocket charts were the most common FCA tools, and were found in 76.3% of the classrooms, while the letter recognition sheets were the least common, and were found in only 42.4% of the classrooms.

Table 3: Availability of FCA Tools in Classrooms, Grades 1 to 4

Availability of FCA Tools in Classrooms, Grade 1 and Grade 2 No. of A4 Letter A6 Flash Index Pocket All Language Grade Classrooms Flash Recognition Cards Card Chart Tools Observed Cards Record Sheet 1 12 11 9 6 11 8 3 Tigrigna 2 14 13 12 10 12 9 5 1 20 14 13 9 16 6 2 Amharic 2 13 5 4 3 6 2 2 Totals 59 43 38 28 45 25 12

Table 4: Availability of FCA Tools in Classrooms, Grades 3 and 4

Availability of FCA Tools in Classrooms, Grade 3 and Grade 4 No. of All Language Grade Classrooms Graphic Organizer Writing Rubric Index Card tools Observed 3 10 6 7 8 4 Tigirigna 4 11 7 8 9 6 3 15 8 10 7 3 Amharic 4 17 4 4 1 1 Totals 53 25 29 25 14

Evaluation of Formative Continuous Assessment

The data also showed that Grade 2 and 4 classrooms in Amhara had the least number of each tool in the classrooms. Index cards were only available in one of 17 Grade 4 classrooms in Amhara.

Respondents also discussed the tools’ availability for classroom use. To ensure that the tools were properly cared for, school administrators and teachers stored FCA tools in different places, such as classroom lockers, school storage rooms, staff lockers, the library, principals’ offices, or even a newly constructed room for this purpose. In most cases, FCA-trained MT teachers seem to have full access and responsibility to the tools. Teachers took measures to properly store the FCA tools because they did not want them to get damaged.

▪ We just don’t leave them in the class after usage because our school has a double shift system, and the tools could get damaged (TGT-20). ▪ Trained teachers were not willing to share the materials due to the fear that others may not use the materials properly (AMPS- 04).

Teachers and principals believed that the FCA tools that they had were important and took measures to protect them. These measures include mending broken windows and fixing locker rooms (AMPS-19: Participant2; AMPS-19; TGPS-25), using substitute tools so that they don’t lose the original FCA tools (TGPS-25), constructing safer rooms (AMPS-25), formally registering the tools (TGT-29), and collecting the tools from teachers who leave the school (AMPS-09). There are also instances of extreme care, which was counterproductive. For example, some teachers locked the tools because the materials are expensive, so they wanted to keep the tools safe (TGPS- 19; TGWE-07). Other teachers lock the tools in the staff lockers and go home (TGPS-25), which prevents them from using the tools that afternoon.

How trained are the education personnel involved in the use of FCA tools? Data from school visits and classroom observations show that 76% of the observed classrooms had teachers trained in FCA and the use of FCA tools (see Table 5). A higher proportion of teachers in Tigray were trained in the use of FCA tools than their counterparts in Amhara. However, two of the fifteen woreda experts interviewed have participated in the FCA training and are familiar with the FCA training and tools (AMWE-04; AMWE-01). Two other woreda experts reported that they were officially contacted to select and send teachers and principals for the FCA training (TGWE-05; AMWE- 02) and yet none was invited from the woreda (TGWE-05). Almost all the woreda experts interviewed confirmed that they have heard about the FCA training and its implementation through informal talks with teachers, school reports, or personal encounters during supervision.

If we compare Tables 3, 4, and 5 we see that there are fewer FCA tools available in Grades 2 and 4 in Amhara classrooms compared to the proportions of teachers trained in FCA in Amhara. This means that teachers are trained, but that they cannot apply what they have learned due to the

Evaluation of Formative Continuous Assessment unavailability of FCA tools. The lack of FCA tools is so pervasive that some teachers dealt with this problem by trying to create their own FCA tools.

Table 5: Number of FCA Trained Teachers in First Cycle Primary Schools

No. of No. of Language Grade Classrooms Teachers Observed Trained 1 12 8 2 14 13 Tigrigna 3 10 10 4 11 8 Total 47 39 1 20 15 2 13 8 Amharic 3 15 9 4 17 14 Total 65 46 Total 112 85

3.1.2. Use of FCA Tools Do MT teachers use FCA tools delivered by READ M&E for assessment purposes? Which FCA tools are frequently used in the classroom and how? Teachers, principals, and woreda experts were interviewed to determine the extent to which teachers have been using the FCA tools for assessment purposes. The three groups’ responses were integrated and a review of the data suggested different themes,4 including tools used and purposes, extent of tool usage, and approaches to using the tools. Data from observation checklist also complemented the interview results.

Purposes of the tools used: FCA is an ongoing process of gathering and interpreting information about student learning to make decisions about what to teach, whom, when, and how (Airasian, 1991; Nitko, 2004). The responses showed that interviewees used the FCA tools to serve a variety of interrelated purposes, including instruction and delivering specific lessons (AMPS-14; TGT- 04; AMT-06), assessing students and categorizing their performance levels (AMPS-14; TGPS-21; TGT-04; AMT-06; TGT-25; AMT-14), and capturing their students’ attention (TGPS-21). The teachers use the tools to identify low performing students, so that they can provide them with appropriate remedial strategies: “I use FCA tools to identify slow learners so that I can plan remediation and interventions” (AMT-06).

4 Ten different sub-themes emerged from the qualitative analysis. Three of these themes (i.e. tools delivered, storage, care for the tools) are presented under Availability, and four of them (Adapting FCA tools to other subjects, engaging in FCA tools production, training of trainers, engagement without training) are presented under Contributions. Hence, the three sub-themes seem relevant and are presented in this report. 17

Evaluation of Formative Continuous Assessment

Extent of use of the tools: Almost all interviewed teachers confirmed that they have been using FCA tools regularly for classroom instruction, conducting assessments, providing feedback and record-keeping. These practices appear to be more common among those who teach Grades 1 and 2. However, there have been conflicting reports from principals and woreda experts about the use of FCA tools. While some principals and woreda experts confirm teachers’ claims, others found that FCA tools were not being used. According to one FCA-trained supervisor: “Teachers complain that FCA is boring. They often use summative assessment instead. We can't say that FCA is being implemented fully” (AMPS-26).

The data collectors used the classroom observation tool to record the frequency of teachers using the FCA tool to assess student performance. Tables 6 and 7 shows how often the data collectors observed the FCA tools being used in the observed classes, and the findings reviled that a significant number of classrooms did not use the FCA tools for the specific lessons and sessions observed.

Table 6: Use of FCA Assessment Tools in Classrooms, Grades 1 and 2

Grade 1 Grade 2 Assessment Tools A4 A6 Index card Pocket chart A4 A6 Index card Pocket chart Not used at all 15 12 25 14 17 14 20 17 Only one time 0 0 0 0 2 0 1 0 Two times 3 2 2 2 3 5 1 4 Three times 6 3 2 3 1 0 3 3 Used at Four times 1 1 1 1 0 3 0 1 least Five times 0 2 0 2 1 0 0 1 once More than five times 6 12 2 10 3 5 2 1 Total 16 20 7 18 10 13 7 10

Table 7: Use of FCA Assessment Tools in Classrooms, Grades 3 and 4

Grade 3 Grade 4 Assessment Tools Graphic Writing Index Graphic Writing Index card organizer Rubric card organizer Rubric Not used at all 18 15 16 22 22 20 Only one time 0 1 1 2 2 0 Two times 2 1 0 2 1 1 Used at Three times 2 0 3 2 1 3 least Four times 0 2 0 0 2 3 once Five times 1 1 1 0 0 0 More than five times 2 5 4 0 0 1 Total 7 10 9 6 6 8

Evaluation of Formative Continuous Assessment

The discrepancy between teachers’ self-reports and others’ reports on using the FCA tools may be partially attributed to differences in perspective. While teachers would report what they have done, supervisors and classroom observers would report what they have missed. These differences may be possible because FCA materials were not used uniformly. Many woreda experts demonstrated that they had limited familiarity with FCA: “We don’t know this training at the woreda level. By chance, I went to supervise the schools and saw the tools. I was surprised at how it happened without the knowledge of the woreda” (TGWE-04). The respondent’s comments, as well as the comments of other woreda experts (TGWE-04; TGWE-03; TGWE-05; TGWE-08; TGWE-24), may reflect that they are not closely invested in classroom practices and are not deeply committed to an external program. This could be because they lack knowledge, proper attitude, or adequate practice necessary for the purpose. As such, their observation that teachers do not use FCA tools in class should be considered in this context.

There are other factors that may affect teachers’ use of FCA tools. For example, woredas do not rely on teachers’ specialization and training when assigning them to subjects and grade levels (AMPS-17; AMT-19: Participant2; AMT-20: Participant2; TGPS-16; TGPS-01). This makes it difficult for teachers to assess their students and teach them properly, because they are teaching a subject and grade that is outside their realm of expertise. Respondents also reported that there was high teacher turnover, which posed problems because qualified and experienced teachers move to urban areas (AMWE-01). In addition, FCA-trained teachers are often re-assigned to higher grade- levels without sharing their FCA knowledge with other colleagues (TGPS-20). Teachers may also be less likely to use FCA tools because there is a shortage of FCA tools, or because they do not receive enough support. The latter two challenges are further discussed in the section on the Challenges of FCA Implementation.

Approaches to using the FCA tools: In a few cases, teachers have been mandated to use the FCA tools, but FCA is being implemented in a systematic and careful manner. According to one supervisor, continuous assessment was conducted in a haphazard manner before participants were trained in FCA. Now everything is done more seriously (AMPS-05). For example, in one school, FCA-trained teachers brainstormed ways to sustain FCA practices and decided to train their peers who did not get the same training opportunities that they did (AMT-11). Other interviewees reported that they classified their students according to performance levels and focused on supporting the low-performing students (TGT-21). Teachers also used FCA tools throughout the week, and on Fridays, they conducted a simple assessment (TGT-29). All these approaches reveal a common theme, in which teachers systematically use the FCA tools by matching them with lesson plans (TGPS-12; AMT-10; AMT-16; AMT-25).

Evaluation of Formative Continuous Assessment

3.1.3. Assessment, Record-keeping, Feedback, Follow-up, and Remedial Measures How do teachers conduct FCA, keep records, give feedback and conduct follow up to enhance student’s learning?

Types of skills assessed: Tables 8 and 9 show the extent of FCA tool usage in the classrooms in terms of assessing basic literacy skills in Grades 1 through 4 of the first cycle primary school curriculum.

Table 8 shows that FCA tools are more commonly used in Grade 1 than in Grade 2, and that A4 and A6 seem to be the most frequently used tools, while index cards seem to be the least used. Pocket charts are used more frequently in Grade 1 than in Grade 2. Table 6 and 8 both concur that A4, A6 and pocket charts are most frequently used in Grades 1 and 2 compared to other tools. Tables 6 and 7 also show that FCA tools are more frequently used in Grades 1 and 2 than in Grades 3 and 4. Finally, Table 8 demonstrates that sequencing, categorization and writing are the least assessed skills in both Grades 1 and 2.

Table 8: Types of Skills Assessed by FCA Tools in Grades 1 and 2

Grade 1 Grade 2 Skills Assessed Index Pocket Index Pocket A4 A6 A4 A6 Card Chart Card Chart Letter identification 14 19 4 12 5 9 2 4 Vocabulary 6 10 2 4 2 7 1 2 Blending 9 12 2 7 4 9 1 2 Segmenting 7 10 2 6 3 7 1 1 Matching sounds with pictures 7 13 1 5 2 3 1 2 Reading vocabulary 6 10 3 7 3 3 1 2 Sequencing 2 0 0 0 1 0 1 1 Categorization 0 0 0 1 1 0 0 0 Writing 4 2 1 4 1 5 2 1 Lower grades total 55 76 15 46 22 43 10 15

With regards to Grades 3 and 4, Table 9 shows that teachers generally use FCA tools less frequently, compared to Grades 1 and 2. The data show that the frequency of FCA tool usage declines as the grades increase. The classroom observation results in Table 7 support this finding, as well as qualitative data. Respondents reported that FCA tools are more useful in Grades 1 and 2. Graphic organizers were introduced as tools for teachers in Grade 3 and 4, but they are not used more frequently than A4 and A6. This is because older students are more familiar with words and letters, so they are less reliant on FCA tools for improved learning of literacy skills.

Record keeping of the assessment: One training component focused on the use of record-keeping tools to follow up on students’ learning. Teachers should be able to record and track each student’s learning progress. Some schools systematically implemented the FCA tools. For example, a group

Evaluation of Formative Continuous Assessment of language teachers discussed ways to enhance students’ skills through FCA tools and prepared a plan to do so (AMT-25).

Table 9: Types of Skills Assessed by FCA Tools in Grades 3 and 4

Grade 3 Grade 4 Skills Assessed Graphic Index Pocket Graphic Index Pocket A4 A6 A4 A6 Organizer Card chart Organizer Card Chart Writing 3 3 3 4 5 0 1 1 4 2 Reading 6 3 3 6 6 1 3 2 4 5 Listening 1 1 2 3 4 0 1 1 2 3 Vocabulary 2 4 4 5 6 0 2 0 3 5 Comprehension 1 1 4 2 3 0 0 2 0 0 Definition 1 1 1 2 2 0 0 1 1 0 Developing 1 1 1 1 1 0 1 1 0 2 vocabulary Synonyms 0 2 1 0 2 0 1 1 3 2 Antonyms 0 2 0 1 2 0 1 1 1 2 Blending 1 6 1 3 6 0 4 1 3 4 Segmenting 1 6 1 3 6 0 4 1 2 3 Word formation 3 5 1 4 4 1 1 1 3 3 Total 20 35 22 34 47 2 19 13 26 31

Interviewees mentioned that they use the FCA record-keeping forms (writing rubric, letter recognition registration) that they received during training (AMT-15; TGT-20) to record students’ learning (TGT-25). Some respondents explained that teachers complemented the rubrics with their own notes, which allowed them to collect more detailed information about their students’ performance (AMT-20; TGT-23; AMT-21: participant2).

Table 10: Use of Record-keeping Tools during Observation of Tigrigna and Amharic Speaking Classrooms

Number of Classrooms Observed Using Language Grade N Letter Total used Writing identification Teacher’s rubric recording notebook sheet 1 12 2 3 3 8 21

2 14 3 6 4 13 (81%) Tigrigna 33 3 10 3 N/A 2 5 12 (70%) 4 11 4 N/A 3 7 (57%) 1 20 3 5 3 11 16 2 13 1 1 3 5 (49%) 28 Amharic 3 15 4 NA 1 5 12 (43%) 4 17 5 NA 2 7 (38%) Totals 112 25 15 21 61

Evaluation of Formative Continuous Assessment

Data collectors noticed that teachers varied in the way that they recorded students’ learning progress and provided follow-up. They used different formats, such as formats provided through the FCA, school, self-developed, or a combination of the two. Some teachers stopped using the format given to them at the FCA training and began using their own because they did not have enough time, found that the format was inconvenient for large classes (TGT-30; AMT- 06), or did not have enough FCA formats for the school, which meant that they had to create their own format (AMT-20: participant1; AMT-17: participant2).

Teachers also varied in the frequency of their follow-up. Some teachers assessed students on daily basis using their own notebooks (AMT-24: participant1; AMT-22 participant2), while others conducted assessments and follow-up of their students on a weekly or monthly basis (AMT-21: participant1, participant2; AMT-11: participant1; TGT-13: participant1). Daily follow-up sessions with students certainly helps teachers effectively track their students’ progress, but this may be cumbersome for teachers in large classrooms.

Table 10 presents the classroom observation findings. READ M&E designed the Writing Rubric, Letter Identification Record List and Teachers’ Notebook to help teachers record students’ progress in reading and writing. The Writing Rubric and Teacher’s Notebook are to be used with all grade levels while Letter Identification Record List is to be used for recording students’ learning progress in Grades 1 and 2. Table 10 shows that the data collectors recorded 61 instances in which they observed teachers using one of the three recording tools. They observed 33 teachers in Tigrigna classrooms using one of the three tools, compared to 28 teachers in Amharic classrooms. Nearly all (92.9%) Grade 2 teachers from Tigrigna language classroom as compared to 5 out of 13 (38.5%) teachers from Amharic language classrooms used all classroom assessment record tools for recording students’ progress in reading and writing. Similarly, 66.7% Grade 1 teachers from Tigrigna used the three tools compared to 55.0% of the same group from Amharic language classrooms. The classroom observations show that teachers have made efforts to systematically implement an assessment and recording process, pay attention to each student’s progress, and integrate activities with teaching, monitoring learning progress and providing support.

Feedback to students: Teachers are expected to provide students with information about their level of performance after conducting assessments. Providing learners with immediate feedback about their performance has a positive impact on their learning (Brown, Bull, & Pendlebury, 1997; Muskin, 2017; Brookhart, 2008). However, teachers in Ethiopia have difficulty providing their students with feedback (Tefera, 2014). To address this challenge, READ M&E trained teachers to use feedback as part of FCA. Teachers’ experiences with the FCA training suggests that they learned how to better communicate feedback to their students. Many of the interviewees reported that they appreciate the children’s efforts, regardless of whether the response was correct, focus on improving reading errors, and emphasize the importance of timely feedback. They provide the feedback in a positive and constructive manner, so that the child may not feel discouraged or

Evaluation of Formative Continuous Assessment humiliated for having read aloud in public. By appreciating children’s efforts, teachers give struggling students multiple chances to improve their reading until they fix their mistakes and master the text. Table 11, which shows teachers’ feedback practices, also support the interview results.

Table 11 shows that data collectors observed teachers providing timely, constructive feedback to their students in 83.3% of Grade 1 Tigrigna language classes and in all Grade 1 Amharic language classes. Data collectors most commonly observed teachers giving their students constructive and positive feedback, while they also less frequently observed negative feedback in both languages.

Teachers also talked about using different types of feedback approaches. They explained that different people can provide feedback, such as the teachers themselves (TGT-22, AMT-17), the children’s peers (AMP-04, TGT-24, TGT-25), or the entire class (TGT-04, TGT-17). According to one teacher, “If a student did not answer a question right, I will give the chance to the class. If a class fails, I will do it immediately” (TGT-04). In some cases, a group of students that correctly answered a question provides feedback to another group that did not answer the question correctly. Teachers also provide students with feedback in front of their parents because parental involvement helps student learning (TGT-25).

Table 11: Teachers’ Feedback during Classroom Observations

No. of Classrooms Observed Teachers Giving Feedback Types of Feedback Tigrigna Amharic Used Grade Grade Grade Grade Grade Grade Grade Grade Total 1 2 3 4 1 2 3 4 Constructive feedback 10 11 9 8 20 12 13 16 99 given to students Specific feedback rather 9 8 8 7 18 9 12 12 83 than general Descriptive rather than 6 5 6 4 15 7 9 8 59 judgmental Share information rather 4 6 6 5 13 8 9 6 57 than giving advice Positive constructive 11 12 8 6 17 11 14 13 92 feedback Negative feedback 2 3 2 1 3 4 4 6 25

The respondents also explained that they communicated feedback in different contexts, whether it be in the presence of others (TGT-25), individually (TGT-26), or through written feedback in their exercise books and textbooks (TGT-26, TGT-30). Some teachers felt that providing feedback in private was more appropriate, so that they do not discourage their students. Others found ways to provide gentle feedback to their students in front of the class: “I used to give chance for another student instead of announcing the status of his or her response. The phrase I use is ‘Anyone else?’ However, I will finally provide them the correct answer to the question I posed” (AMT- 18: participnt2).

Evaluation of Formative Continuous Assessment

What are the remediation strategies frequently employed by teachers in supporting students with reading problems? What supports are provided to teachers? Remedial Strategies Employed: One of FCA’s objectives is to provide remediation to those who need special support. Table 12 presents the extent to which various remedial strategies were used in Grades 1–4 classrooms. Table 12 indicates that remedial strategies were used less frequently as the grades progressed. One-to-one instruction was the most common (70 teachers observed) remediation strategy, followed by small group instruction (55 teachers observed). Prescribing students additional practice, the third most frequent strategy observed, was also a recurring theme in the qualitative data.

Table 12: Teacher Remedial Strategies Used during Classroom Observation

No of Classrooms with Specific Remedial Strategy Type of remedial strategy used Grade 1 Grade 2 Grade 3 Grade 4 Total Re-teaching 10 6 9 9 34 Alternative instructional strategies 11 9 3 5 28 Task analysis 7 7 5 6 25 Small group instruction 13 14 13 15 55 One-to-one instruction 23 18 16 13 70 Additional practice 13 9 11 8 41 Total 77 63 57 56 253

Follow-up and Support to Teachers: Teachers need follow-up and support to smoothly implement what they learned from the FCA training. These supports are expected to come from principals, supervisors, or woreda experts. The research team identified seven sub-themes regarding follow-up and support for teachers:

▪ With whom do they follow up? (subjects of follow up) ▪ How do they follow-up? (methods of follow-up) ▪ How often do they follow-up? (frequency) ▪ What they focus on during the follow-up? (purpose) ▪ What tools are used? (tools of follow-up) ▪ What did they find out? (outcomes of follow-up), and ▪ What measures do they take based on the outcome? (implications and support of follow-up)

With whom do they follow-up? Principals and supervisors’ interviews reveal that in addition to following up with teachers, they monitor students’ learning progress (TGPS-13, TGPS-01, TGPS- 26): “We have a follow up for our student too. We have some networks and group students in our

Evaluation of Formative Continuous Assessment school. So, we have discussed about the teaching learning process in our school with the network and group leaders weekly” (TGPS-26).

How do they follow-up? The principals and supervisors conducted follow-up through meetings and discussion forums with teachers (AMPS-08), teachers’ discussion groups (AMPS-15) planned (TGPS-11; TGPS-19) and unplanned classroom visits and observations (TGPS-01), and discussions with student network groups and leaders (TGPS-26).

How frequently do they follow-up? The principals and supervisors varied in their response. Sometimes they supervised on a monthly basis (TGPS-01; TGPS-22; AMPS- 12), every two weeks (TGPS-13), held weekly discussions with student network and group leaders (TGPS-26), conducted classroom observations every 2 or 3 months (AMPS- 06; AMPS- 07; AMPS- 08; AMPS-01), and met with teachers every 15 days (AMPS-15).

What do they follow-up on? The supervisors reported following-up on teachers’ implementation of FCA activities (AMPS-01) and use of FCA tools (AMPS-08). In addition, they ensured that teachers properly used teacher guides (TGPS-23), textbooks (TGPS-26), lesson plans (AMPS-01). The supervisors made sure that teachers used the new “I do, we do, you do” teaching methodology (TGPS-25) and effectively implemented student assessments. They also monitored students’ learning progress (TGPS-01) and whether students knew and understood the lessons (TGPS-01).

What tools do they use for supervision? Woreda experts and supervisors primarily relied on discussions and meetings with teachers and student groups. They also used checklists during classroom supervision (TGPS-18; AMPS-20: participnt1; TGPS-01; TGPS-24).

What did they find out? They found that students have demonstrated remarkable achievements in identifying letters, reading, blending and segregation and forming sentence (AMPS-06). However, they reported that teachers struggled with using the FCA tools. For some teachers, the FCA is boring (AMPS- 26). They also found that teachers do not understand how to use the FCA tools (TGPS-16). Others did not use the FCA tools, and kept them locked up instead (TGPS-19).

What measures do they take based on the outcome? Based on what they find, the supervisors often provide teachers with feedback (TGPS-11). They identify areas for improvement, then take immediate steps to address them. They provided teachers with professional development opportunities, such as technical and material support, orientation, and short-term training; providing advice on addressing the gaps observed during the learning–teaching process (AMPS- 12; TGPS-19; TGPS-16); providing opportunities for teachers to evaluate and support each other (AMPS-21: participnt1); and identifying ways to extend best practices to other cluster schools (AMPS-08). They also encouraged teachers to build stronger relationships, by asking them to help the students and meet with parents to discuss their children’s performance (AMPS-14; AMPS-09;

Evaluation of Formative Continuous Assessment

AMPS-01). Finally, they provided teachers with logistical support, by helping them find a space to keep their FCA tools (AMPS-07). The supervisors are generally supportive of the teachers.

Woreda experts’ follow-up and support: Woreda experts reported following up on schools, teachers (i.e. those that are trained on FCA or not), and students in their respective woredas. Themes from woreda experts’ narrative on follow-up include reports from cluster schools, regular school supervision, assessment data generated on school performances, regular checklist used for the supervision of education activities that incorporate FCA, regionally prepared teacher evaluation formats, and self-constructed forms to check student progress. Data are not available on how frequently woreda experts provided follow-up specific to the implementation of FCA. During their follow-up, woreda experts checked educational performance, school discipline and character, MT teachers’ training, and student learning progress.

Outcomes: Woreda experts found positive outcomes associated with the FCA. One school, located in the Debre Birhan Woreda, successfully implemented FCA, which resulted in students’ improved reading skills. Some schools in the Mekelle Hadinet Woreda ensured the sustainability of the FCA materials by transferring the information on the laminated flash cards to permanent materials, using wood and metal. However, respondents also mentioned challenges associated with visiting all the schools with Grades 1-4, which resulted in limited follow-up from the woreda. The woreda experts also struggled with turnover, transfer, and promotion of FCA-trained MT teachers and the difficulties related to finding replacements. Finally, woredas had a limited budget for trainings and materials.

Measures: To address the aforementioned challenges, woreda experts provided assistance to teachers and schools by providing assessment feedback, providing trainings for new teachers and refresher trainings for trained teachers. They also coached individual teachers by helping them use the FCA tools more effectively and rely on assessment checklists to improve student learning. They also provided incentives by ranking schools according to their performance and encouraging experience-sharing and scaling up best practices. 3.2. Contributions of FCA Training and Tools What are the perceptions of teachers, supervisors and woreda experts about the contributions of the FCA training and tools in improving children are reading ability? Research on assessment practices in Ethiopian schools suggest that teachers may misunderstand or inappropriately use continuous assessment practices. For example, teachers implemented the summative form of continuous assessment, rather than FCA (Aytaged, 2010), used continuous assessments as a way of continuously testing their students (Abiy, 2013), or implemented continuous assessment activities that are not in line with best practices (Fisseha, 2010). To address teachers’ unfamiliarity with continuous assessment, Sintahehu (2016) argued in favor of a harmonized, continuous assessment policy or guidelines. This following section discusses

Evaluation of Formative Continuous Assessment teachers’ improved understanding of FCA, its immediate impact on students, and how these outcomes contribute to FCA’s sustainability and scalability.

3.2.1. Teachers’ Improved Understanding of FCA’s Role Principals and teachers overwhelmingly had positive things to say about the FCA trainings. They felt that the trainings contributed to teacher professional development: “The FCA training was practical and helped enhance my professional competence” (TGT-11). Another expressed the same sentiment, saying, “After the training, I have observed great changes in my understanding and the process of teaching learning (AMT-21: participant1). The FCA trainings involved supervisors, school principals, and teachers, and trained them all at the same time. Participants felt that this approach helped participants understand each other and helped them cascade the training to other schools in their cluster (TGPS-17). Teachers also reported sharing their experiences with one another. For example, one respondent shared that a new, inexperienced teacher joined their school, and did not have any training in FCA. To address this, the new teacher’s colleagues taught her what they learned in FCA training (AMT-25). They also shared their experiences with one another and discussed their own performance (AMT-11, AMPS-12).

Participants were asked to compare their professional practices before the trainings, and their performance after the trainings. The respondents agreed that the trainings helped teachers better understand teacher’s guides (AMT 18: participnt4), teaching methods, assessment skills, feedback skills, and reward techniques. Teachers learned how to implement the continuous assessment approach more consistently (TGPS-10). They also found that the “I do, we do, you do” teaching methodology was helpful because it shifted the teaching approach from being teacher-focused to student-focused, and improved collaboration among students (TGT-17: partcipnt1; TGT-19: participnt1; AMT-17: participnt2; AMT-17: participnt3). In addition, teachers found that the FCA tools helped teachers save time preparing for class, thus improving education quality (TGPS-24; TGT-29; TGT-17). Some teachers were even inspired to develop their own additional aides to complement the FCA tools (AMT-27).

Respondents’ narratives revealed that assessment was becoming more systematic (AMPS-20), evidence-based (TGPS-17: participnt2), and constructive (AMT-06; AMT-07). Teachers reported that whereas previously they provided their students with negative feedbacks, they are shifting to provide positive feedback and encourage students to perform better (AMT-05; AMPS-08). Teachers have shifted from summative-focused continuous assessment to a more formative- oriented approach; that is, a change from general to individual assessment, and from use of exam and tests for grading, to a daily-based assessment for monitoring learning progress: “Before the training, I was assessing the whole class without considering their differences; but now, I am assessing my students at different phases of instruction and I treat them separately by considering differences in their levels of knowledge” (AMT-17: participnt1). Another teacher shared, “We used to assess our students using only exams, but now we assess how our students cope with the daily lessons and record their progress using the format given during the training” (AMT-15).

Evaluation of Formative Continuous Assessment

In general, interview results seem to indicate that the FCA training has contributed a lot in the professional practice of teaching and learning of reading. The FCA approach is engaging teachers across Ethiopia and reviving a spirit of hard work, commitment and motivation.

3.2.2. Changes in Students’ Learning and Achievement Research has shown that many factors—including student socioeconomic status, teacher-student ratios, class sizes, and teacher quality—are associated with an overall improvement in student achievements. However, evidence indicate that of these factors, teacher quality appears the most highly correlated one with student learning (Darling-Hammond, 1999; National Research Council, 2001). If teachers are positively impacted by a new professional development experience, then they will adopt improved classroom practices, which will ultimately impact student learning and achievement (Guskey, 1997).

The research team examined narratives from teachers, principals, and woreda experts about the FCA training’s impact on student performance. For the most part, respondents expressed positive results with the FCA, which were organized into three broad themes: Direct contributions, improvements in learning and achievement, and psychological contributions.

General contributions: Sub-themes that emerged from the data include strengths of FCA materials and student interest, improved reading skills and abilities, improved academic performance, and special contributions for early grades.

Almost all interviewees mentioned ways that FCA generally contributed to improved instruction quality and student outcomes. Teachers noted that the FCA tools helped improve students’ overall performance (TGWE-04) and that students were better able to understand their lessons (AMT-20: participnt1; TGT-17). According to a teacher from Kombolcha, “Our performance before and after the training is different. I wonder how many students we could have supported if these materials were provided earlier” (AMT-27).

Respondents praised the FCA materials as sturdy (TGT-29), visual (seeing is better than hearing) (TGT-30), colorful (TGT-23), and attractive (TGT-01). They were considered to be an effective way to conduct lessons (AMT-25: participnt2). The students also responded positively to the FCA materials and found them to be interesting (AMT-26; AMT-16), enjoyable (AMT-19: participnt1), and engaging (TGT-30). The FCA tools stimulate students’ curiosities, interest, and engage them in activities rather than putting them in a passive listening position. According to a teacher, the FCA tools contributed a significant improvement in their students’ learning, motivation, attention, and performance (AMT-08). By using FCA tools to engage students in the lesson, teachers make learning more enjoyable, which ultimately helps them improve students’ academic performance:

Evaluation of Formative Continuous Assessment

“When we use these tools, students usually feel and assume that they are playing not learning in the classroom. This makes them pay enough attention to the lesson and understand the lesson quickly” (TGT-17).

The impact of FCA on children’s learning behaviors, interests, and even achievements has long been documented in various researches (see Ahmed, 2016; Birhanu, 2013; and Desalegn, 2014). Qualitative evidence shows that FCA helped improve student learning through the READ M&E FCA initiative.

Improvements in learning and achievement: Research confirms that proper implementation of continuous assessment is likely to result in impressive improvements in student performance (Iqba and Samiullah, 2017; Abejehu, 2016). If the tools are attractive and engaging in learning, then it naturally follows that learning reading becomes fun, relaxing, and ultimately effective. Interviewees from all groups reported significant developments in students’ reading and writing abilities, such as voice and word recognitions, letters differentiation and word constructions, co- relating letters and drawings, differentiating between root words and annexes, reading by segmenting and blending words. One teacher reported, “Many of our students are now able to read fidels. They can use punctuation marks when they construct different kinds of sentences. Students’ reading and writing skills improved as the tools facilitate segmenting and blending activities” (AMT- 25). Another teacher said, “I have 35 students and 25 of them scored 95 and above and the remaining 10 on B rank” (TGT-03: participnt2). Improved student performance has manifested in positive ways, such as students’ increased interest in using the library and reading at home (AMWE-04) and increased participation in public reading competitions (TGWE-08).

Teachers reported that FCA practices had a more meaningful impact on students in Grades 1 and 2 than in Grades 3 and 4. This is because older students are more familiar with words and letters, so they are less reliant on FCA tools for improved learning of literacy skills: “In practice, the FCA tools are more important for Grade 1. I teach grade four. Truth to speak, I never used it… I did not find it important for fourth graders” (TGT-03).

FCA tools and classroom approaches were helpful in providing support to students that need additional attention. For example, the tools helped teachers identify children with weaker reading skills, such as children who don’t speak the local language (AMT-11), or those who struggle to read and identify letters (TGT-19: participnt1). Some of the materials are big, which benefited children with low vision (AMT-27). Teachers supported individual students that did not benefit from regular teaching, and required special assistance to learn reading.

Psychological contributions: Iqbal and Samiullah (2017) conducted experimental research on the impact of continuous assessment among students and found that students enjoyed continuous assessment practices. Similarly, READ M&E’s research on the FCA pilot found that the

Evaluation of Formative Continuous Assessment intervention-built self-confidence, created a spirit of competition, and improved participation among students. Many respondents reported that the approach boosted children’s confidence in reading (TGT-15; TGT-02; TGT-09; TGT-10). Pictures helped children successfully identify and remember letters in the alphabet. which ultimately improved their self-confidence in reading.

Interviewees found that the FCA training and tools have brought a sense of healthy competition (TGT-17; AMT-24) among the children: “When our students are asked to do exercises using the FCA materials, they act like a teacher and they like it. It brings a sense of healthy competition among them” (TGT-30). In addition, others reported improved teaching-learning atmosphere (AMT-29), increased student motivation (AMPS-23: participnt1; TGT-12), and reduced student absence and dropouts (AMPS-23; AMT-05; AMT-08; TGPS-13).

3.2.3. Cascading and Sustainability Even though the READ M&E FCA intervention targets MT teachers and classrooms, it can still have an impact on different subjects, teachers, and classrooms. For example, FCA can be applied to subjects other than the mother-tongue language, the FCA tools can be reproduced into consumable products, and local stakeholders can provide FCA training to untrained colleagues. Local stakeholders reported that the FCA approach has had a positive impact on parents regarding their children’s learning (TGT-15; TGT-11).

The FCA training and tools had positive, unintended side effects. For example, the FCA ready- made tools saved teachers time and energy, created a spirit of competition among the children, and ensured greater participation. This section explores these issues in greater detail.

Cascading from Woreda experts’ experiences: Cascading plays an important role in ensuring FCA sustainability. As expected, woreda experts reported varied experiences in cascading the FCA approach from woredas to schools. Although there have been cases in which woredas did not successfully cascade the FCA approach (TGWE-03; TGWE-24; TGWE-04; TGWE-08), some schools in woredas reported taking the initiative to copy tools, methods, and assessment forms from other experienced schools. Similarly, trained teachers have shared their experiences with untrained ones, with the woreda’s limited involvement (AMWEO-03). The remaining nine woredas reported taking measures to cascade the FCA initiative through training, experience sharing, and sharing resources. They did so despite the fact that they were neither officially informed nor directly involved in the FCA implementation.

The cascading effort was primarily implemented through an experience sharing forum among schools, training, or both (TGWE-07; TGWE-02). Two woreda experts reported that they have arranged experience sharing with effective schools (AMWE-04; AMWE-01), while other woredas provided experience sharing in conjunction with supportive training. Few have even extended a portion of their limited resources to schools (AMWE-06; AMWE-05). One woreda expert reported that the woreda has assessed the FCA intervention three months into its implementation period

Evaluation of Formative Continuous Assessment and learned that the project was encouraging. As a result, they submitted a proposal to their superiors for scale up and are still waiting for feedback (AMWE-02). Generally, the interviews yielded evidence that suggest that schools and teachers have made encouraging efforts to sustain the FCA approach. For example, some schools produced local aids and materials. Teachers also expressed satisfaction with the FCA and wanted the government to be more involved in FCA activities (AMWE-06). The teachers’ receptivity toward FCA shows that schools are ready to implement change toward improved education quality.

Adapting FCA tools to other subjects: Teachers did not limit FCA activities to their own classrooms, but rather, they have disseminated FCA practices to other schools (AMPS- 07), to other grades (AMPS-14; TGPS-22; AMPS-21), and to other subjects such as English (AMPS- 16; AMT- 06; AMPS-22; TGT-02) natural sciences (TGT-29), and mathematics (TGT-02; AMT- 25).

Engaging in FCA tools production: Some schools already constructed a room designated for FCA tools (e.g. AMPS-26) or plan to do so (e.g. AMPS- 16). Teachers in other schools prepared their own FCA tools in different ways, such as creating their own graphic organizer, customizing the FCA tools for their own purposes, photocopying existing tools, or writing them down. These efforts are significant, given that schools in Ethiopia are operating under a small budget. Furthermore, these interesting initiatives should be scaled up to ensure sustainability of the FCA initiative in Ethiopian.

Training of trainers: FCA trained teachers also trained the untrained MT teachers (AMT- 11; AMPS-11; TGPS-29; TGT-04), teachers in other fields (AMT- 25; TGPS-04), teachers from other schools and departments.

Engagement without training: Some teachers who did not receive the FCA training were still engaged in the FCA initiative. For example, some untrained teachers used the FCA tools: “Even if I did not take the training, I have used the FCA tools depending on the nature, the content and the lesson of the subject.” (AMT- 03). The FCA tools are demonstrably popular through word-of- mouth and seem to facilitate positive change shortly upon their use. 3.3. Challenges, Opportunities and Measures What were the major challenges encountered and opportunities available while using FCA tools? What suggestions were given by participants to deal with the challenges? This section discusses the challenges of FCA implementation, opportunities, and suggested strategies for addressing both.

3.3.1. Challenges The FCA is a very exciting and popular way to conduct assessment in Ethiopia. Due to the government’s lack of involvement in the FCA implementation, it is understandable that there are

Evaluation of Formative Continuous Assessment significant challenges in the implementation process that may frustrate the educators involved. What are the challenges that teachers and schools encounter while using FCA tools for assessment?

No Major Challenges: In all of the responses provided under this category, participants do not mention any problems with the FCA training, tools, and implementations.

▪ No problem: Respondents stated that they did not face any problems regarding the availability and use of FCA tools. ▪ Initial challenges alone: Respondents shared that they initially encountered problems that gradually faded away, such as students getting distracted by the colors of the FCA tools (TGT-11), teachers’ negative attitudes (AMPS-23: participants1, 2), and teachers’ unwillingness to share resources (AMPS-20; AMPS- 06; AMPS- 04). ▪ Non-FCA problems: Respondents shared challenges that were unrelated to FCA tools and approaches. Examples include personal issues that affect staff and student participation, or problems inherent to MT education.

Design issues: problems of graphics and contents of tools: A lot respondents encountered technical, mechanical, conceptual, and contextual problems when using the FCA tools.

▪ Technical and mechanical problems: Teachers found it difficult to hang the FCA tools on the walls, such as the mind map, web map, context and connection map, sequence map, and KWL (TGT-30). They did not have enough space in the FCA forms to record students’ names and other important information. The cards were very attractive, which distracting the students and were likely to be stolen. Finally, teachers complained that the materials were expensive, and that it was difficult to prepare them locally (TGT-17). ▪ Relevance: Participants argued that the tools need to be adapted to the local context. For example, the tools contained words that were unfamiliar in the local language (TGT-13: participant2) or had pictures of plants that are unknown and unfamiliar in the local region (TGT-24). In addition, the materials contained words that are not familiar in the local language: “The word and picture of ice cream is not understood easily by our students because it is not contextual. It would be better if these words are substituted by those that are more contextual” (AMPS-11). ▪ Grade-Appropriateness of the Tools: According to the teachers, some of the tools are not grade-appropriate. They found that the picture flashcards are not suitable for Grade 2 students and that the Graphic Organizer are not as relevant for first graders (AMT- 27). They argued that the tools for Grades 3 and 4 need to be redesigned to fit the students’ capacity, and that continuous assessment tends to focus on students in Grades 1 and 2: “Like Grades 1 and 2, focus should be on students’ performance in

Evaluation of Formative Continuous Assessment

Grades 3 and 4. The assessment should focus on how the students read and how many words they read within the given time” (AMT-17). ▪ Complexity and Ambiguity: Some of the letters on the drawings are difficult to understand and identify, even for teachers (AMT-10). There are also discrepancies between the drawings and letter names (AMT-11; TGPS-13). The discrepancies also exist between the textbook and the FCA materials, making it challenging for teachers to align the FCA materials with the lesson plan and teacher guide (TGT-17: participant2). The teachers complained that the scoring system is subjective, making it challenging for them to score students consistently (TGT-21).

Teacher-Related Problems: One recurring theme in teachers’ complaints is that the FCA has increased their workload (TGT-10; TGPS-12; AMWEO-03; TGPS-04). This is consistent with previous research that found that teachers in Tanzania complained about increased workload due to continuous assessment (Mpapalika, 2013). FCA requires individualizing classroom work on a regular basis through assessment, follow-up, and support. Teachers of large classrooms struggle with this practice, because it demands significant time and effort on their behalf. Their stress is compounded by the fact that the textbooks contain numerous topics that they must teach. One teacher explained, “The time is short to cover the lesson on the textbook because there is reading, writing, listening activities, in addition to the questions and group work activities within the period” (TGT-12: participant1). Teachers do not have enough time to complete their lessons in class, review their students’ work, identify areas for improvement, and provide corresponding follow-up and support (AMT-25). While homework and group work are potential approaches to dealing with time constraints, it is still the teacher’s responsibility to check the work. Furthermore, cooperative learning must be supervisor, particularly in the early grades.

Respondents also noted instances in which teachers inconsistently used the FCA tools, prepared false reports, and refused to cooperate (TGPS-24; TGPS-02; AMPS-04). Teachers also struggled with motivational and attitudinal problems, such as declining enthusiasm and negative attitudes about FCA activities (AMPS-08; AMPS-10; AMPS-05; AMPS-06). This may be due to teachers’ inadequate skills, teacher shortage, assignments in subjects and grades outside of their expertise, and turnover of trained MT teachers.

Student absenteeism: Researchers found that the FCA approach minimized student absenteeism. Nonetheless, when students are absent, it complicates their learning. Respondents reported that student tardiness and absenteeism posed a threat to FCA implementation. According to a teacher, “Student absenteeism is a major challenge. It requires additional time to repeat the lesson. Yet, students are not willing to come for remedial actions” (AMT-02).

FCA Tools: Many interviewees struggled with finding a safe place to store the FCA materials. They mentioned different problems, including inadequate classroom conditions and school environment, lack of a designated space for storing the FCA tools, and lack of cleanliness. 33

Evaluation of Formative Continuous Assessment

Respondents from different schools frequently mentioned a shortage of FCA tools. They complained that they used all the FCA tools, such as flash cards and recording sheets. The schools reported that the FCA tools are expensive and that they cannot reproduce them locally. They also argued that they do not have enough graphic organizers to teach all the topics.

Problems with Woredas, Schools, and Principals: Some woreda experts, supervisors and principals were not included in trainings, so they could not provide their teachers with follow-up and supervision (TGWE-04; TGPS-09). According to one principal, “As supervisors and directors, we did not take the training, so we do not know how to help them in their teaching and assessment” (TGPS-01). Similarly, a woreda expert says, “When we supervise we see them while they are using it. But, we do not know whether they were using it properly and sufficiently or not” (TGWE-03). Consequently, some supervisors and principals do not provide adequate follow-up (AMPS-05; AMPS-01).

3.3.2. Opportunities It is important to identify potential opportunities for using FCA tools for assessment in the classroom and elsewhere. As such, we must search for solutions to the challenges reported earlier.

There are good opportunities for the Federal Ministry of Education to promote FCA implementation. For example, the Education and Training Policy of the Federal Democratic Republic Government of Ethiopia (1994: 21) indicates that ‘’Continuous assessment in academic and practical subjects, including aptitude tests will be conducted to ascertain the formation of all- round profile of students at all levels’’.

The government also suggested a self-contained classroom strategy for the first cycle primary education (Grades 1 to 4) in which a single teacher is to handle all the subjects and also preferably work with the children through Grades 1 through 4. This builds a strong bond between the teacher and students through proximity, ensures better knowledge about follow-up and support to the children, and integrates education with care, which is particularly needed in the early years. More importantly, this strategy creates a good platform for conducting FCA. The Government has also expressed preference for formal continuous assessment with the purpose of promoting early learning and development, rather than certifying learning with summative assessments. This helps children meet minimum education standards and progress to subsequent grade levels. Research has consistently documented that early appearance of tests is likely to infuse anxiety and apathy to learning right at the very beginning of education. Furthermore, summative assessments tend to promote external control to learning (working for exams and grades than for knowledge), and in doing so instigates an unhealthy competition among students. The Education Sector Development Programme V (ESDP V) (2015) – which serves as a central strategy document for educational development in Ethiopia from 2015/2016 to 2019/2020 – holds that delivering quality education that meets the diverse needs of children is one of its top priorities. This would create a strong basis for FCA implementation as a tool to improve early grade reading in Ethiopian primary schools. Finally, the newly launched teachers’ continuous professional development program and its

Evaluation of Formative Continuous Assessment working document provide another opportunity to incorporate FCA implementation, despite its implementation problems (Girum, et al., 2018).

Professional groups and other education stakeholders that catalyze FCA interventions are the actors that connect the top (policy provisions) with the bottom (FCA implementation at school level). As a result, we to understand the opportunities on the ground toward managing challenges and facilitating FCA implementation. Researchers asked the three groups of interviewees to share envisioned opportunities so that they can be leveraged to improve FCA activities.

The interviewees provided a list of opportunities related to materials and tools, as well as education stakeholders, such as teachers, schools, and parents.

Materials and Tools: Participants believe that the existing textbooks provided an opportunity because they complement the FCA tools and materials. Teachers who are teaching segmenting can use the FCA materials and textbooks as pedagogical tools. They also argued that it would be helpful for decisionmakers to prepare an explanation and week-by-week timeline for using the textbook, so that teachers could better understand how to pace themselves (TGPS-17).

Teacher-related opportunities: Teachers have strong attitudes and motivation for change, new approaches, and methods, which would serve as an opportunity to launch and disseminate FCA activities. They were partially motivated by the desire to improve primary education, particularly reading. Teachers demonstrated that they were open-minded, receptive, motivated and enthusiastic of the FCA approach (AMPS- 26; AMPS-02; AMPS-05). They thought that sharing experiences with other teachers was a great way to implement FCA (AMT- 03; AMT-02; AMT-08; AMT-05; AMT- 06). One teacher said, “When it is my break time is, I will go to the class of the students and ask the other teachers to send me the weak students and I teach them. This brought really a good change.” (TGT-04). Teachers have demonstrated creativity by developing instructional aids and technologies to complement FCA materials.

School-Related Opportunities: Some schools expressed their enthusiasm for FCA activities by constructing a special room for FCA tools, hiring FCA-trained teachers to train their peers, duplicating FCA materials, among other things. One respondent notes, “The school administration is also providing necessary supports to the implementation. The school administration for example, provided us with hard-paper with which we prepared fidels that students take to their home and practice” (AMT- 25).

Parent-related opportunities: Other opportunities include leveraging parents’ interest in their children’s education and encouraging them to be more involved with the school. Some participants shared promising experiences with school parents (AMPS-14).

Evaluation of Formative Continuous Assessment

3.3.3. Participants Suggestions for Enhancing FCA A lot of themes have emerged from the narratives of teachers, principals and woreda experts regarding measures to enhance and sustain the use of FCA at all levels. The first and most important recommendation is that FCA should be sustained and scaled-up for future use.

FCA Expansion, Scalability, and Continuity: When the project phases out, the government and local communities must take responsibility for project scale-up (AMWE-06). Participants recommended raising awareness about FCA among the school-community and regional education as a means of sustaining the program (AMT-05; AMT- 07; AMPS-18; TGWE-07). Others believed that the program should be expanded to other schools and grades, for all subjects (AMPS- 06). In particular, they thought it would be useful to expand the program to English language classes (AMPS-23; TGT-17; TGPS-19; TGPS-17) and to preschool (AMPS-21; AMPS- 14).

Resources (Materials, FCA Tools and Classrooms): The interviewees mentioned that one of the biggest challenges they encountered was a shortage of FCA tools. A number of them requested additional supply of FCA tools (TGT-17; AMT-05; TGT-04). Others also mentioned the need for a safe and independent room (TGT-29; AMPS-17; TGPS-30; AMPS-10) to properly store and use the tools. Participants argued that the woreda and regional offices need to give attention and allocate a budget to FCA implementation. A more viable and sustainable strategy is to adapt FCA tools using locally available materials (AMPS-06; AMT-13; AMT-05; AMPS-15; AMPS-14; AMT-08). One woreda expert suggested that the woreda education office should support schools by training them on how to prepare effective teaching aids. He also suggested creating links with local institutions like vocational colleges to train them how to produce teaching aids from local materials (AMWE-06).

Participants also pointed out the need to ensure that the curriculum is compatible with the FCA approach. An interviewee asks, “There are seven periods in a week for natural science and five periods for Tigrigna. How could a child read natural science passages before knowing the letters?” (TGPS-29).

Other resource-related suggestions include solutions to address the teacher shortage, the need to support children from poor households in rural areas, providing supplemental reading materials, and providing computers and related equipment to schools.

Training Needs: Some interviewees indicated that the training should be for all the stakeholders: teachers, principals and supervisors, woreda experts, and regional officers (TGT-29). Some argued that both trained and untrained teachers need to receive this training, “The training given is not enough to fully implement these techniques. Teachers who have not obtained training need to be trained and those who were already trained need additional training” (AMT-24). In fact, almost two-thirds of respondents expressed the need to train untrained teachers. Many untrained teachers

Evaluation of Formative Continuous Assessment are eager to get training because they believed that it would build their teaching competencies. Participants also recommended refresher trainings, so that trained teachers can also improve their capacities in implementing FCA. Other interviewees underscored the importance of training supervisors (AMPS- 15; TGPS-21).

Regarding the training itself, participants felt that the trainings were too short, and that they should be continuous (TGT-13; AMT-21) rather than a one-time session (AMT- 06). They also suggested expanding the training scope to be more in-depth (AMT- 15; AMT -15; TGPS-22; TGT-11): “The training should be intensive to avoid misunderstanding on the implementation. Especially, there is a skill gap in using the recording checklist because we don’t have enough knowledge on how to record students’ performance and to follow their improvement” (AMPS -02: participant1).

Finally, principals and teachers recommended facilitating teacher experience exchanges to ensure uniformity (TGPS-04; TGPS-29), empower new teachers (AMPS-10), and augment previous training (TGT-29). They suggested using high-performing MT language teachers as a model for other teachers (AMPS- 04; AMPS-08). This opportunity could serve as a reward for their hard work (AMPS- 14).

Teacher Assignment: Teachers should be assigned according to their field of study and training (AMT- 18; AMT-17; AMPS-19; AMPS- 07), even when teachers are transferred from one school to another school (AMT-08; TGPS-04).

Uniformity of FCA Implementation: Interviewees thought that the FCA intervention should be implemented uniformly across all schools (AMT-05; AMT-07). To do so, local stakeholders should jointly plan FCA activities. One interviewee noted that in Tigrigna, teachers from the same school cluster prepare the annual lesson plan to ensure uniformity, and suggested that this approach would work for ensuring a uniform FCA approach (TGPS-04).

Supervision and Follow-up Gaps: Many teachers pointed out that there were gaps in the supervision and follow-up that they received. They argued in favor of a system in which supervisors provide regular follow-up on FCA implement and provide teachers with timely feedback about proper use of the tools and assessment approach (AMT-21). A teacher from Mekelle, for example, explained: “If the director or supervisor comes, they do not give me any feedback except that they see my way of teaching and leave. They have to guide us how to teach… We need to be checked how we are implementing. Training is good, but it is not enough if you do not follow us. We need follow-ups and supports” (TGT-03). Teachers welcomed data collectors, believing that they were there to provide them with follow-up and supervision.

Evaluation of Formative Continuous Assessment

School principals stressed that they are trying their best to provide guidance and feedback, but that they need support from the woreda office: “We are working hard but there is no support from woreda on the implementation. So, woreda experts have to follow continuously by recording the improvements on their checklist” (AMPS-23: participant6).

In fact, participants recommended supervision and follow from various sources, including READ M&E staff (AMPS-23; AMPS-21; AMT-22; TGPS-16; AMPS-24), regional education offices, woreda offices, and school principals. READ M&E was already late in its follow-up and needs to make continuous follow-up and give teachers immediate feedback. This feedback is a crucial means of improving FCA implementation and relies on a strong relationship with the woreda and region. Regional, woreda, and school level personnel also requested supervision and follow-up.

Collaboration Among Stakeholders (Region to School): One of the most important opportunities for sustainability is to pass ownership of FCA activities to actors at the woreda level (AMPS-17) and to ensure that regional education bureau attributes importance to FCA activities (TGT-04). The entire education structure (woreda, zone, and region education bureau) should collaborate to achieve improved education outcomes (AMPS-05). A woreda expert said, “All stakeholders from region down to classroom teacher should work to sustain FCA. The region should include FCA issues in the yearly strategic plan. It is not a one institution issue. So, regions and woredas should work hard on it continuously to enhance and sustain the methodology (TGWE-07).

To ensure and sustain FCA, school and teachers also need to put in extra time and energy to facilitate change (TGPS-03). Schools must assign teachers based on their field, provide follow-up on FCA implementation, communicate with parents, and take feedback from students about the approach (AMT-21). Teachers also need to help students develop reading skills through different mechanisms, such as encouraging their students to read texts on different subjects.

Other suggestions include engaging parents to follow up with their children (TGPS-25; TGT-13), encouraging other NGOs to participate in FCA implementation (TGWE-04), expanding support services to the rural poor, and engaging the Ethiopian government at all levels, particularly the political leadership (AMWE-02).

Research and Assessment: Participants also proposed studying implementation and systematizing lessons learned and best practices (TGPS-13; TGPS-03; AMPS-13; TGPS-21).

Evaluation of Formative Continuous Assessment

IV. Conclusions and Recommendations 4.1. Conclusions The following conclusions can be drawn from the findings:

▪ Neither FCA training nor FCA tools were fully available in the sampled classrooms. In the classrooms where FCA trained teachers and FCA tools where available, the teachers drew on their training to properly use the FCA tools. Because the FCA training and tools seemed popular in project areas, data collectors identified several instances in which local actors disseminated what they learned from training to nearby schools and departments. Teachers in Grades 1 and 2 seemed to more effectively use the training and tools than their peers in Grades 3 and 4. Data collectors observed a gradual decline in using FCA tools as the grades progress, which is natural because learners become more independent with age. Tigrigna-language classrooms seemed to be more consistent in using FCA tools compared to Amharic-language classrooms. A4 and A6 cards were more frequently used in lower grades, while index cards and pocket charts were more frequently used in the later grades. Sequencing, categorization, and writing skills were the least assessed skills in Grades 1 and 2, while writing and vocabulary were more regularly assessed in Grades 3 and 4. In general, definitions, building vocabularies, comprehension, antonyms, and synonyms are least assessed skills. ▪ FCA tools yielded very encouraging results and contributions. Teachers’ motivation and attitude have remarkably improved; pedagogical methods have become more student-friendly and collaborative; assessment practices are becoming more diagnostic, progressive, and informative; record-keeping is becoming more professional and systematic; and providing feedback has also become more supportive, encouraging, and positive. In the same token, student motivations, behaviors, attendance and participation, and achievements also improved. ▪ In spite of these significant contributions and achievements, there were equally competing challenges that could threaten FCA’s current success. These challenges stem from the design of FCA tools and trainings and stretch all the way to structural factors, classroom practices, and administrative and supervisory activities. These challenges can be addressed by taking advantage of opportunities such as working with the MoE to develop programs and strategies; partnering with other organizations to provide material, technical and professional support; and working with local actors to build school-level opportunities.

Evaluation of Formative Continuous Assessment

4.2. Recommendations What should be done by READ M&E and MoE to properly implement FCA in the future?

The following major recommendations are forwarded based on the findings presented so far:

1. There is strong evidence suggesting that the FCA training and tools can address the early grade reading problems of children in Ethiopian schools. As a result, the FCA approach needs to be scaled up to all schools and classrooms that speak Tigrigna and Amharic. Decisionmakers should give serious thought to scaling up the FCA approach to other language groups of first-cycle primary schools, as well as upper grades and preschools.

2. Decisionmakers need to develop a strategy to integrate the FCA approach with the conventional education approach. For example, they need to determine whether it is more feasible to integrate the FCA approach into the existing pre-service teacher training programs or into the in-service teacher training program. If it becomes part of the in- service program, should it be a separate topic? Or should it be incorporated into other courses that address pedagogy and assessment, to make it more effective? If the FCA approach is to become part of continuous professional development, then decisionmakers need to determine when it should be introduced, to whom, and for how long.

3. The FCA training scope needs to be broadened. FCA training should not be confined to helping trainees understand and practice FCA tools, it also needs to help them understand how to prepare FCA tools with locally available materials.

4. Education offices across all levels – ranging from the federal Ministry of Education to the regional, zonal, and woreda levels – need to own FCA process and integrate it into their plans, activities, budget, supervision, and reporting. FCA training needs to be offered to officers at various levels of the government. Moreover, supervision formats need to incorporate FCA components.

5. Organizing experience sharing programs in cluster schools would also help disseminate and sustain FCA practices in a cost-effective manner.

6. Before READ M&E transfers information about the FCA to the Ministry of Education or educational offices, it should address participants’ concerns about the design of FCA materials in terms of graphics and local relevance. Teachers also suggested that some of the pictures on the flash cards need revision to reflect socio-cultural contexts on the ground. Accordingly, future projects need to consider the match between letters and pictures to reflect the level of children and socio-cultural situations of the respective MT languages.

Evaluation of Formative Continuous Assessment

7. Schools must be prepared for implementing FCA. This includes finding appropriate spaces to store FCA tools, or even preparing new ones. Class size has been repeatedly mentioned as a challenge to implementing FCA. As such, decisionmakers need to identify ways to minimize teachers’ work load so that FCA can be fully and effectively implemented. Establishing a school-level FCA task force would help monitor, evaluate and improve the FCA approach until it becomes well integrated into school practices and routines.

8. FCA implementation faced challenges because teachers received the requisite training, but did not have the tools or means to effectively implement the intervention in some places. Capacity building efforts should be backed by the dissemination of adequate materials and tools, accompanied by strict follow-up and supportive supervision. In particular the utilization of graphic organizers in Grade 3 and 4 need improvement.

9. Decisionmakers must work with local communities to develop mechanisms to leverage locally available resources to prepare FCA materials. This will help minimize the financial and material challenges associated with getting the requisite materials.

10. Teachers need to be trained, encouraged, and supported in their efforts to expand their FCA skills and the tools to non-language subjects.

11. There needs to be a mechanism that recognizes successful teachers and creates healthy competition among teachers.

12. The evaluation revealed that teachers are using the FCA tools and keeping records. However, the teachers are not implementing a standard approach and are not receiving follow-up at all levels. FCA implementation needs to incorporate a mechanism for ensuring a consistent and uniform implementation mechanism.

13. In the context of the existing challenges, the FCA approach is reportedly engaging teachers and reviving a spirit of hard work, commitment and motivation. Such interventions should be ongoing with the full involvement of teachers’ training institutions, education offices, and the MoE.

14. Commitment of supervisors, principals and vice principals is critical for effective implementation of reading improvement interventions. It is important to have strict follow-up system and supportive supervision.

Evaluation of Formative Continuous Assessment

References Abejehu, S.B. (2016). The practice of continuous assessment in primary schools: The case of Chagni, Ethiopia. Journal of Education and Pratice, 7(13). Abiy Yigzaw. (2013). High school English teachers' and Students’ perceptions, attitudes and actual practices of continuous assessment. Academic Journals, 8(16), 1489-1498. Ahmed, R. (2016). The impact of continuous assessments on academic performance: a review and synthesis. In Proceedings of the Allied Academies International Internet Conference 2016 (18), pp. 85-92. From: Allied Academies International Internet Conference 2016, 27-29 July 2016, Toronto, Canada. Airasian, P. W. (1991). Classroom assessment. New York: McGraw-Hill. Alausa, Y. A. (2004). Continuous assessment in our schools: advantages and problems. Namibia: Kolin Foundation Arandis. Aytaged Sisay. (2013). A comparative study on the practice of continuous assessment between Addis Ababa and Unity Universities. Global Journal of Comparative Education, 1 (1), 50- 58. Belay Tefera and Belay Hagos (2018). Know Thy Self: Viability of early childhood education delivery through traditional priest schools. Ethiopian Journal of Education, Vol. 38, N0.2. Birhanu Moges. (2013). Continuous Assessment Issues and Practices in Secondary Schools of Oromia Regional State, Ethiopia: THE “BIG PICTURE” of Assessment Mechanism. Available online: http://www.palgojournals.org Brookhart, S. M. (2008).How to give effect feedback to your students. Association for Supervision and Curriculum Development, Alexandria, Virginia USA. Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education. London: Routledge. Darling-Hammond, L., Wise, A.E., Klein, S.P. (1999). A license to teach: raising standards for teaching. Jossey-Bass Publishers. Desalegn Chalchisa.(2014). Practices of Assessing Graduate Students’ Learning Outcomes in Selected Ethiopian Higher Education Institutions. Journal of International Cooperation in Education, 16(2), 157 180. Ellington, H. and Earl, S. (1997). Making Effective use of continous assessment and portfolios. 1st ed. Aberdeen: The Robert Gordon University. EQ (2003). Measuring pupil achievement: Continuous Assessment. EQ Review, (1)1 http://www.equip123.net/EQ-Review/1_1.pdf.

Evaluation of Formative Continuous Assessment

Esere, M. and Idowu. A. (2009). Continuous Assessment Practices in Nigerian Schools. University of Ilorin, Nigeria: Retrieved from: http://www.iaea.info/documents/paper_2fb222d82.pdf. Federal Democratic Republic Government of Ethiopia. (1994). Education and Training Policy. Addis Ababa, April, 1994. St. George Printing Press Fisseha Mikre (2010). Review Article: The Roles of Assessment in Curriculum Practice and Enhancement of Learning. Ethiopian Journal of Education & Science, 5(2),101-114. Guskey, T.R. (1997). Research Needs to Link Professional Development and Student Learning. Journal of Staff Development, 18(2), 36-40. Girum Tareke, Belay Tefera and Gosim Derib (2018). The Status and Psychological Challenges of Implementation of Continuous Professional Development/CPD/ in Primary Schools of South Wollo Zone. Journal of Mass Communication & Journalism, 8 (3), 1-9. Iqbal, M. and Samiullah, A. (2017). Effect of Continuous Assessment Techniques on Students’ Performance at Elementary level. Bulletin of Education and Research, 39 (1), 91-100. MoE (1994). Education and Training Policy of the Federal Democratic Republic Government of Ethiopia. Addis Ababa: St. George Printing Press. MoE (2015). Education Sector development Programme V (ESDP V) of the Federal Democratic Republic of Ethiopia. Ministry of Education, Addis Ababa. Mpapalika K.(2013). Tanzania science teachers’ practices and challenges in continuous assessment. Learning through Assessment: Assessment for Learning in the Science Classroom. Arlington, Virginia: NSTA Press. Muskin, J.A. (2017). Continuous Assessment for Improved Teaching and Learning: A Critical Review to Inform Policy and Practice. In-Progress Reflection No. 13 on Current and Critical Issues in Curriculum, Learning and Assessment, UNESCO & IBE, IBE/2017/WP/CD/13. National Research Council (2001). Educating Teachers of Science, Mathematics, and Technology: New Practices for the New Millennium. Washington, DC: National Academy Press. Nitko , A.J.(2004). Educational Assessment of Students. 4th ed. Ohio: Merrill Prentice Hall. Prouty, J. D and George. E. S. (2003). Continuous assessment: A practical guide for teachers. American Institute for Research. Sintayehu Belay (2016). The Practice of Continuous Assessment in Primary Schools: The Case of Chagni, Ethiopia. Journal of Education and Practice, 7, (31), 24-30. Tefera Gashaw. (2014). Teachers` Perceptions and Practices of Continuous Assessment in Mathematics Class in Dera woreda General Secondary and Preparatory Schools. MA thesis, Addis Ababa University, Addis Ababa.

Evaluation of Formative Continuous Assessment

Tigistu Alemu, Alemayehu Fanta, and Belay Tefera, (2004). Continuous assessment manual for grade 9 & 10 History. National Organization for Examination, Ministry of Education, Ethiopia. Yiheyis Seyoum and Getachew Seyoum. (2014). The Implementation of Continuous Assessment in Writing Classes the of Jimma College of Teachers Education. Ethiopian Journal of Education & Science, 10(1), 109-135.

Evaluation of Formative Continuous Assessment

Annexes Annex 1: List of Names of Regions, Woredas and Schools This table contains corresponding codes for interviews and discussions with teachers, principals/supervisors, and woreda office expert

Interview Code Region Zone Woreda Name of School MT Principal & Woreda Teachers Supervisor Expert 1 Amhara West Gojjam Jabitenan Genet AMT-03 AMPS-03 2 Amhara West Gojjam Jabitenan Mankusa AMT-04 AMPS-04 AMWE-01 3 Amhara West Gojjam Jabitenan Geray AMT-01 AMPS-01 4 Amhara West Gojjam Jabitenan Zindib AMT-02 AMPS-02 5 Amhara West Gojjam Guagusa Absela AMT-05 AMPS-05 6 Amhara West Gojjam Guagusa Hibret AMT-06 AMPS-06 AMWE-02 7 Amhara West Gojjam Guagusa Tilili AMT-08 AMPS-08 8 Amhara West Gojjam Guagusa Kili AMT-07 AMPS-07 9 Amhara South Gondar Farta Hiruy Abaregay AMT-20 AMPS-20 10 Amhara South Gondar Farta Soras AMT-17 AMPS-17 AMWE-05 11 Amhara South Gondar Farta Tigle Fire AMT-18 AMPS-18 12 Amhara South Gondar Farta Talda AMT-19 AMPS-19 13 Amhara South Gondar Dera Anbesame AMT-21 AMPS-21 14 Amhara South Gondar Dera Goha AMT-22 AMPS-22 AMWE-06 15 Amhara South Gondar Dera Emashenkoro AMT-24 AMPS-24 16 Amhara South Gondar Dera Mutansa AMT-23 AMPS-23 17 Amhara North Shewa Basso Baso 1st cycle AMT-09 AMP-09 18 Amhara North Shewa Basso Bakelo AMT-12 AMP-12 AMWE-03 19 Amhara North Shewa Basso Keyit AMT-11 AMP-11 20 Amhara North Shewa Basso Wushawushign AMT-10 AMP-10 21 Amhara North Shewa Debre Berhan Zanjira AMT-15 AMPS-15 22 Amhara North Shewa Debre Berhan Andinet AMT-16 AMPS-16 AMWE-04 23 Amhara North Shewa Debre Berhan Biruh Tesfa AMT-14 AMPS-14 24 Amhara North Shewa Debre Berhan Zeryacob AMT-13 AMPS-13 25 Amhara South Wollo Kombolcha Shishaber AMT-25 AMPS-25 26 Amhara South Wollo Kombolcha Kombolcha Junior AMT-29 AMP-29 AMWE-07 27 Amhara South Wollo Kombolcha Kombolcha N02 AMT-26 AMPS-26 28 Amhara South Wollo Kombolcha Yekatit 25/67 AMT-27 AMPS-28 29 Amhara South Wollo Legambo Tulu Awalia AMT-30 AMPS-31 30 Amhara South Wollo Legambo Chiro AMT-29 AMPS-29 AMWE-08 31 Amhara South Wollo Legambo Akesta AMT-31 AMPS-32

Evaluation of Formative Continuous Assessment

Interview Code Region Zone Woreda Name of School MT Principal & Woreda Teachers Supervisor Expert 32 Amhara South Wollo Legambo Terad AMT-32 AMPS-30 Central Tigray Adwa Zuria Mayguagua TGT-17 TGPS-17 33 Tigray Central Tigray Adwa Zuria Soloda TGT-18 TGPS-18 34 Tigray TGWE-05 Central Tigray Adwa Zuria Adi Abieto TGT-19 TGPS-19 35 Tigray Central Tigray Adwa Zuria Tsion TGT-20 TGPS-20 36 Tigray Central Tigray Axum Zuria Mai Tuum TGT-23 TGPS-23 37 Tigray Central Tigray Axum Zuria Ena Mered TGT-22 TGS-22 38 Tigray TGWE-24 Central Tigray Axum Zuria Sero TGT-24 TGS-24 39 Tigray Central Tigray Axum Zuria Mai Tsadik TGT-21 TGPS-21 40 Tigray 41 Tigray South Tigray Endamekoni Meswuaeeti TGT-10 TGPS-10 42 Tigray South Tigray Endamekoni Tesfaye Ferede TGT-12 TGPS-12 TGWE-03 43 Tigray South Tigray Endamekoni Ashara TGT-09 TGPS-09 44 Tigray South Tigray Endamekoni Endergan TGT-11 TGPS-11 45 Tigray South Tigray Raya Azebo Zibandas TGT-13 TGPS-13 46 Tigray South Tigray Raya Azebo Baso TGT-15 - TGWE-04 47 Tigray South Tigray Raya Azebo Eyob TGT-14 - 48 Tigray South Tigray Raya Azebo Boye TGT-16 TGPS-16 Mekelle Tigray Quiha Quiha TGT-02 TGPS-02 49 Zuria TGWE-02 Mekelle Tigray Quiha Weldu Tatewos TGT-01 TGPS-01 50 Zuria Mekelle Tigray Semen Lekatit II TGT-04 TGPS-04 51 Zuria TGWE-04 Mekelle Tigray Semen Elala TGT-03 TGPS-03 52 Zuria Mekelle Tigray Hadnet Metkel TGT-30 TGPS-30 53 Zuria TGWE-08 Mekelle Tigray Hadnet Hadnet TGT-29 TGPS-29 54 Zuria

Evaluation of Formative Continuous Assessment

Interview Code Region Zone Woreda Name of School MT Principal & Woreda Teachers Supervisor Expert Mekelle Tigray Ayder Fre Siwat TGT-26 TGPS-26 55 Zuria TGWE-07 Mekelle Tigray Ayder Ayder TGT-25 TGPS-25 56 Zuria

Evaluation of Formative Continuous Assessment

Annex 2: Common Themes of the Interview Protocols 1. Using FCA and Reasons: Do you use FCA tools delivered by READ M&E for assessment purposes? Would you please explain T? Probe. Do teachers use the FCA tools delivered by READ M&E for assessment purposes; would you please explain P? In your experience, are teachers using these FCA tools for Assessment purposes? How do you know W? 2. Contribution of FCA and FCA training: What changes did you observe on your students learning and achievement after implementing the skills you received from the FCA training? Is there anything promising T? Do you believe that the FCA training delivered and the experience sharing forum conducted last year enhanced teacher’s understanding of FCA for assessment, record keeping, and providing feedback P? What kind of improvements in children’s reading ability have you seen as a result of the teachers implementing the skills gained from the FCA training W? Do you believe FCA training and experience sharing enhanced your understanding of FCA for assessment, record keeping and feedback? 3. Record keeping, feedback and Follow up: How do you keep record of and communicate feedback the progress of your students T? How do you follow up and support MT teachers P? What kind of follow up and support do you provide for MT teachers W? 4. Challenges: What are the challenges you encounter while using FCA tools for assessment in your classroom T? What challenges did teachers encounter while using FCA for assessment purposes in their classrooms? What challenges did schools in your woreda encounter while implementing FCA? What solutions were sought W? 5. Opportunities: What opportunities are there for using FCA tools for assessment in your classroom T? What opportunities are there for using FCA in the classrooms P? 6. What to be done to enhance FCA: In your opinion, what should be done to enhance and sustain the use of FCA in your school, woreda, and region respectively P? In your opinion, what should be done to enhance and sustain the use of FCA in your school in woreda, and region respectively P? In your opinion, what should be done to enhance and sustain the use of FCA in your school, woreda, and region respectively W?

Extra issues

1. Any other: Is there anything you would like to add T? Is there anything you would like to add P? Is there anything you would like to add W? What overall comments do you have on the use of FCA for assessment purposes W?

2. What do you know about the FCA training delivered and materials provided by READ M&E to teachers in your woreda?

3. Did the WEO cascade the FCA intervention to other schools in the woreda? Why or why not? If so, did the schools find the intervention useful W?

Evaluation of Formative Continuous Assessment

Annex 3: Classroom Observation Guide for READ M&E’s FCA Pilot (Grades 1 & 2) This guide is prepared to observe classrooms regarding implementation of FCA in READ M&E supported schools. The observations gathered using this guide helps READ M&E to follow up, support, and report about what is happening at school level in relation to the teachers’ training delivered and material support provided earlier to improve the reading proficiency of children in the early grades.

This classroom observation guide attempts to get data on the following five constructs: I. Availability of FCA tools in the classroom /in the hands of the teacher II. Frequency of the FCA tools used in the classroom III. Implementation of the FCA tools for assessment purpose IV. Remediation strategy V. Feedback

Background information 1. Region: ______Zone: ______Woreda: ______2. Name of the School: ______3. Date of Observation: ______Name of Observer: ______4. Sex: ______Age:______5. Has the observed teacher received the FCA training delivered by READ M&E? YES___ NO__ 6. Week & Lesson Observed: ______Grade observed: ______7. Total number of enrolled students in the classroom: ______( ___ boys and ____ girl) 8. Number of students in the classroom during observation: Boys: ______Girls: ______

Evaluation of Formative Continuous Assessment

I. Availability of FCA tools in the classroom Instructions: Put a tick mark (√) for your observation

Table 1. FCA tools available in the hands of the teacher or in the classroom during observation Availability S.N. FCA Tools Remarks Yes No 1 A6 Flash card 2 A4 Flash card 3 Index card 4 Pocket Chart 5 Letter recognition record sheet

Please mention here if you observe any other tool ______

II. Frequency of the FCA tools used in the classroom Instructions: Put a tick mark (√) for your observation of the instructional process (for a single period 40 – 45 minutes).

Table 2: Number of times or frequency of use of the tools during observation Assessment Tools Not 1x 2x 3x 4x 5x More than Used 5x A4 Flash Card A6 Flash Card Index Card Pocket Chart Other Tools Used (if any) A. B.

Evaluation of Formative Continuous Assessment

III. Implementation of FCA Tools for Assessment 1. Number of students assessed or asked to respond to teacher’s questions during the observation period.

Boys: ______Girls: ______

2. What skills were assessed using the FCA tools? (N.B. more than one responses are possible).

Instructions: Put a tick mark (√) for your observation of the instructional process (for a single period 40 – 45 minutes).

Table 3: Skills assessed during classroom observation S.N Skills Assessed Tools utilized Remark Flash Index Card Pocket chart card 1 Letter identification A4 A6 2 Vocabulary 3 Blending 4 Segmenting 5 Matching sounds with pictures 6 Reading vocabulary 7 Sequencing 8 Categorization 9 Writing

3. Has the teacher keep records of each student after assessment? Yes: No:

If yes, which tools were used? 1 Writing rubric Letter identification recording 2 sheet 3 Teacher’s notebook 4 Other Please specify

Evaluation of Formative Continuous Assessment

IV. Remediation Strategies Instructions: Put a tick mark (√) for your observation of the instructional process (for a single period 40 – 45 minutes).

Table 4: Remediation strategies used S/N Remediation strategy observed Yes No

1 Re-teaching 2 Alternative instructional strategies 3 Task analysis 4 Small group instruction 5 One-to-one instruction for individual student 6 Additional practice

V. Feedback Instructions: Put a tick mark (√) for your observation on the teacher’s provision of feedback during the specific period of observation.

Table 5: Feedback Observed Feedback Yes No 1 Has the teacher give constructive feedback to students? 1.1 Is the feedback specific and rather than general 1.2 Descriptive rather than judgmental 1.3 Share information rather than giving advice 1.4 Positive constructive feedback 1.5 Negative constructive feedback 2 Monitors and supports learners through immediate feedback

Evaluation of Formative Continuous Assessment

VI. Additional Comments

1. Please provide feedback on observed strengths and weaknesses during the classroom observation

Strengths observed in: • Using the distributed materials ______

• Implementation of skills observed during classroom observation gained from training ______

Weaknesses observed in:

• Using the distributed materials ______

• Implementation of skills observed during classroom observation gained from training ______

Name of Observer: ______Signature: ______Date:______

Evaluation of Formative Continuous Assessment

Annex 4: Classroom Observation Checklist for READ M&E’s FCA Pilot (Grades 3 & 4) This guide is prepared to observe classrooms regarding implementation of FCA in READ M&E supported schools. The data gathered using this guide helps READ M&E to follow up, support, and report about what is happening at school level in relation to the teachers’ training delivered and material support provided earlier to improve the reading proficiency of children in the early grades.

This classroom observation guide attempts to get data on the following five constructs: VI. Availability of FCA tools in the classroom /in the hands of the teacher VII. Frequency of the FCA tools used in the classroom VIII. Implementation of the FCA tools for assessment purpose IX. Remediation strategy X. Feedback

Background information 9. Region: ______Zone: ______Woreda: ______10. Name of the School: ______11. Date of Observation: ______Name of Observer: ______12. Sex: ______Age:______13. Has the observed teacher received the FCA training delivered by READ M&E? YES____ NO___ 14. Week & Lesson Observed: ______Grade observed: ______15. Total number of enrolled students in the classroom: ______(_____boys and ____ girls) 16. Number of students in the classroom during observation: Boys: ______Girls: ______

Evaluation of Formative Continuous Assessment

I. Availability of FCA tools in the classroom during observation

Instructions: Put a tick mark (√) for your appropriate response

Table 1: FCA Tools Available During Observation

Availability S.N. FCA Tools Remark Yes No 1 Graphic Organizer 2 Writing Rubric 3 Index card

Please state here if the teacher used any other tools ______

II. Frequency of FCA Tool Use in the Classroom

Instructions: Put a tick mark (√) for what you have observed during the specific period of observation

Table 2: Frequency of Tool Use During Observation Assessment Tools Not 1x 2x 3x 4x 5x More than Used 5x Graphic Organizer Index Card Pocket Chart Other Tools Used (if any) A. B.

Evaluation of Formative Continuous Assessment

III. Implementation of FCA tools for assessment purpose

Number of students assessed or asked to respond to teachers’ questions Boys: ______Girls: ______Table 3: Skills Assessed During Classroom Observations Tools Utilized Flash S.N. Skill Assessed Graphic Index Pocket Remarks card Organizer Card Chart A6 A4 1 Writing 2 Reading 3 Listening 4 Vocabulary 5 Comprehension 6 Developing definition Developing vocabulary and 7 phrases 8 Synonyms 9 Antonyms 10 Blending 11 Segmenting 12 Word formation 13 Segmenting N.B. More than one response is possible

Has the teacher keep record of each student after assessment?

Yes: No: If yes, which tools were used? 1 Writing rubric 2 Teacher’s notebook 3 Other Please specify

Evaluation of Formative Continuous Assessment

IV. Remediation Strategy Instructions: Put a tick mark (√) for your observation on the teacher’s use of remediation strategy during the specific period of observation

Table 4: Remediation Strategies Observed S.N. Remediation strategy observed Yes No 1 Re-teaching 2 Alternative instructional strategies 3 Task analysis 4 Small group instruction One-to-one instruction for individual 5 student 6 Additional practice

V. Feedback Instructions: Put a tick mark (√) for your observation on the teacher’s provision of feedback during the specific period of observation

Table 5: Feedback Observed Feedback Yes No S.N. 1 Did the teacher give constructive feedback to students 1.1 Is the feedback specific and rather than general 1.2 Descriptive rather than judgmental 1.3 Share information rather than giving advice 1.4 Positive constructive feedback 1.5 Negative constructive feedback 2 Monitors and supports learners through immediate feedback

VI. Additional Comments

Please provide feedback on observed strengths and weaknesses during classroom observation

Strengths observed in:

• Using the distributed materials ______

• Implementation of skills observed during classroom observation gained from training ______

Evaluation of Formative Continuous Assessment

Weaknesses observed in:

• Using the distributed materials ______

• Implementation of skills observed during classroom observation gained from training ______

Name of Observer: ______Signature:______Date:______

Evaluation of Formative Continuous Assessment

Annex 5: Group Discussion with Teachers

Interviewer: Please obtain verbal consent from the interviewees/discussants Material required: Voice recorder

Interviewer’s name: ______Code of participant: ______Region: ______Woreda ______Cell phone of the interviewer: ______Date of interview: ______Time of interview: Started ______Completed ______

My name is (name of the interviewer) and I will be talking with you today. I work for READ M&E. READ M&E/AIR intends to collect data on the implementation of the skills received from the FCA training and the materials provided by READ M&E earlier.

The purpose of this interview is to learn about the implementation of FCA in your school. I will be recording this conversation to help capture what you say accurately. READ M&E will treat all the information gathered confidentially and keeps your name private and separate. Participation in this interview is voluntary and you do not have to answer any question you do not want to answer.

The interview will last approximately 45 minutes. Do you agree to participate in this interview?

• Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview.

Thank you!

Evaluation of Formative Continuous Assessment

Background Information 1. Name of the School: ______2. Number of participants: ______3. Have participants received FCA training READ M&E has provided? YES____ NO___

Table 1: Information on Participants Level of S.N. Name/Code Sex Age Grade Remark Education

Group Discussion Questions

1. Do you use FCA tools delivered by READ M&E for assessment purposes? Would you please explain? Probe 2. What changes did you observe on your students learning and achievement after implementing the skills you received from the FCA training? Is there anything promising? 3. How do you keep record of the progress of your students? 4. How do you communicate feedback to your students? 5. Do you believe FCA training and experience sharing enhanced your understanding of FCA for assessment, record keeping and feedback? 6. What are the challenges you encounter while using FCA tools for assessment in your classroom? 7. What opportunities are there for using FCA tools for assessment in your classroom? 8. In your opinion, what should be done to enhance and sustain the use of FCA in your school, woreda, and region respectively? 9. Is there anything you would like to add?

Evaluation of Formative Continuous Assessment

Annex 6: Group Discussion Questions for Supervisors, Principals, and Vice Principals Interviewer: Please obtain verbal consent from the interviewees/discussants Material required: Voice recorder Interviewer’s name: ______Code of participant: ______Region: ______Woreda ______Cell phone of the interviewer: ______Date of interview: ______Time of interview: Started ______Completed ______

The interview will last approximately 45 minutes. Do you agree to participate in this interview?

• Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview.

Thank you!

Evaluation of Formative Continuous Assessment

I. Background information 1. Name of the School: ______2. Number of participants: ______3. Have participants received FCA training READ M&E has provided? YES____ NO____

Table 1: Information on Participants Level of S.N. Name/Code Sex Age Grade Remark Education

II. Group Discussion Questions 1. Do teachers use the FCA tools delivered by READ M&E for assessment purposes? Would you please explain? 2. How do you follow up and support MT teachers? 3. Do you believe that the FCA training delivered and the experience sharing forum conducted last year enhanced teacher’s understanding of FCA for assessment, record keeping, and providing feedback?

4. What challenges did teachers encounter while using FCA for assessment purposes in their classrooms? What opportunities are there for using FCA in the classrooms? 5. In your opinion, what should be done to enhance and sustain the use of FCA in your school in woreda, and region respectively? 6. Is there anything you would like to add?

Evaluation of Formative Continuous Assessment

Annex 7: Interview Guide for Woreda Education Experts Who Took the READ M&E FCA Training

Interviewer: Please obtain verbal consent from the interviewees/discussants Material required: Voice recorder Interviewer’s name: ______Code of participant: ______Region: ______Woreda ______Cell phone of the interviewer: ______Date of interview: ______Time of interview: Started ______Completed ______

My name is (name of the interviewer) and I will be talking with you today. I work for READ M&E. READ M&E /AIR intends to collect data on the implementation of the skills received from the FCA training and the materials provided by READ M&E earlier.

The interview will last approximately 45 minutes. Do you agree to participate in this interview?

• Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview.

Thank you!

Evaluation of Formative Continuous Assessment

I. Interviewee Background Data

1. Gender ______2. Age ______3. Place of work ______4. Current position ______5. Year of service ______6. Highest level of education attained ______

II. Interview Items

1. What do you know about the FCA training delivered and materials provided by READ M&E to teachers in your woreda? 2. In your experience, are teachers using these FCA tools for Assessment purposes? How do you know? 3. What kind of follow up and support do you provide for MT teachers? 4. What kind of improvements in children’s reading ability have you seen as a result of the teachers implementing the skills gained from the FCA training? 5. What challenges did schools in your woreda encounter while implementing FCA? What solutions were sought? 6. Did the WEO cascade the FCA intervention to other schools in the woreda? Why or why not? If so, did the schools find the intervention useful? 7. What overall comments do you have on the use of FCA for assessment purposes? 8. In your opinion, what should be done to enhance and sustain the use of FCA in your school, woreda, and region respectively? 9. Is there anything you would like to add?

Evaluation of Formative Continuous Assessment

Annex 8: FCA Feedback Communication Form READ M&E prepared this form to provide formal feedback regarding the implementation of FCA to the visited MT teachers. Assessors are expected to build their feedback based on the information they gathered by Tool I &Tool II of FCA II assessment. Assessors are also expected to provide technical support to MT teachers using their own expertise, experiences and training received from READ M&E Project. Assessors, once providing the support and the constructive feedbacks, will fill this form and share one copy to the MT teachers and the other copy to READ M&E Project together with other completed forms. Assessors will provide feedbacks to the MT teachers in one group or individually depending on the timing and the context.

1. Observed strengths and weaknesses during supervision

Strengths observed in:

• Using the distributed materials

• Implementation of skills observed during classroom observation gained from training

Weaknesses observed in:

• Using the distributed materials

• Implementation of skills observed during classroom observation gained from training

2. Technical supports provided

Evaluation of Formative Continuous Assessment

3. Consensus reached

4. Action Points

Name of Teacher: ______Signature: ______Date:______Name of Observer: ______Signature: ______Date:______Stamp of the school: