USAID SINDH READING PROGRAM

EARLY GRADE READING AND MATHEMATICS ASSESSMENT BASELINE REPORT SINDH PROVINCE, KASHMORE DISTRICT

Contract No. AID-391-C-14-00001

Submission Date: 2 September 2015

This document was produced for review by the United States Agency for International Development. It was prepared by Chemonics International, Inc. and School-to-School International. EARLY GRADE READING AND MATHEMATICS ASSESSMENT BASELINE STUDY REPORT SINDH PROVINCE, KASHMORE DISTRICT

Contracted under Order No. AID-391-C-14-00001

DISCLAIMER The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

ACKNOWLEDGEMENTS

The School-to-School International and Chemonics teams would like to thank USAID/Pakistan for their support. We would also like to recognize the stellar faculty and staff from the Sindh Education and Literacy Department, the Bureau of Curriculum, the Provincial Education Assessment Centre (PEACE) Sindh, and the Provincial Institute of Teacher Education (PITE) Sindh. Finally, this endeavor would not have been possible without the dedication of the Master Trainers, Quality Control Officers, data collection partners, subject experts and administrators who lent their continuous support throughout the assessment activity.

The following persons played a major role in the process:

Xxx (Chemonics International) Xxx (Chemonics International) Xxx (Chemonics International) Xxx (Chemonics International) Xxx (Chemonics International)

Mark Lynd (School-to-School International) Aftab Khushk (School-to-School International) Hetal Thukral (School-to-School International) Beth Fincham (School-to-School International) Gaëlle Simon (School-to-School International)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 i

CONTENTS

Contents ...... ii Acronyms ...... vi Executive Summary ...... 1 Chapter 1: Introduction ...... 11 Chapter 2: Methodology ...... 15 Chapter 3: EGRA Results – Kashmore District ...... 23 Chapter 4: EGMA Results – Kashmore District ...... 37 Chapter 5: Contextual Variables and EGRA and EGMA Performance ...... 44 Annexes ...... 49

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 ii

List of Tables and Figures

Table 1: Baseline Schools in Kashmore ...... 16 Table 2: Baseline Participants in Kashmore ...... 16 Table 3: EGRA Test Structure ...... 17 Table 4: EGMA Test Structure ...... 18 Table 5: Enumerators training by District ...... 20 Table 6: Reliability Estimates ...... 22 Table 7: EGRA Percent Correct Scores by Grade and Task ...... 24 Table 8: Average Phonics and Reading Fluency by Grade ...... 25 Table 9: Average Phonics and Reading Fluency by Grade and Gender ...... 28 Table 10: Fluency Rates by PRP Thresholds And Grade ...... 29 Table 11: Grade 3 Reading Fluency and Reading Comprehension ...... 30 Table 12: Grade 5 Reading Fluency and Reading Comprehension (Grade 2-level Text) ...... 31 Table 13: Grade 5 Reading Fluency and Reading Comprehension (Grade 4-level Text) ...... 32 Table 14: EGRA Proportion of Students with Zero-scores by Grade and Task ...... 33 Table 15: EGRA Proportion of Students with Zero-scores by Grade, Gender and Task ...... 33 Table 16: EGMA Percent Correct Scores by Grade and Task ...... 38 Table 17: Mathematics Fluency Rates by Grade ...... 39 Table 18: Mathematics Computation Fluency Rate Means by Grade and Gender ...... 41 Table 19: EGMA Proportion of Students with Zero-scores by Grade and Task ...... 42 Table 20: EGMA Proportion of Zero-scores by Grade, Gender and Task ...... 42 Table 21: Contextual variables from Student Questionnaire Associated with EGRA or EGMA Performance ...... 45 Table 22: Contextual variables from Teacher and Head Teacher Questionnaire Associated with EGRA or EGMA Performance ...... 46 Table 23: Kashmore Reliability Estimates ...... 49 Table 24: EGRA Tasks Statistics for Kashmore ...... 49 Table 25: EGMA Tasks Statistics for Kashmore ...... 50 Table 26: EGRA Grade 3 Score Ranges and Calculations ...... 51 Table 27: EGRA Grade 5 Score Ranges and Calculations ...... 52 Table 28: Example of EGRA Percent Correct and Summary Scores ...... 52 Table 29: Example of EGRA Timed Task Scores ...... 52 Table 30: EGMA Grade 3 Score Ranges and Calculations ...... 53 Table 31: EGMA Grade 5 Score Ranges and Calculations ...... 53 Table 32: Example of EGMA Percent Correct and Summary Scores ...... 54 Table 33: Example of EGMA Timed Task Scores ...... 54 Table 34: Complete EGRA Item Statistics by Grade for Kashmore ...... 55 Table 35: Complete EGMA Item Statistics by Grade for Kashmore ...... 56 Table 36: Distribution of Students By Primary Language Spoken At Home ...... 60 Table 37: Distribution of Students Speaking the Same Language at Home as the Test ...... 60 Table 38: Average EGRA/EGMA Scores by Reading Material at Home ...... 61 Table 39: Percentage of students by Stories Read Aloud to Student at Home...... 61 Table 40: Average EGRA/EGMA Scores by Practice Reading Stories Aloud to Someone at Home...... 62 Table 41: Average EGRA/EGMA Scores by Silent Reading at Home ...... 62 Table 42: Average EGRA/EGMA Scores by Read Quran at Home ...... 62 Table 43: Average EGRA/EGMA Scores by Study or Complete Classwork With Classmates ...... 63

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 iii

Table 44: Average EGRA/EGMA Scores by students Missing School Days During the Previous Week 63 Table 45: Average EGRA/EGMA Scores by Parent’s Knowledge of School Performance ...... 63 Table 46: Distribution of students by students Reporting Working Before or After School ...... 64 Table 47: Average EGRA/EGMA Scores by Students Who Watch Television at Home ...... 64 Table 48: Average EGRA/EGMA Scores by Students Listening to the Radio at Home ...... 64 Table 49: Distribution of students by Computer Access at Home ...... 65 Table 50: Distribution of students by Type of Vehicles at Home ...... 65 Table 51: Distribution of Teachers’ Academic Qualifications ...... 65 Table 52: Distribution of Teachers’ Professional Qualifications ...... 66 Table 53: Average EGRA/EGMA Scores by Teaching Multi-grade Classes ...... 66 Table 54: Average EGRA/EGMA Scores by Teachers Using Books Other Than Textbooks in the Classroom ...... 66 Table 55: Distribution of Classroom Tools used by teachers ...... 67 Table 56: Distribution of Teaching Practices ...... 67 Table 57: Methods by which Teachers’ Udate their Knowledge of Teaching ...... 68 Table 58: Distribution of Head Teachers’ Academic Qualifications ...... 68 Table 59: Distribution of Head Teachers’ Professional Qualifications ...... 68 Table 60: Distribution of Head Teachers Teaching Classes ...... 69 Table 61: Distribution of Head Teachers Who Received Training for Implementing Reading Programs . 69 Table 62: Distribution of Head Teachers Who Received Training for Implementing Mathematics Programs ...... 69 Table 63: Distribution of Schools having a Timetable ...... 69 Table 64: Average EGRA/EGMA Scores by School timetable ...... 70 Table 65: Average EGRA/EGMA Scores by Including Reading in the timetable ...... 70 Table 66: Schools by Including Math in the timetable ...... 70 Table 67: Distribution of Head Teachers Supporting Reading Instruction ...... 71 Table 68: Distribution of Head Teachers Supporting Mathematics Instruction ...... 71 Table 69: Average EGRA/EGMA Scores by Head Teacher’s Knowledge About Student Progress ...... 71 Table 70: Average EGRA/EGMA Scores by Head Teachers’ ObservinG teachers’ use of Lesson Plans 72 Table 71: Distribution of The Person Responsible for Observing Teachers in the Classroom ...... 72 Table 72: Distribution of SMC Decision Making Authority/Responsibility ...... 72 Table 73: Distribution of Facility Improvements ...... 73

Figure 1: Grade 3 EGRA: Percent Correct Scores by Task ...... 3 Figure 2: Grade 3 EGRA: Proportion of Zero-Score Students by Task ...... 4 Figure 3: Grade 5 EGRA: Percent Correct Scores by Task ...... 4 Figure 4: Grade 5 EGRA: Proportion of Zero-Score Students by Task ...... 5 Figure 5: Grade 3 EGMA: Percent Correct Scores by Task ...... 6 Figure 6: Grade 3 EGMA: Proportion of Zero-Score Students by Task ...... 6 Figure 7: Grade 5 EGMA: Percent Correct Scores by Task ...... 6 Figure 8: Grade 5 EGMA: Proportion of Zero-Score Students by Task ...... 7 Figure 9: Evaluation Design ...... 13 Figure 10: Distribution of EGRA Scores ...... 23 Figure 11: Grade 3 Scores by Task and Gender ...... 27 Figure 12: Grade 5 Scores by Task and GendeR ...... 28 Figure 13: Grade 3 Reading Performance levels and Reading Comprehension Intervals ...... 30 Figure 14: Grade 5 Reading Performance levels and Comprehension Intervals (Grade 2-level text) ...... 31 Figure 16: EGRA Summary scores by District ...... 34

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 iv

Figure 17: Distribution of EGMA Scores ...... 37 Figure 18: Grade 3 EGMA Scores by Gender ...... 40 Figure 19: Grade 5 EGMA Scores by Gender ...... 41 Figure 20: Explanation of Box Plots ...... 57 Figure 21: Phonics and Reading-Rate Fluency Box Plots for Grade 3 ...... 58 Figure 22: Phonics and Reading-Rate Fluency Box Plots for Grade 5 ...... 58 Figure 23: Mathematics Fact-Rate Fluency Box Plots for Grade 3 Figure 24: Mathematics Fact-Rate Fluency Box Plots for Grade 5 ...... 59

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 v

ACRONYMS

AEO Area/Assistant Education Officers BoC Bureau of Curriculum CWPM Correct Words Per Minute EGMA Early Grade Mathematics Assessment EGRA Early Grade Reading Assessment GoS Government of Sindh I-SAPS Institute of Social and Policy Studies MDC Management and Development Center MT Master Trainer PEACE Provincial Education Assessment Centre PITE Provincial Institute of Teacher Education PMIU Program Monitoring and Implementation Unit PRP Pakistan Reading Project PTC Primary Teaching Certificate QCO Quality Control Officer SBEP Sindh Basic Education Project SEMIS Sindh Education Management Information System SMC School Management Committee SRP Sindh Reading Program STBB Sindh Textbook Board STS School-to-School International USAID United States Agency for International Development

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 vi

EXECUTIVE SUMMARY

Introduction The Sindh Reading Program (SRP) is a collaborative initiative of the Government of Sindh (GoS) and USAID/Pakistan that aims to help 410,400 students in Sindh province acquire the foundational reading and mathematics skills in the early grades to set them up for success in later grades (the project’s primary emphasis is reading). To this end, SRP’s interventions include teacher training to improve early grade reading instruction and assessment practices; the provision of additional reading materials; and activities to enhance parent and caregiver engagement in early literacy for students in Grades 2 and 4. 1 SRP’s interventions are being implemented across eight districts in the province of Sindh in Pakistan: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and five towns in Karachi. By 2018 (year 5 of the program), one of SRP’s key indicators is to improve reading for at least 200,887 Grade 2 students.2 To determine the extent to which students’ reading skills improve as a result of participating in SRP interventions, an evaluation was designed to assess students’ reading skills at three time-points in the project’s life cycle – at baseline (i.e., prior to the start of SRP interventions), at midline (i.e., at the halfway point of the SRP project in 2016) and at endline (i.e., at the conclusion of SRP interventions in 2018).3 This report provides results of the baseline conducted in September 2014; midline and endline results will be compared to this baseline in order to measure the impact of SRP interventions. The baseline draws on data collected using the Early Grade Reading Assessment (EGRA) in order to assess students’ reading skills, as well as the Early Grade Mathematics Assessment (EGMA) in order to assess their mathematics skills. The baseline also included interviews with students, teachers and Head Teachers in order to capture contextual variables that may be associated with strong reading and mathematics outcomes.4 This report summarizes the results of the baseline conducted in Kashmore district. It describes the baseline methodology; results for EGRA and EGMA, and teacher and head teacher questionnaires; and recommendations for improving SRP’s interventions.

Methodology Sampling. The EGRA and EGMA assessments were administered in September 2014 to 12,793 students in 560 schools across eight districts of Sindh Province that were participating in SRP: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur – assessed in Sindhi-language - and five towns in Karachi (which is treated as a district for this study, assessed in Urdu-language). To select students for participation in the baseline student assessments, a sample of students were selected using a

1 In its original design, along with a focus on improving reading skills among students in the early grades, SRP also included interventions to improve student’s numeracy skills. This design was subsequently revised to retain only the core indicator of improving reading skills in the early grades; however, this revision came after baseline data had already been collected. As such, baseline data on both students’ literacy and numeracy were collected and analyzed; at midline and endline, data on literacy only will be collected. 2 These targets reflect revisions as of April 2015. 3 The evaluation utilizes a non-experimental pre-test/post-test design, in which students participating in the intervention are assessed prior to the start of the interventions (pre-test) and following the conclusion of the interventions (post-test); additionally, a mid-point measure is also collected. This design was appropriate for this study since an adequate comparison group (i.e., students in schools not participating in SRP interventions, but similar otherwise) was not feasible. A comparison group is necessary in order to establish impact of the interventions (in other words, to be able to attribute all the changes from pre-test to post-test among participating students to the SRP interventions). 4 Student questions were related to language spoken at home, availability of reading materials at home, study habits, access to communication technology and vehicles at home. Teacher and head teacher questions were related to qualifications, attendance at trainings, scheduling, teacher support, head teacher knowledge of student performance, teacher observation procedures, School Management Council (SMC) authority and facility improvements.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 1 stratified random sampling approach, making it possible to generalize the results of this baseline to all students in SRP Sindhi-language schools in the seven districts and all students in SRP Urdu-language schools in the five towns participating in SRP in Karachi City.

The assessment tools were successfully administered in 70 schools to 1,519 students (90 percent of the goal), 62 Grade 3 and 63 Grade 5 teachers (89 percent of the goal), and 69 head teachers (99 percent). The percent of teachers sampled was slightly lower than the sampling rate of head teachers because some teachers teach both third and fifth grade.

It should be noted that the results from this baseline are considered a measure of learning at the end of Grades 2 and 4, even though the baseline was conducted for Grades 3 and 5 students. The decision to conduct the baseline with this population was made because of the requirement to conduct the baseline before the startup of project activities in Year 1. This timing presents a complication – the effects of “summer learning loss” – i.e., that students typically forget some of what they have learned in a given school year over the summer. The extent of this loss cannot be estimated and the results presented here may in fact be underestimating the actual performance of students prior to the break. In other words, assessments conducted after the summer break will reflect a short-term decline in student achievement. Note that SRP midline and endline assessments will be conducted at the same point in the school year in order to ensure comparability of results across time points.

Instrument Development. The baseline EGRA and EGMA tools were developed by SRP to measure core reading and mathematics skills targeted by SRP in its teacher training, instructional materials, and parental involvement interventions. Six types of reading skills were tested with the EGRA: phonemic awareness, phonics, vocabulary, fluency (tested, as appropriate, using passages at two different reading levels: Grade 2 and Grade 4), reading comprehension (Grade 2 and Grade 4-level text) and listening comprehension. Seven types of mathematics skills were tested with the EGMA: number identification, number discrimination, word problems, missing number, computation and multiplication. With both tools, fluency rates were calculated for tasks that were timed. Students were given a set of tasks to perform for each of these skills. EGRA and EGMA tools were developed in “test adaptation workshops,” then piloted and revised prior to baseline data collection.

Data Collection. Baseline data were collected from September 2 to September 26, 2014. Students were tested on paper formats for the baseline; subsequent data collections will be conducted using tablets. Data were analyzed by SRP staff, and the results were reported by district and language (Sindhi and Urdu).

Quality of Assessment Tools • The quality of the EGRA and EGMA tools was measured by examining their internal consistency, item difficulty, and item-total correlations. 5 Based on the results of the pilot, tasks were revised to ensure high internal consistency, appropriate levels of difficulty (allowing sufficient space to capture growth over the life of the project), and adequate item discrimination. • Analyses of test quality showed that the EGRA and EGMA baseline instruments were of high- quality and can be used as a reference for the assessments that will be developed for use at midline and at endline.6

5 Internal consistency (Cronbach’s alpha) indicates whether the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and numeracy, respectively; (2) item difficulty (p- values) indicates the proportion of students who were able to answer an item correctly – this statistic is also important to discriminate between the performance of test-takers; and (3) item-total correlations (or item discrimination), indicates the extent to which an item correlates between the question score and the overall assessment and discriminates between stronger and weaker students (students who score high on the test overall should also have answered individual items correctly). 6 As with the baseline instruments, the midline and endline assessments will also need to be piloted to ensure equivalent levels of difficulty to baseline, thus ensuring comparability of measures over time.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 2 Results This section describes the students’ baseline results for each EGRA and EGMA task in Kashmore district. Three kinds of results are presented: (1) average percent correct; (2) proportion of zero-score students (students who did not get any of the items on the task correct); and (3) comparisons by gender.7 EGRA Results The EGRA assessment consisted of six tasks for Grade 3 students and eight tasks for Grade 5 students. The tasks for both the Grade 3 and Grade 5 EGRAs were the same, assessed at the Grade 2 level, with the exception of two additional tasks on the Grade 5 EGRA: fluency and reading comprehension using a story at a Grade 4 reading level. 8 Note that results are reported as Grade 3 and 5, though as noted above, these measures represent student reading skills at the end of Grades 2 and 4.

Figure 1 shows that at baseline, students in Grade 3 had the highest scores in Vocabulary, followed by Listening Comprehension; the lowest scores were observed in Phonemic Awareness. Furthermore, as shown in Figure 2, 87 percent of Grade 3 students had zero-scores in Phonemic Awareness – that is, these students could not correctly identify the phoneme (smallest unit of sound) at the end of the word. (Zero-scores provide critical warning signs to a system, highlighting areas needing significant, targeted instruction.) Across all tasks, the average percent correct was 37.7 percent. Promisingly, the highest scores – in the areas of Vocabulary and Listening Comprehension – are indicators of prior knowledge and precursors to a student’s future success as a reader.

FIGURE 1: GRADE 3 EGRA: PERCENT CORRECT SCORES BY TASK

58.6% 58.1% 47.2% Average: 37.7%

30.2% 25.6%

6.7%

Phonemic Phonics Vocabulary Fluency (Grade 2- Reading Listening Awareness level text) Comprehension Comprehension (Grade 2-level text)

7 Rural versus urban location was analyzed in districts with sufficient numbers of schools in each group. In most districts, including Kashmore, the percentage of urban schools was too low (3%) to allow for meaningful comparisons. 8 Phonemic Awareness (Phoneme Isolation task): On this untimed task, students were asked to identify a phoneme – the smallest unit of sound in a word – at the end of 10 words; Phonics (Non-Word Reading or Decoding task): On this timed task, students were presented with 50 non-words (words created for this exercise) and asked to read the word using decoding skills (if readers know how to decode, they can read a word even when it’s not a real word) in one minute; Vocabulary (Expressive Vocabulary task): On this untimed task, students were presented with 10 pictures and asked to identify what they saw; Fluency (Passage Reading task): On this timed task, students were given one minute to read a passage consisting of 60 words- Grade 3 students were tested using a passage at a Grade 2 reading level and Grade 5 students were tested using both Grade 2-level text and Grade 5-level texts; Reading Comprehension (Reading Comprehension task): On this untimed task, students were asked comprehension questions based on the same 60-word passage they read for the fluency task; Listening Comprehension (Listening Comprehension task): On this untimed task, students were first read a story, then asked three questions to assess their understanding of the story’s meaning.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 3 FIGURE 2: GRADE 3 EGRA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

87.0%

54.9% 43.3% 38.6%

12.7% 7.9%

Phonemic Phonics Vocabulary Fluency (Grade 2- Reading Listening Awareness level text) Comprehension Comprehension (Grade 2-level text)

As Figure 3 shows, students in Grade 5 scored highest on Fluency (Grades 2 and 4), followed by Listening Comprehension. The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores (Figure 4).

FIGURE 3: GRADE 5 EGRA: PERCENT CORRECT SCORES BY TASK

75.4% 76.1% 67.0% 60.6% 49.5% 51.8% 55.4% Average: 53.2%

9.5%

Phonemic Phonics Vocabulary Fluency (Grade Reading Listening Fluency (Grade Reading Awareness 2-level text) Comprehension Comprehension 4-level text) Comprehension (Grade 2-level (Grade 4-level text) text)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 4 FIGURE 4: GRADE 5 EGRA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

82.0%

21.7% 24.0% 22.7% 16.2% 15.8% 8.6% 9.1%

Phonemic Phonics Vocabulary Fluency (Grade Reading Listening Fluency (Grade Reading Awareness 2-level text) Comprehension Comprehension 4-level text) Comprehension (Grade 2-level (Grade 4-level text) text)

Fluency Fluency is a reader’s ability to read accurately with speed and proper pacing; it is a common measure for determining reading skill. EGRA measures accuracy and speed, called correct words per minute (CWPM), or the number of words a reader reads correctly in one minute on a timed reading task. At baseline in Kashmore, Grade 3 students’ averaged 38.9 CWPM and Grade 5 students averaged 72.3 CWPM on the Grade 2-level text; Grade 5 students averaged 76.5 CWPM on the Grade 4-level text. As a reference, the Pakistan Reading Project (PRP) baseline proposed fluency levels (in Urdu) to be considered as provisional standards until national reading standards are officially established. According to these standards, Grade 3 students in Kashmore would be classified in the top of the “early readers” range and Grade 5 students would be in the top of the “intermediate” range. Importantly, 38.6 percent of Grade 3 students and 24.0 percent of Grade 5 students were unable to read a single word of Grade 2-level text. With Grade 4-level text, 22.7 percent of Grade 5 students could not read a single word correctly. Comparison of Scores by Gender: In Kashmore, Grade 3 girls significantly outperformed boys in Vocabulary while Grade 3 boys significantly outperformed girls in Fluency. In Grade 5, boys significantly outperformed girls in Phonics and Listening Comprehension.

EGMA Results The EGMA assessment consisted of six tasks for Grade 3 students and seven tasks for Grade 5 students. The tasks for both the Grade 3 and Grade 5 EGMAs were the same, assessed at the Grade 4 level; the Grade 5 EGMA included an additional task in multiplication, assessed at the Grade 4 level. 9

As Figure 5 and Figure 6 show, Grade 3 students scored highest on Number Identification, Number Discrimination and Word Problems tasks. Students scored higher on the Addition task compared to the Subtraction; the proportion of zero-scores was highest on the Subtraction task. Grade 3 students scored lowest on the Missing Number task.

9 Number Identification: On this timed task, students were asked to identify 20 numbers ranging up to 999; Number Discrimination: On this untimed task, students were asked to discriminate between a pair of numbers by identifying the larger number of the two; Missing Number: On this untimed task, students were asked to identify the missing number in a pattern (ranging up to 999); Computation (Addition and Subtraction): On these two timed tasks, students were presented with 20 addition and 20 subtraction problems in one minute for each task; Word Problems: On this untimed task, students were presented with a word problem and were expected to respond with the correct answer based on one or several computations needed to answer the question; Multiplication (Grade 5 only): On this timed task, students in Grade 5 were given a printed list of 20 multiplication problems and were asked to multiply two single-digit numbers in one minute.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 5 FIGURE 5: GRADE 3 EGMA: PERCENT CORRECT SCORES BY TASK

66.3% 58.6% 54.4% Average: 46.8% 39.6% 33.9% 28.2%

Number Number Missing Number Addition Subtraction Word Problems Identification Discrimination

FIGURE 6: GRADE 3 EGMA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

40.8% 37.5%

25.6%

11.9% 9.0% 4.9%

Number Number Missing Number Addition Subtraction Word Problems Identification Discrimination

As Figure 7 and Figure 8 show, Grade 5 students scored highest in Number Identification, Number Discrimination and Word Problem tasks, while they scored lowest on the Missing Number and Multiplication tasks. Approximately half of Grade 5 students were unable to answer a single question correctly on the Multiplication task, and one-quarter were unable to answer a single subtraction question.

FIGURE 7: GRADE 5 EGMA: PERCENT CORRECT SCORES BY TASK

84.4% 81.0%

68.4% Average: 65.4% 59.4% 53.9% 45.3%

27.7%

Number Number Missing Addition Subtraction Word Problems Multiplication Identification Discrimination Number (Grade 5 only)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 6 FIGURE 8: GRADE 5 EGMA: PROPORTION OF ZERO-SCORE STUDENTS BY TASK

50.3%

25.5% 22.0% 13.2%

2.0% 3.5% 3.1%

Number Number Missing Number Addition Subtraction Word Problems Multiplication Identification Discrimination (Grade 5 only)

Comparison of Scores by Gender: In Grade 3, boys had a higher overall score on the EGMA than girls by 12 percentage points. In Grade 5, boys outperformed girls, overall, by 8.5 percentage points. At the task level, boys in both grades significantly outperformed girls in all tasks except the Missing Numbers task.

Questionnaire Results In addition to the EGRA and EGMA assessments, questionnaires were administered with students, teachers, and head teachers in order to identify factors most frequently associated with strong reading and mathematics outcomes. The following are key findings from the questionnaires and their relationship (if significant) with EGRA and EGMA scores: Student questionnaire: • Reading materials at home: One-quarter of Grade 3 and one-fifth of Grade 5 students said they had no reading materials at home. Students who had reading materials at home had significantly higher EGRA and EGMA scores. • Practice reading at home: About half of students in both Grades reported that someone at home read stories aloud to them; this was associated with higher EGRA scores among Grade 5 students. Slightly more than half of students reported that they practiced reading stories aloud to someone in their home; this was associated with significantly higher EGRA scores among Grade 3 students. Grade 5 students who practiced reading silently at home had significantly higher EGRA scores as did Grade 3 students who reported reading the Quran at home. • Work with peers: About 60% of students reported completing classwork with peers and this was associated with higher assessment scores. • Absenteeism: Just under one-third of students missed days from school the previous week; in Grade 5, these students’ EGRA and EGMA scores were lower than those who were not absent. • Work outside of school: Almost all students reported working before or after school, but this was associated with higher EGRA scores among Grade 3 students only. • Access to Communication Technology: About a third of students reported listening to the radio at home; Grade 5 students who listened had slightly higher EGMA scores. Teacher and head teacher questionnaires: • Teacher qualifications: No significant differences were associated with teacher qualifications in EGRA or EGMA scores. • Multi-grade classroom: More than three quarters of teachers taught multi-grade classrooms. Teaching multi-grade classrooms was not associated with student assessment scores. • Timetable implementation: Seventy-one percent of head teachers indicated that their schools followed a timetable; of those who did follow a timetable, about half set aside time for teaching

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 7 reading and math. Including reading in the school timetable was not associated with student scores. • Head Teacher Support: Almost all head teachers reported supporting teachers in reading and math instruction, those who supported teachers on how to teach reading skills had higher Grade 3 EGRA scores. Interestingly, head teachers monitoring student progress was associated with lower EGMA scores in Grade 5. • School Infrastructure: Over half of schools had facility improvements such as clean water, electricity, and toilet facilities; 42% of schools had gender separate toilet facilities at mixed schools; only one school reported having a library. Facility improvements were not associated with higher assessment scores. Recommendations Reading Grade 3 students had their highest scores in vocabulary and listening comprehension, whereas Grade 5 students had their highest scores in fluency and listening comprehension. These results suggest that students’ oral foundation and background knowledge are areas of relative strength. Students’ fluency rates were in top of the “early reader” range in Grade 3 and at the top of the “intermediate range” in Grade 5. These fluency rates, and the low performance of students in Phonemic awareness, suggest that generally, students in Kashmore need continued support to build both fluency and comprehension but struggle most with sound identification of words in isolation. To this end, following recommendations are proposed in order to focus SRP interventions on areas of need for students, teachers, and head teachers: 1. Recommendations to strengthen reading instruction: • Capitalize on students’ relative strengths in the areas of Vocabulary and Listening Comprehension throughout reading instruction. For example, book selection should incorporate topics where students have prior background, where possible. • Ongoing formative assessment should guide instructional decision-making for the teacher to ensure that students’ instructional skills are identified and targeted for instruction. • Small group instruction is provided to students via reading groups that have students together of a similar reading level. Leveled books at the student’s “instructional level” are used in small group reading instruction. “Instruction level” is with text where the student can read between 90-95% of the words with automaticity. • Students should be provided with adequate time each day to practice independent reading skills, regardless of their reading level. Independent reading practice should be with text where the student can read at least 95% of the words with automaticity, have prior knowledge of the topic, and is of high interest to the student. Students of similar reading levels can pair up and read to each other aloud, to practice reading text at their “independent level.” Students can also read independently to themselves to practice reading. Students’ reading endurance is to be increased as they development, starting with 5-10 minutes for younger grades and 20 or more minutes for older grades. • To strengthen reading Fluency, Reading Comprehension, and Vocabulary development, daily read alouds should be incorporated by the classroom teacher. Book selection for read alouds should be at a reading level higher than what the students can read independently. This allows for exposure to different genres, content vocabulary, and builds prior knowledge in students. • Ensure phonemic awareness is taught regularly and explicitly, particularly in Grade 1, as it will provide a foundation for later phonics instruction. • Daily, explicit, structured phonics instruction should be taught in an engaging, multisensory, authentic manner. Adequate practice is incorporated, such that mastery of each decoding skill is attained prior to moving on to the next skill. Phonics skills include asking students to identify the relationship between letters and sounds. Activities may include writing new words and word parts on the blackboard and asking students to decode them.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 8 • Incorporate reading practice to enhance reading fluency with a “repeated reading” strategy, whereas a student re-reads a passage more than one time. This enhances comprehension and allows the student to experience reading with enhanced fluency. This strategy can also be used as “paired reading,” where two students of similar reading levels are paired and engage in “repeated reading” with each other. • Incorporate regular and explicit practice with “sight words” to enhance reading automaticity. Sight words are high frequency words that a student can read by sight” (i.e., not by sounding out). Using a “Drill Sandwich” strategy to learn sight words is an effective method for mastery of sight words, whereas adequate challenge is incorporated into the learning set. Typically, a learning set where 70% of the words are mastered and 30% of the words are new is considered to be adequate challenge. If this proves too difficult, the learning set can be made easier (e.g., 90% mastered and 10% unknown). • Provide teachers with explicit strategies for strengthening listening and reading comprehension skills such as pre-reading (asking questions before reading a story, teaching vocabulary, discussing the subject of the story before reading), read-alouds (the teacher reads a story and asks questions along the way to draw students’ attention to important details, query their comprehension, or ask them to predict) and post-reading (such as group, pair, and independent reading, drawing pictures, continuing or dramatizing the story) to bolster comprehension and reading skills, simultaneously.

2. Recommendations to strengthen school support for reading: • Develop a support system for reading teachers so they can share best practices and promising practices across and between colleagues for reading instruction. • Provide high quality, targeted professional learning for teachers regarding the implementation of best practices in reading instruction. Included in this professional learning is a framework to use formative assessment to guide instruction. • Expand time spent reading in and out of class. For example, designate class time for reading each day, ensure that students have access to reading materials at school and at home, develop highly engaging reading events such as a community “reading marathon,” and use volunteers or “cross-age buddies” to build language skills. • Create a library within each classroom so that students have ready access to high interest reading materials. • Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult). Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests. 3. Recommendations for further research or actions by SRP staff:

• Establish standards for reading fluency in the Sindhi language. • Identify critical areas of intervention to focus on schools and/or sub-groups where large proportions of children are being left behind (i.e., those with zero-scores). Targeted interventions may include pullout of non- and struggling readers for intensive teaching of core competencies/foundation skills, early enrichment activities in formal educational care or via informal mother training to help develop pre-reading foundational skills, implementation of procedures for diagnosing reading difficulties, remediation, and the use of formative assessment to ensure students who lack foundational reading skills are identified early. • Examine factors that may contribute to girls lagging behind boys in the development of reading skills in Grades 3 and 5. Provide targeted interventions for girls where needed.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 9 Mathematics 1. Recommendations to strengthen mathematics instruction:

• Provide teachers with strategies for building number sense and computational approaches, which would enable students to identify numbers and patterns and build computational skills (addition, subtraction, and multiplication). • Provide teachers with strategies for diagnosing students experiencing difficulties in number sense and computation, and formative assessment to ensure students who lack foundational mathematics skills are identified early. • Examine trends by sub-groups and geography to determine root causes for the poor performance among girls in both Grades 3 and 5 relative to that of boys. Use these findings to develop targeted intervention for girls. • Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult). Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 10 CHAPTER 1: INTRODUCTION

1.1 The Sindh Reading Program (SRP)

Under the umbrella of the Sindh Basic Education Project (SBEP), the Sindh Reading Program (SRP) is a five-year collaborative initiative of the Government of Sindh (GoS) and USAID/Pakistan (funded by USAID). SRP is designed to improve student performance in early grade reading and mathematics for 410,400 children across eight districts in the province of Sindh in Pakistan.10 SRP’s main objectives, when the project was initiated, were to:

1. Improve teacher competencies for effective early grade reading and mathematics instruction; 2. Improve early grade reading and mathematics assessment practices; 3. Improve student access to supplementary reading and mathematics materials; and 4. Enhance participation of parents and caregivers in support of early grade literacy and mathematics for out of school children. SRP is being implemented across eight districts in the province of Sindh in Pakistan: Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot, Sukkur, and five towns in Karachi (which are treated as a single district for the purposes of this report). By 2018 (year 5 of the program), one of SRP’s key indicators is to improve reading for at least 200,887 Grade 2 students.11

Through the end of the project, the Education and Literacy Department of the Government of Sindh (GoS), USAID, and SRP have and will collaborate to implement multiple strategies to improve teaching practices and learning performance in early grade literacy with the ultimate goal of improving reading skills among students in participating schools. SRP’s strategies include:

• Design and implement a baseline study of student performance in early grade reading and mathematics; • Analyze current performance levels to target project interventions; • Define “grade level” competencies through the development of performance standards; • Implement follow-up studies to monitor project progress towards established goals; and • Institutionalize early grade reading and mathematics assessment practices at the government and school levels.

This report focuses on the first strategy, the design and implementation of the SRP baseline study for one of the eight target districts: Kashmore. In its original design, along with a focus on improving reading skills among students in the early grades, SRP also included interventions to improve students’ mathematics skills. This design was subsequently revised to retain only the core indicator of improving reading skills in the early grades; however, this revision came after baseline data had already been

10 The focus on mathematics has since been removed from the evaluation and program design; this shift, however, occurred after the baseline data were collected and is discussed below. 11 The target reflects revisions as of April 2015.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 11 collected. As such, baseline data on both students’ literacy and mathematics were collected and analyzed; at midline and endline, data on literacy only will be collected. Baseline data on students’ reading and mathematics skills were collected, and the results analyzed and reported to 1) provide a starting point for measuring progress on improved reading to determine changes in the reading skills of students participating in the SRP interventions; and 2) to inform the implementation of SRP interventions for the remaining years of the project. Furthermore, the baseline study also served as a reference point for establishing the student assessment tools for midline and endline data collections. The tool used to assess students’ reading skills is called the Early Grade Reading Assessment (EGRA) and the tool used to assess students’ mathematics skills is called the Early Grade Mathematics Assessment (EGMA). In addition to student assessment data, the evaluation also sought to capture contextual variables that may be associated with strong reading and mathematics outcomes, and thus included a Questionnaire for students taking the EGRA and EGMA tests, and one for their teachers and head teachers. Student questions related to language spoken at home, availability of reading materials at home, study habits, access to communication technology and vehicles at home. Teacher and head teacher questions related to qualifications, attendance at trainings, scheduling, teacher support, head teacher knowledge of student performance, teacher observation procedures, School Management Council (SMC) authority and facility improvements. The evaluation baseline was conducted in September 2014, prior to the launch of SRP interventions. In accordance with USAID program evaluation guidelines, samples of students at two grade levels – Grade 3 and Grade 5 – were assessed throughout Sindh province so that independent baselines can be established in each district. It should be noted that the results from this baseline are considered a measure of learning at the end of Grades 2 and 4, even though the baseline was conducted for Grades 3 and 5 students. The decision to conduct the baseline with this population was made because of requirement to conduct the baseline before the startup of project activities in Year 1. This timing presents a complication – the effects of “summer learning loss” – i.e., that students typically forget some of what they have learned in a given school year over the summer. The extent of this loss cannot be estimated and the results presented here may in fact be underestimating the actual performance of students prior to the break. In other words, assessments conducted after the summer break will reflect a short-term decline in student achievement. Note that SRP midline and endline assessments will be conducted at the same point in the school year in order to ensure comparability of results across time points.

These three time points (baseline, midline and endline), when compared, provide a measure of the growth made by students in reading in schools participating in SRP interventions. 12

As part of the SRP baseline study, the following activities were carried out for each of the districts. Below and in Chapter 2, each activity is described in greater detail. 1. Evaluation Design. A cross-sectional design was followed in accordance with USAID guidelines. 2. Sampling. A stratified cluster random sampling method was used in order to be able to generalize the results to the district level. 3. Instrument development. EGRA/EGMA tools were developed for the Sindh province and piloted.

12 The evaluation utilizes a non-experimental pre-test/post-test design, in which students participating in the intervention are assessed prior to the start of the interventions (pre-test) and following the conclusion of the interventions (post-test); additionally, a mid-point measure is also collected. This design was appropriate for this study since an adequate comparison group (i.e., students in schools not participating in SRP interventions, but similar otherwise) was not feasible. A comparison group is necessary in order to establish impact of the interventions (in other words, to be able to attribute all the changes from pre-test to post-test among participating students to the SRP interventions). Furthermore, adjustments made to SRP’s focus, overall, following the collection of baseline data, dictates that only reading will be assessed at midline and endline.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 12 4. Training. Workshops were conducted to train Master Trainers (MTs), supervisors, enumerators, and Quality Control Officers (QCOs). Enumerators and supervisors were observed to ensure clear understanding and skills adequate to implement the EGRA and EGMA tools. 5. Implementation. The survey was implemented in all targeted schools, followed by data entry and preparation of a clean data file. 6. Analysis. Two types of analyses were conducted: 1) test quality and 2) analysis of baseline data to report findings by subgroup, proportion of students with zero-scores and average percent correct by task. 7. Reporting. One summary report was produced for each of the two language groups – Sindhi and Urdu; one district-level report was produced for each of the seven districts, including this report for Kashmore; and one report was produced for the five towns of Karachi (treated as a district for the purposes of the evaluation).

1.2 Evaluation Design

USAID required a cross-sectional design, assessing students at the same grade levels (including equal representation by gender) over the course of SRP in both reading and mathematics skills. As shown in Figure 9, the design features two grade levels (3 and 5) and three time points (baseline, midline, and endline). The intent of the evaluation is to determine gains in reading across three time-points (2014, 2016 and 2018) among students in SRP schools using results from students in Grades 3 and 5 as indicative of student performance at the ends of Grade 2 and 4. All students were assessed using the same instruments that measured student’s ability at the Grade 2 level. Grade 5 students were assessed on two additional tasks measured at the Grade 4-level (therefore, Grade 5 students had two reading tasks that were each assessed at both the Grade 2 and Grade 4 level). Because students were selected at random, results of this baseline can be generalized to all Grade 2 and Grade 4 students in schools in which students speak that language (Urdu in Karachi and Sindhi outside of Karachi) in each sampled district.

FIGURE 9: EVALUATION DESIGN

BASELINE MIDLINE ENDLINE

2014 (Y1) 2016 (Y3) 2018 (Y5)

Grade 3 Grade 3 Grade 3 Grade 5 Grade 5 Grade 5

Note: Resu lts from Grade 3 students are indicative of results from Grade 2 students at the end of the year; similarly, results from Grade 5 students are indicative of results from Grade 4 students at the end of the year.

1.3 What are EGRA and EGMA?

The Early Grade Reading and Mathematics Assessments (EGRA and EGMA) were developed with funding from USAID and the World Bank. These assessments measure student performance on the most basic foundational skills required for fluency in reading and mathematics. Policymakers have used the results of EGRA and EGMA to develop reforms to improve reading and mathematics performance of students in the early grades. Moreover, these data have been used by educationists to establish student performance standards so that teachers understand expectations for students at each stage of reading and mathematics development and can tailor instruction appropriately. EGRA and EGMA are individually administered, oral assessments that require approximately 35 minutes per student.

SRP developed the EGRA and EGMA tools used for the baseline by adapting models from other countries to the Sindh context. Adaptations are necessary in order to ensure cultural appropriateness and alignment with SRP interventions. In Karachi, the assessments were conducted in Urdu and to students

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 13 in Urdu-language schools, while students in Sindhi-language schools in the remaining seven districts were assessed in Sindhi. Detailed descriptions of each task are provided in Section 2.3.

Chapter 2 details the methodology for the SRP baseline study. EGRA results are presented in Chapter 3. And EGMA results are presented in Chapter 4. Finally, relationships between contextual variables (student and teacher questionnaire results) and EGRA and EGMA performance are presented in Chapter 5. Additional supporting materials are provided in the annexes at the end of this report.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 14 CHAPTER 2: METHODOLOGY

This chapter describes the SRP sampling process, development of the EGRA and EGMA assessment tools and piloting, data collection, data entry, data analysis and results supporting the quality of the assessment tools.13 The same procedures were used in all districts.

2.1 SRP Sampling Design

A stratified cluster random sampling method was used in order to meet two SRP requirements: (1) that equal numbers of girls and boys be assessed, and (2) that results be generalizable to the population of students in the eight districts. In seven of the SRP intervention districts (Dadu, Jacobabad, Kashmore, Khairpur, Larkana, Kambar-Shahdadkot and Sukkur), 70 schools were randomly selected from the total pool of Sindhi-speaking schools in those districts. Similarly, 70 schools were selected from the total pool of Urdu-speaking schools in the five towns of Karachi City being served by SRP (for the purposes of this report, the five towns of Karachi City were treated as a district). The following is a description of the procedure used for the selection of schools and students in each of the seven districts, including Kashmore. Data from the 2013 Sindh Education Management Information System (SEMIS) were used as a basis for sampling.

1. All schools within the district were sorted by their location: urban or rural. 2. Schools were deselected if they did not meet the minimum requirement of at least 12 male students or 12 female students for Grades 3 and 5. 3. Thirty-five boys’ schools and 35 girls’ schools were randomly sampled, proportionately according to urban/rural ratios. 4. Ten additional schools were selected using the same procedure (stratified by location and gender) in case of inaccessibility or inaccurate SEMIS data. Note that mixed-gender schools were sometimes selected as replacement schools because an insufficient number of single-gender replacement schools were available. When a mixed-gender school was selected, only students of the respective gender were included in those samples. 5. Within each school, 12 students were randomly selected in Grade 3 and 12 students in Grade 5. 6. Teams of QCOs (Quality Control Officers) and SRP project staff visited the districts to verify the preliminary sample of schools, and replacements were made wherever the actual number of students was less than the target number of students. In some instances, schools were retained in the final sample if the number of students at each grade level was close to 12 and if the replacement schools did not have a larger number of students.

2.2 Kashmore School and Student Sample

The EGRA and EGMA assessments were administered to 12,793 students in 560 schools across eight districts of Sindh Province that were participating in SRP. The total number of schools that were sampled in Kashmore is shown in Table 1. Of the 1,209 schools where Sindhi was the primary language of instruction, 1,170 schools were in rural areas and 39 in urban areas. An equal number of boys and girls schools was sampled. Since only 3 percent of schools were located in urban areas, comparing EGRA and EGMA scores by location (urban versus rural) schools was not advised because of the unequal sampling.

13 For more details on the tool development process and implementation of testing, see Chemonics (2014) “Process Report for EGRA/EGMA Baseline Study 2014.”

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 15 TABLE 1: BASELINE SCHOOLS IN KASHMORE

Baseline Number of Proportion of Schools Location Schools Sample (n) (n) (%) Boys Girls Rural 1,170 97.0 34 34 Urban 39 3.0 1 1 Total 1,209 100 35 35

The baseline sample consisted of 70 schools from each district and within each school, 12 students from Grade 3 and 12 students from Grade 5 were assessed, for a total of 13,440 students across all eight districts. In Kashmore, the assessment tools (EGRA, EGMA and student questionnaire) were administered to students in 70 schools for a total sample of 1,519 students (see Table 2). This final dataset for Kashmore represented 90 percent of the total target sample. Additionally, teacher questionnaires were administered to 62 Grade 3 and 63 Grade 5 teachers (representing 89 percent of the target sample), and to 69 head teachers (99 percent of the target sample). The number of teachers on whom data were collected was slightly lower than the target sample because some teachers taught both third and fifth grade.

TABLE 2: BASELINE PARTICIPANTS IN KASHMORE

Grade 3 Grade 5 Total Boys 512 501 1,013 Girls 257 242 499 Missing Gender Code 5 2 7 Total Students 774 745 1,519 Male Teachers 21 24 45 Female Teachers 40 39 79 Missing Gender Code 1 0 1 Total Teachers 62 63 125 Male Head Teachers -- -- 44 Female Head Teachers -- -- 25 Total Head Teachers -- -- 69

Note: Students with missing gender codes are excluded from gender-based comparisons but are included in all other analyses.

2.3 EGRA/EGMA Tool Development

In Sindhi-speaking districts (including Kashmore), EGRA and EGMA were administered in Sindhi; in Urdu-speaking districts, EGRA and EGMA were administered in Urdu. The development of both these EGRA and EGMA tools was a multi-step process that included:

1. The choice of tasks to be included in the tools (the competencies to be assessed); 2. The development of the content for each of these tasks (the items); 3. The piloting of the tools in local schools; 4. The analysis of data from the pilot and the determination of changes required to improve the tools; and 5. The finalization of the tools.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 16 Selection of EGRA/EGMA Tasks The baseline EGRA and EGMA tools were developed by SRP to measure core reading and mathematics skills targeted by SRP in its teacher training, instructional materials, and parental involvement interventions. Six types of reading skills were tested with the EGRA for students in Grade 3: phonemic awareness, phonics, vocabulary, fluency, reading comprehension and listening comprehension. The same tasks were used for students in Grade 3 and Grade 5 (using text at a Grade 2 reading level); two additional Fluency and Reading Comprehension tasks (6a and 6b) were used with students in Grade 5 using text at a Grade 4 reading level. Some of these tasks were timed in order to determine the speed with which students were able to decode words (phonics) and read connected text (fluency). A summary of these tasks and the skills they were designed to assess are presented in Table 3 followed by a brief description of each task.

TABLE 3: EGRA TEST STRUCTURE

SRP EGRA Tasks Core Reading Skills Task Length (Task Number and Task Name) 1. Phonemic Awareness 1. Phoneme Isolation Untimed 2. Phonics 2. Non-word Reading (Decoding) Timed (1 minute) 3. Vocabulary 3. Expressive Vocabulary Untimed 4. Fluency (Grade 2-level text) 4a. Passage Reading Timed (1 minute) 5. Reading Comprehension 4b. Reading Comprehension Untimed (Grade 2-level text) 6. Listening comprehension 5. Listening Comprehension Untimed 7. Fluency (Grade 4-level text) 6a. Passage Reading (Grade 5 only) Timed (1 minute) 8. Reading Comprehension 6b. Reading Comprehension (Grade 5 only) Untimed (Grade 4-level text)

Phonemic Awareness (Phoneme Isolation task): On this untimed task, students were asked to identify a phoneme – the smallest unit of sound in a word – at the end of 10 words.

Phonics (Non-Word Reading or Decoding task): On this timed task, students were presented with 50 non-words (words created for this exercise) and asked to read the word using decoding skills (if readers know how to decode, they can read a word even when it is not a real word) in one minute.

Vocabulary (Expressive Vocabulary task): On this untimed task, students were presented with 10 pictures and asked to identify what they saw.

Fluency (Passage Reading task): On this timed task, students were given one minute to read a passage consisting of 60 words- Grade 3 students were tested using a passage at a Grade 2 reading level and Grade 5 students were tested using both Grade 2-level text and Grade 5-level texts.

Reading Comprehension (Reading Comprehension task): On this untimed task, students were asked comprehension questions based on the same 60-word passage they read for the fluency task.

Listening Comprehension (Listening Comprehension task): On this untimed task, students were first read a story, then asked three questions to assess their understanding of the story’s meaning.

Six types of mathematics skills were tested with the EGMA for students in Grade 3: number identification, number discrimination, word problems, missing number, addition and subtraction. The same tasks were used for students in Grade 3 and 5; one additional multiplication task (Task 7) was used for students in Grade 5. Again, some of these tasks were timed in order to determine the speed with

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 17 which students were able to identify numbers and perform basic mathematical procedures. A summary of these tasks is presented in Table 4 followed by a brief description of each task.

TABLE 4: EGMA TEST STRUCTURE

Core Mathematics Skills SRP EGMA Tasks Task Length 1. Number Identification 1. Number identification up to 999 Timed (1 minute) 2. Number discrimination up to 2. Number Discrimination Untimed 999 3. Missing Number 3. Missing Numbers up to 999 Untimed 4. Addition 4. Addition within 10 Timed (1 minute) 5. Subtraction 5. Subtraction within 10 Timed 6. Word Problems 6. Word Problems Untimed 7. Simple Multiplication 7. Multiplication up to 2 digits Timed (1 minute) (Grade 5 only)

Number Identification: On this timed task, students were asked to identify 20 numbers ranging up to 999.

Number Discrimination: On this untimed task, students were asked to discriminate between a pair of numbers by identifying the larger number of the two.

Missing Number: On this untimed task, students were asked to identify the missing number in a pattern (ranging up to 999).

Computation (Addition and Subtraction): On these two timed tasks, students were presented with 20 addition and 20 subtraction problems using numbers up to 10; students were given one minute for each task.

Word Problems: On this untimed task, students were presented with a word problem and were expected to respond with the correct answer based on one or several computations needed to answer the question.

Multiplication (Grade 5 only): On this timed task, students in Grade 5 were given a printed list of 20 multiplication problems and were asked to multiply two single-digit numbers in one minute.

EGRA/EGMA Tool Revision The SRP team conducted a six-day workshop from April 14-19, 2014 at the Bureau of Curriculum (BoC) in Jamshoro to draft the tasks for the EGRA and EGMA tools in both Sindhi and in Urdu. This workshop served as an introduction to EGRA and EGMA and the first capacity building activity for GoS, BoC, Provincial Institute of Teacher Education (PITE), District Examination Committee and Sindh Textbook Board (STBB) officials.

A total of 19 participants, 14 men and 5 women, including 16 subject experts (four language and reading experts in Urdu and five in Sindhi, and seven mathematics education experts), two typists and one illustrator attended the workshop. Participants reviewed EGRA and EGMA models used in other countries, examined the proposed tasks against the national curriculum,14 determined the appropriate content- and difficulty- levels for end of Grade 2 and end of Grade 4 and drafted the content for all the

14 Though EGRA and EGMA are not curriculum based, participants used the national curriculum in order to ensure that the content and level of difficulty of the EGRA and EGMA tasks were appropriate for these grade levels.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 18 assessment tasks. They also developed questionnaires for students, teachers, and head teachers in Sindhi and Urdu to capture contextual information that may be related to EGRA and EGMA outcomes. Piloting of EGRA/EGMA Tools In Hyderabad, the SRP team piloted the EGRA and EGMA tools to determine their validity and reliability before using them for the operational baseline study.15 The pilot was conducted with students in Grades 3 and 5 in May 2014 in 24 randomly selected schools of which half were Sindhi- language and half were Urdu-language. The EGRA and EGMA were administered to students in a one-to-one format using paper-based tools. A total of 40 students were targeted in each school (20 students in Grade 3 and 20 students in Grade 5). Four teams composed of four enumerators each and one QCO comprised a team that visited each school; in all, the teams assessed 406 students with the Sindhi language tools and 413 with the Urdu language tools.

Subject experts were invited to observe the pilot so that they could gain practical insights about the challenges faced by students with the content of the tasks. Tool Finalization Following the pilot, data were entered by one of SRP’s partners (VTT) and analyzed in order to calculate statistics to establish the quality of the assessment tools (results are presented at the end of this chapter). To review student results and quality of the assessment tools and to make necessary adjustments to the tools, SRP convened a workshop from July 7-12, 2014 in Hyderabad. For the EGRA, one change that was reviewed was the level of difficulty of the phonemic awareness task for students in Grades 3 and 5 in both languages; a revision seemed necessary based on the relatively low scores observed in the pilot.16 As a result of reviewing the pilot results, the team decided to rewrite the task to focus on the sounds at the end of words, rather than the middle sounds. For the EGMA, the assessment, overall was found to be too long (in the pilot, this assessment lasted, on average, between 50 and 60 minutes, per student). As a result, the team decided to eliminate three tasks, namely, Addition and Subtraction at Level Two and Division.

2.4 Data Collection

Preparation for the baseline data collection in September commenced in June 2014. A timeline was developed to ensure the timely procurement of services and materials, the training of assessment personnel, and the execution of the data collection. To adequately prepare for the baseline data collection, SRP had to solicit for and select subcontractors to oversee data collection, entry and cleaning in each of the eight districts; enumerator training had to be conducted (and final enumerators selected) and data entry and cleaning procedures developed. Each of these tasks are described below. Subcontractor Selection In July, SRP issued a Request for Proposals to solicit services from firms to manage data collection, data entry, and cleaning. Nine firms submitted proposals. A committee composed of four senior SRP technical staff evaluated each proposal using predefined criteria. Three firms were selected to manage the recruitment of enumerators, to provide administrative support, and to supervise data collection in each district. The three firms and their areas of management were:

o VTT (Voice Tel Tech): Karachi

15 Hyderabad was selected because of the availability of both Urdu-language and Sindhi-language schools. 16 The piloted phonemic awareness task resulted in 91.4 percent of Grade 3 students and 82.2 percent of Grade 5 students receiving zero-scores (the term “zero-scores” indicates that a student was unable to answer a single question correctly). With such a large proportion of zero-scores, there is insufficient data to adequately measure the quality of the items.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 19 o MDC: Dadu, Larkana, Khairpur, and Kambar-Shahdadkot o I-SAPS: Jacobabad, Kashmore, and Sukkur

Training of Master Trainers, QCOs, and Enumerators Master Trainers and QCOs A 12-day training was organized from August 11-22, 2014 in Karachi for the training of Master Trainers (MTs) and Quality Control Officers (QCOs). The purpose of the MTs and QCOs is to monitor data collection and to ensure that data collection methods are consistent across all enumerators. To ensure that MTs and QCOs were adequately prepared to supervise data collection in the field, SRP made an effort to first recruit candidates with experience in EGRA, or at the very least with experience in educational assessments, evaluation and/or training. From the candidates that attended the training, trainers selected 8 MTs and 8 MTs/QCOs based on the following criteria: 1) they demonstrated adequate understanding of the tools and could administer the assessments with minimal support required; 2) they had high Inter-Rater Reliability scores, and 3) they were available to monitor data collection activities during baseline.

These sixteen individuals served as MTs and/or QCOs in teams of two in each of the districts; they supported field work for the teams of enumerators deputed to each district. When team assignments were made, each team consisted of at least one member MT and QCO who had participated in the pilot in May. SRP staff trained the MTs and QCOs who, in turn, led the training sessions for all enumerators and supervisors.

Enumerators In each district, enumerators were recruited and selected using a similar approach as that described above for MTs and QCOs. Table 5 outlines the details of the enumerators training, by district.

TABLE 5: ENUMERATORS TRAINING BY DISTRICT

District Enumerator Training Criteria for Selection Number of Dates Enumerators Aug 24 to 29 Graduation with 2 years’ Dadu experience conducting surveys in 20 education Larkana Aug 31 to Sep 5 Same as above 20 Khairpur Aug 31 to Sep 5 Same as above 20 Kambar- Aug 24 to 29 Same as above 20 Shahdadkot Jacobabad Aug 24 to 29 Same as above 20 Kashmore Aug 31 to Sep 5 Same as above 20 Sukkur Aug 31 to Sep 5 Same as above 20

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 20 Data Collection Data collection began on September 2, 2014 in Dadu, Jacobabad, Kambar-Shahdadkot and Karachi; and on September 9, 2014 in Kashmore, Khairpur, Larkana and Sukkur. All data collection was completed by September 26, 2014 with approximately 95 percent of the target number of students assessed in Grades 3 and 5.

To ensure quality of data collection, three supervisors as well as two QCOs monitored data collection and provided feedback to the enumerators during data collection in each district. At the end of each day, enumerators, MTs and QCOs discussed progress and problems encountered that day. Assessment forms and questionnaires were checked for possible errors or omissions prior to submission for data entry.

Despite several challenges, including low student attendance, security risks and limited accessibility to sampled schools, each team visited one school per day and were able to reach most schools as planned; where necessary, replacement schools were used. Data Entry and Cleaning In May, the EGRA team developed a customized data entry application so that the data from the booklets and questionnaires could be entered into computers that would then be saved on a networked server. In September, the team trained the Data Coordinator, supervisors, and data entry operators. Following USAID requirements, all student data was entered using the double data entry approach to minimize errors; any discrepancies between the first and second entries were reconciled. A clean data file was then provided to the data analysis team.

2.5 Data Analysis

In January 2015, EGRA statisticians and a psychometrician developed a research plan for the student and teacher data. This plan included the calculation of statistics required to establish the quality of the student assessment tools (reliability as measured by Cronbach’s alpha, item difficulty, item-total correlations), as well as summary scores, by task, for student groups (overall and by gender). Teacher and head teacher questionnaire data were also summarized and their relationship to student assessment results were analyzed.

Descriptive and inferential statistics were also generated by task, grade level and gender to summarize results, measure significance of differences between groups, and determine the strength of the relationship between EGRA and EGMA scores and contextual variables (from student, teacher, and head teacher questionnaires). In some cases, double analyses were conducted to confirm the accuracy of the findings. Raw data were cleaned in Excel and SPSS statistical software. All analyses were conducted in SPSS including the production of means tables and significance tests. Final graphs (those presented in this report) were generated using Word and Excel.

2.6 Quality of Assessment Tools

Overall, the quality of the EGRA and EGMA assessments in measuring reading and mathematics skills among students in Kashmore was found to be high, as suggested by three statistics: (1) internal consistency (Cronbach’s alpha), which indicates whether the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and mathematics, respectively; (2) item difficulty (p-values) which indicates the proportion of students who were able to answer an item correctly – this statistic is also important to discriminate between the performance of test-takers; and (3) item-total correlations (or item discrimination) which indicates the extent to which an item correlates

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 21 between the question score and overall assessment and discriminates between stronger and weaker students (students who score high on the test overall should also have answered individual items correctly). Reliability Cronbach’s alpha is a measure of internal consistency which indicates the extent to which the selected tasks can be generalized to all tasks measuring reading (and not only those selected in this version of the tool). Common guidelines for student assessments (such as the EGRA and EGMA) provide a threshold value of 0.75 to establish the reliability of the assessment. Table 6 provides the reliability estimates for the EGRA and EGMA assessments in Kashmore for Grades 3 and 5. In both grades and content areas, the reliability estimates are greater than the threshold value of 0.75 and therefore render the tools as reliable measures of the student’s reading and mathematics skills. These levels are evidence of the tests’ high levels of internal consistency, and indicate that the results on the selected EGRA and EGMA tasks are a reliable estimate of the expected score for a student on reading and mathematics, respectively.

TABLE 6: RELIABILITY ESTIMATES

Assessment Grade Level Tasks Number Cronbach alpha Grade 3 6 744 0.79 EGRA Grade 5 8 745 0.89 Grade 3 6 744 0.90 EGMA Grade 5 7 745 0.88

Difficulty Another measure of test quality is the use of p-values, which is the proportion (or percentage) of students that got an item correct. Generally, p-values in the range of 0.1 to 0.9 are considered acceptable (and indicate that the item can be retained). For both the EGRA and EGMA assessments in Kashmore, p- values for all tasks in the baseline assessments fell within this range. The proportion of students answering an item correctly in Grade 3 in Kashmore ranged from 0.07 to 0.59 for EGRA and from .28 to .66 for EGMA, suggesting that no items were extremely easy (which would be indicated by a very high proportion of students getting the item correct). The proportion of Grade 5 students answering an item correctly ranged from 0.09 to 0.76 for EGRA and from 0.28 to 0.84 for EGMA. These levels of difficulty for both grade levels were deemed appropriate for these baseline measures since the scales have adequate space to capture growth at the midline and endline assessments.

Discrimination A third measure of test quality is the extent to which each task is able to discriminate between strong and weak students (based on their overall performance on the test). This is determined by measuring the strength of the correlation between each task and the overall test (i.e., this metric is called the item-total correlation, but for the EGRA and EGMA, correlations were calculated between tasks and overall score). All EGRA and EGMA tasks for both Grades 3 and 5 had high item-total correlations. For Grade 3, the lowest item-total correlation was observed for Vocabulary and the highest was observed for Fluency, Reading Comprehension and Phonics. For Grade 5, the lowest item-total correlation was observed for Phoneme Isolation and the highest was observed for Reading Comprehension, Fluency, and Phonics. These results suggest that a student’s score on a task and on their overall score were related, indicating that the tasks were appropriately measuring overall reading performance.

Detailed results of test quality measures are reported in Annex 1.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 22 CHAPTER 3: EGRA R ES ULT S – KASHMORE DISTRICT

This section reports the results for EGRA for students in Kashmore district. The EGRA assessment consisted of six tasks for Grade 3 students and eight tasks for Grade 5 students. First, overall results (percent correct), by grade and across all tasks, are presented as a distribution of students. Second, results are disaggregated by task and gender with results summarized by percent correct and fluency (for timed tasks). Third, the proportion of students who were unable to answer a single question on the task (or zero-scores) are reported. Fourth, results from other Sindhi-language SRP districts are presented for providing context for interpreting results for students in Kashmore. The section concludes with recommendations for supporting students in Kashmore to improve their reading.

Additional details on score calculations, including percent correct, timed task scores (fluency) and task score ranges and calculations are provided in Annex 2. EGRA item-level statistics, by task and grade, are provided in Annex 3.

3.1 Overall EGRA Results

The graphs below show the number of students by the percent of total items correct on the EGRA. For example, the tallest bar in the first graph (Grade 3) shows that approximately 70 students had correctly answered 20 percent of the items correctly. This graph (or histogram) provides a snapshot of the number of students at each level of performance, as well as a profile of the group as a whole. The black trendline summarizes the overall pattern of percent correct for the group based on a calculation of averages over the entire group.

As shown in the first graph in Figure 10: Distribution of EGRA Scores, the trendline for Grade 3 peaks slightly to the left of center (i.e., a deviation from a normal, or bell, curve), indicating a slightly weaker performance, overall, on the test; on average, Grade 3 students correctly answered 37.7 percent of the items. The trendline for Grade 5 peaks at the center (i.e., similar to a normal, or bell, curve), indicating relatively average performance, overall, on the test. On average, Grade 5 students correctly answered 52.3 percent of the items (second graph in Figure 10).

FIGURE 10: DISTRIBUTION OF EGRA SCORES

Grade 3 Grade 5

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 23 3.2 EGRA Results by Task

While the overall results are useful to determine how students in Grades 3 and 5 performed across all EGRA tasks, the remainder of this section focuses on EGRA results by task. Table 7 shows the average percent correct by task (hereafter, percent correct scores are simply referred to as ‘scores’).17 The average score for students in Grade 3 in Kashmore ranged from 6.7 percent correct to 58.6 percent correct. In Grade 5, the average score ranged from 9.5 percent correct to 76.1 percent correct.

TABLE 7: EGRA PERCENT CORRECT SCORES BY GRADE AND TASK

Core Reading Skills SRP EGRA Task Grade 3 Grade 5 (Number and Task Name) (% correct) (% correct) 1. Phonemic Awareness 1. Phoneme Isolation 6.7% 9.5% 2. Non-word Reading 2. Phonics (Decoding) 30.2% 49.5% 3. Vocabulary 3. Expressive Vocabulary 58.6% 60.6% 4. Fluency (Grade 2-level text) 4a. Passage Reading 47.2% 75.4% 5. Reading Comprehension 4b. Reading Comprehension Grade 2-level text) 25.6% 51.8% 6. Listening comprehension 5. Listening Comprehension 58.1% 67.0% 7. Fluency (Grade 4-level text) 6a. Passage reading -- 76.1% 8. Reading Comprehension 6b. Reading comprehension (Grade 4-level text) -- 55.4% Summary Score (all tasks) 37.7% 52.3%

Fluency Fluency is a reader’s ability to read accurately with speed and proper pacing; it is a common measure for determining reading skill. EGRA measures accuracy and speed, called correct words per minute (CWPM), or the number of words a reader reads correctly in one minute on a timed reading task. Fluency rates were calculated for three EGRA tasks: non-word reading, passage reading Grade 2-level text and passage reading Grade 4-level text. The average fluency rates in Correct Words per Minute (CWPM), by grade, are shown in Table 8 with maximum scores shown in parentheses. 18 Fluency rates are also shown in box-plot diagrams in Annex 4.

17 The score for each task was calculated using the total number correct and dividing by the number of items. Student responses were coded either as a 0 (incorrect) or 1 (correct). These item scores were summed for each task and divided by the total number of items to calculate the percent correct. For instance, a student who scored 3 out of 5 on Phonemic Awareness has a percent correct score of 60 percent. 18 The maximum figures are much higher than the percent correct calculations because they are based on the number of words successfully read in a minute. These maximum scores should provide a reference for comparing the mean scores also listed. Please note that maximum scores can contain extremely high scores, called outliers. In addition, adjustments were made to the raw scores for those students who finished the task before the end of one minute. For instance, if a student read 50 words correctly in 30 seconds, their words correct per minute score would be 100 (50 words x 60 seconds/30 seconds). Because these calculations are different from percent correct, the maximum scores are higher (see Annex 4).

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 24 TABLE 8: AVERAGE PHONICS AND READING FLUENCY BY GRADE

Grade 3 Grade 5 Phonics Fluency Subtest Average CWPM (max) Average CWPM (max) 2. Non-word reading 15.8 (91.9) 26.7 (110.8) Reading-Rate Fluency Grade 3 Grade 5 Subtest 4. Passage Reading (Grade 2- 38.9 (216.9) 72.3 (211.8) level) 6a Passage Reading (Grade 4- -- 76.5 (225.0) level)

At baseline in Kashmore, Grade 3 student’s averaged 38.9 CWPM and Grade 5 students averaged 72.3 CWPM on the Grade 2-level text; Grade 5 students averaged 76.5 CWPM on the Grade 4-level text. As a reference, the Pakistan Reading Project (PRP) baseline proposed fluency levels (in Urdu) to be considered as provisional standards until national reading standards are officially established. According to these standards, Grade 3 students in Kashmore would be classified as “early readers” and Grade 5 students would be in the top of the “intermediate” range. Importantly, 38.6 percent of Grade 3 students and 16.2 percent of Grade 5 students were unable to read a single word of Grade 2-level text. With Grade 4-level text, 15.8 percent of Grade 5 students could not read a single word correctly. Proportions of students, by fluency performance level, are discussed in section 3.5.

3.3 Discussion of EGRA Scores by Task Task 1: Phonemic Awareness To assess students’ phonemic awareness, an untimed phoneme isolation task was included in the EGRA (Task 1). Phonemic isolation is the ability of a learner to identify a phoneme - the smallest unit of sound in a word. It is an oral activity and does not require an ability to read, though it is a strong predictor of a child’s ability to learn to read. According to the International Reading Association, phonemic awareness “skills in reading words produces better outcomes on reading comprehension as well as word and pseudo word reading.”19 In the EGRA, the task for students was to identify the last sound in a series of words - for example, the last sound in the word cat is /t/.

Both Grade 3 and Grade 5 students had the lowest average percent correct scores on phonemic awareness compared to all other EGRA tasks. This was also the task with the highest proportion of students with zero-scores (or students who were unable to correctly identify the end sound in a single word; zero-score results are discussed in greater detail later in this chapter). Grade 3 students’ average score was 6.7 percent, which translates to the average Grade 3 student being able to identify up to one sound correctly out of ten. Grade 5 students’ average score was 9.5 percent which translates to the average Grade 5 student being able to identify the last sound in only one out of ten sounds.

Task 2: Phonics To assess students’ core reading skills in phonics, a timed, non-word reading task was included in the EGRA (Task 2). On this task, students were asked to read non-words (words created for this exercise) in order to test their decoding skills. If readers know how to decode, they can read a word, even if it is not a real one. In this timed task, students were asked to read up to 50 non-words in one minute.

On this task, Grade 3 students’ scored an average 30.3 percent correct; in other words, they correctly decoded an average of 15 out of 50 non-words per minute. Grade 5 students scored an average 49.6 percent correct or 25 correct non-words per minute.

19 International Reading Association (2002), “Summary of the (U.S.) National Reading Panel Report Teaching Children to Read”. Page 5.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 25

Task 3: Vocabulary To assess students’ vocabulary skills, an untimed, expressive vocabulary task was included in the EGRA (Task 3). On this task, students were presented with ten pictures and asked to identify what they saw. For example, a picture of a boy drinking would be interpreted correctly as “the boy is drinking”. Like phonemic awareness, expressive vocabulary does not require reading ability but is nevertheless an important foundational skill for learning to read as “vocabulary instruction leads to gains in reading comprehension.”20

At baseline, students in Grade 3 had the highest relative scores on this compared to all other tasks. Grade 5 students’ scores were the fourth highest compared to all other tasks; in both grades, the lowest proportion of students with zero-scores was observed on this task. The results show that Grade 3 and 5 students have a strong oral foundation – an essential component of learning to read. Grade 3 students’ average score was 58.6 on this task and Grade 5 students’ average score was 60.6. For students in both grades, these scores translate to the average student having identified six pictures out of 10 in one minute.

Tasks 4a (Grade 2-level text) and 6a (Grade 4-level text): Fluency To assess students’ fluency, two timed passage reading tasks were included – one using a Grade 2-level text (Task 4a, administered to students in both Grades 3 and 5) and one using a Grade 4-level text (Task 6a, administered to students in Grade 5 only). Students were asked to read, aloud, as much of the 60- word passage as they could in one minute. Like vocabulary, fluency is an important predictor of comprehension. Grade 3 students’ average score was 47.2 when reading Grade 2-level text. Grade 5 students’ average score was 75.4 when reading a Grade 2-level text and 76.1 when reading a Grade 4-level text.

While percent correct gives an indication of how students performed on this task in comparison to other tasks on the EGRA, a useful metric for this task is the fluency rate measured in Correct Words per Minute (CWPM). Fluency rates are discussed in detail in section 3.2 above.

Task 4b (Grade 2-level text) and 6b (Grade 4-level text): Reading Comprehension To assess students’ reading comprehension skills, two timed tasks on reading comprehension were included in the EGRA – these tasks included up to five questions on Grade 2-level text (Task 4b, administered to both Grade 3 and 5 students) and up to five questions on Grade 4-level text (Task 6b, administered to Grade 5 students only). The questions assessed students’ comprehension of passages read in previous fluency tasks (Tasks 4a and 6a). The number of questions a student was asked depended on their reading speed: if a student only read the first sentence correctly, she was asked only one question, relating to the content of that sentence. If however, she finished the entire passage, she was asked all five questions. If she had only reached the middle of the passage, she was only asked two or three questions.

The results show that reading comprehension was the second most difficult task for students in Grade 3 and the third most difficult task for students in Grade 5. In Grade 3, students’ average score was 25.6 and in Grade 5, the average score was 51.8 for Grade 2-level text and 55.4 for Grade-4 level text. 21

While the percent correct scores are useful to compare student performance across tasks, the proportion of reading comprehension questions answered correctly is also a useful metric. On average, Grade 3 students were able to answer one question out of five correctly while Grade 5 students were able to answer at most three out of five correctly. Because the number of questions asked was dependent on

20 Ibid. Page 12. 21 These results are based on the number of questions the student answered correctly out of five to ensure comparability across students. Note, however, that the number of questions a student was asked depended on their reading speed: if a student only read the first sentence correctly, she was asked only one question, relating to the content of that sentence. If however, she finished the entire passage, she was asked all five questions; if she had only reached the middle of the passage, she was only asked two or three questions.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 26 how far the student read in the text, low EGRA comprehension scores are signs of both low levels of reading comprehension and low fluency rates.

Task 5: Listening Comprehension To assess students’ ability to understand the meaning of a story that is read to them, a task on listening comprehension (Task 5) was included in the EGRA. For this task, students did not need to read the story themselves; instead, they were asked to listen while the story was read, out loud, to them. The enumerator then asked students to answer three comprehension questions based on the passage. This test is important because it provides information on learners’ information processing skills (e.g., understanding language and memory) as well as possible disabilities (e.g., auditory, attention). It is also indicative of students’ prior knowledge.

Grade 3 students scored an average of 58.1 percent which translates to the average student having correctly answered at most two of the three questions correctly. Grade 5 students’ average score was 67.0 percent for this task, which translates to the average student having answered two of the three questions correctly.

3.4 EGRA Results by Gender

EGRA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared. Figure 11 shows the average score, by gender, for Grade 3 and Figure 12 shows the average score, by gender, for Grade 5. A comparison of boys’ and girls’ overall EGRA scores in Grade 3 and Grade 5 were not significantly different.

At the task level, Grade 3 girls had significantly higher scores in Vocabulary and Grade 3 boys had significantly higher scores in Fluency. In Grade 5, boys significantly outperformed girls in Phonics and Listening Comprehension.

FIGURE 11: GRADE 3 SCORES BY TASK AND GENDER

64.4% 59.0% 55.5% 56.2% 49.3%

Gr.3 Boys Avg. Score 38.1% 42.1% Gr. 3 Girls Avg. 31.6% 27.0% 26.6% Score 36.7% 23.3%

6.5% 7.2%

Phonemic Phonics Vocabulary * Fluency * Reading Listening Awareness (Grade 2-level text) Comprehension comprehension (Grade 2-level text) G3 Boys G3 Girls G3 Boys Summary Score G3 Girls Summary Score

*Indicates that the performance of the group was significantly higher, p< 0.05.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 27 FIGURE 12: GRADE 5 SCORES BY TASK AND GENDER

77.8% 76.4% 73.8% 69.5% 72.9% Gr.5 Boys Avg. 60.0% 61.7% 62.1% 56.8% Score 53.3% 51.7% 52.5% 50.7% 52.6% Gr.5 Girls Avg. Score 50.4% 45.0% 9.7% 9.2%

Phonemic Phonics * Vocabulary Fluency Reading Listening Fluency Reading Awareness (Grade 2-level Comprehension comprehension * (Grade 4-level Comprehension text) (Grade 2-level text) (Grade 4-level text) text)

G5 Boys G5 Girls G5 Boys Summary Score G5 Girls Summary Score

*Indicates that the performance of the group was significantly higher, p< 0.05.

For timed tasks, fluency rates were also disaggregated by gender (Table 9). In Grade 3, no significant differences were noted by gender. In Grade 5, boys outperformed girls in Phonics (non-word reading).

TABLE 9: AVERAGE PHONICS AND READING FLUENCY BY GRADE AND GENDER

Grade 3 Grade 5 Phonics Fluency Subtest Boys Girls Boys Girls 2. Non-word reading 16.5 14.1 28.1* 24.1 Grade 3 Grade 5 Reading Fluency Subtest Boys Girls Boys Girls 4a. Passage Reading 40.2 35.6 73.7 69.9 (Grade 2-level) 6a. Passage Reading -- -- 78.9 72.1 (Grade 4-level) *Indicates that the performance of the group was significantly higher, p< 0.05

3.5 EGRA Fluency by Performance Levels

Fluency rates can also be used to classify students into categories, or performance level. Using fixed intervals established for districts where Urdu is the medium of instruction (in Punjab under PRP), baseline results for students in Kashmore are summarized. This information is provided as a point of comparison since Sindhi and Urdu are comparable; however, accurate interpretations of students’ performance levels can only be made once standards (proficiency levels) are developed for use in the Sindh Province. The five levels used for Punjab by the PRP baseline are: non-readers (0 WCPM); early readers (1-40 WCPM); intermediate readers (41-80 WCPM); fluent readers (81-120 WCPM); and advanced readers (121 and above WCPM). The proportion of SRP students at each level will be tracked at midline and endline.

Table 10 shows that over one-third of Grade 3 students (38.6 percent) can be classified as non-readers when asked to read Grade 2-level text; 18 percent of Grade 3 students can be classified as fluent or

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 28 advanced readers at baseline. In Grade 5, 16.2 percent and 15.8 percent of students can be classified as non-readers when asked to read Grade 2 and Grade 4-level texts, respectively. Almost a half of Grade 5 students (43.4 percent and 47.3 percent, respectively) can be classified as fluent or advanced readers when asked to read Grade 2 and Grade 4-level texts. Among Grade 5 students, the proportion of students classified in each performance level using Grade 4-level text was relatively comparable to the proportion of students classified using simpler text (Grade 2-level text).

TABLE 10: FLUENCY RATES BY PRP THRESHOLDS AND GRADE

% of Students Performance WCPM Grade 3 Grade 5 Grade 5 Level Grade 2-level Grade 2-level Grade 4-level Non-Reader 0 38.6% 16.2% 15.8% Early Reader 1 to 40 19.4% 8.9% 8.7% Intermediate 41 to 80 24.0% 31.5% 28.2% Fluent Reader 81 to 120 13.6% 27.8% 28.2% Advanced Reader 121 and above 4.4% 15.6% 19.1%

3.6 EGRA Fluency and Reading Comprehension by Performance Levels

More important than how fluently students can read a text is whether or not they understand what they have read. Not surprisingly, these two measures, fluency and reading comprehension, are related. By examining fluency and reading comprehension together, the level of comprehension can be examined as fluency rates increases. In Table 11 (graphically represented in Figure 13), the proportion of students at each comprehension and fluency level (i.e., in each cell) are shown as a percentage of the total students. Reading comprehension levels were calculated in terms of percent correct scores (e.g., 20 percent is the same as correctly answering one question out of five total questions).

For example, in Grade 3, 100 percent of non-readers demonstrated no comprehension since they were not asked the comprehension questions. On the other end hand, 20 percent of fluent readers correctly answered all five comprehension questions, thereby demonstrating high levels of fluency and reading comprehension (note, however, that the number of students represented by this proportion is quite small at 21 students). Using 80 percent and higher as a threshold for high reading comprehension levels, 25 percent (16 percent + 9 percent) of intermediate readers and 51 percent (31 percent + 20 percent) of fluent readers meet or exceed this threshold. The data for advanced readers should be interpreted with caution because there were only 34 Grade 3 students who were able to read at or above 121 words per minute.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 29 TABLE 11: GRADE 3 READING FLUENCY AND READING COMPREHENSION

Category % of Students by Comprehension Level WCPM (Performance Level) 0% 20% 40% 60% 80% 100% Total N

Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 299 Early Reader 1 to 40 62% 20% 11% 4% 3% 0% 100% 150 Intermediate Reader 41 to 80 16% 17% 25% 18% 16% 9% 100% 186 Fluent Reader 81 to 120 4% 12% 12% 20% 31% 20% 100% 105 Advanced Reader 121 and above 0% 6% 9% 15% 29% 41% 100% 34 Total 55% 10% 10% 8% 10% 7% 100% 774

Figure 13 is a graphical representation of the data presented in Table 11. The blue areas represent students at zero to 60% comprehension levels and the orange areas symbolize higher degrees of comprehension. 22 This figure shows that, as one would expect, non-readers and early readers have very low levels of comprehension while more fluent readers have a higher level of comprehension of what they read.

FIGURE 13: GRADE 3 READING PERFORMANCE LEVELS AND READING COMPREHENSION INTERVALS

Grade 3 Reading Fluency and Comprehension at Grade Level 2 Difficulty

0 WCPM 100%

0% Comprehension 1 to 40 WCPM 62% 20% 11% 4%3% 20% Comprehension 40% Comprehension 41 to 80 WCPM 16% 17% 25% 18% 16% 9% 60% Comprehension 80% Comprehension 100% Comprehension 81 to 120 WCPM 4% 12% 12% 20% 31% 20%

121 WCPM and above 6% 9% 15% 29% 41%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Fluency and reading comprehension levels for Grade 5 students reading Grade 2-level text are shown in Table 12 and Figure 14. Using the 80 percent threshold for high comprehension, 54 percent of advanced readers, 47 percent of fluent readers, and 28 percent of intermediate readers in Kashmore met or exceeded this threshold.

22 Eighty percent correct comprehension is a USAID goal for student comprehension.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 30 TABLE 12: GRADE 5 READING FLUENCY AND READING COMPREHENSION (GRADE 2-LEVEL TEXT)

% of Students by Comprehension Level Task 4 WCPM Performance Level 0% 20% 40% 60% 80% 100% Total N

Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 121 Early Reader 1 to 40 49% 24% 17% 9% 2% 0% 100% 66 Intermediate Reader 41 to 80 10% 12% 19% 30% 21% 10% 100% 235 Fluent Reader 81 to 120 2% 5% 9% 27% 33% 24% 100% 207

Advanced Reader 121 and above 0% 1% 2% 15% 38% 45% 100% 116

Total 24% 7% 10% 20% 22% 17% 100% 745

Figure 14 is a graphical representation of the data presented in Table 12. Using the 80 percent threshold for high comprehension, 83 percent of advanced readers, 57 percent of fluent readers, and 31 percent of intermediate readers in Grade 5 in Kashmore met or exceeded this threshold.

FIGURE 14: GRADE 5 READING PERFORMANCE LEVELS AND COMPREHENSION INTERVALS (GRADE 2-LEVEL TEXT)

0 WCPM 100%

1 to 40 WCPM 49% 24% 17% 9% 2%

41 to 80 WCPM 10% 12% 19% 30% 21% 10%

81 to 120 WCPM 5% 9% 27% 33% 24%

121 WCPM and above 2% 15% 38% 45%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% Comprehension 20% Comprehension 40% Comprehension 60% Comprehension 80% Comprehension 100% Comprehension

Fluency and comprehension levels for Grade 5 students reading Grade 4-level text are shown in Table 13 and in Figure 15. Using the 80 percent threshold for high comprehension, 85 percent of advanced readers, 62 percent of fluent readers, and 35 percent of intermediate readers met or exceeded this threshold. These proportions are comparable to those observed when Grade 5 students read Grade 2- level text suggesting that the difficulty of the text did not influence reading fluency and comprehension classifications for Grade 5 students.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 31 TABLE 13: GRADE 5 READING FLUENCY AND READING COMPREHENSION (GRADE 4-LEVEL TEXT)

% of Students by Comprehension Level WCPM Performance Level 0% 20% 40% 60% 80% 100% Total N Non-Reader 0 100% 0% 0% 0% 0% 0% 100% 118 Early Reader 1 to 40 45% 14% 25% 15% 2% 0% 100% 65 Intermediate Reader 41 to 80 9% 13% 15% 29% 18% 17% 100% 207 Fluent Reader 81 to 120 2% 6% 11% 19% 28% 34% 100% 208

Advanced Reader 121 and above 0% 2% 2% 11% 32% 53% 100% 142

Total 23% 7% 10% 17% 19% 25% 100% 740

FIGURE 15: GRADE 5 READING PERFORMANCE LEVELS AND COMPREHENSION INTERVALS (GRADE 4-LEVEL TEXT)

0 CWPM 100%

1 to 40 CWPM 45% 14% 25% 15% 2%

41 to 80 CWPM 9% 13% 15% 29% 18% 17%

81 to 120 CWPM 2% 6% 11% 19% 28% 34%

121 CWPM and above 2% 11% 32% 53%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0% Comprehension 20% Comprehension 40% Comprehension 60% Comprehension 80% Comprehension 100% Comprehension 3.7 EGRA Proportion of Zero-Scores (non-readers)

The results presented thus far examine the percent of items answered correctly or fluency rates – i.e., what students were actually able to do, on average. The proportion of zero-scores, however, provides information on the number of students who were unable to correctly answer even a single item on a task. Table 14 shows the proportion of zero-score students by grade and EGRA task. Zero-scores provide critical warning bells to a system, highlighting students who will continue to be left behind if not provided significant remedial help.

The largest proportions of zero-scores for both Grade 3 and Grade 5 students were observed in the Phoneme Isolation task (which measures a student’s phonemic awareness) with 87.0 percent of Grade 3 and 82.0 percent of Grade 5 students unable to answer a single question correctly. The second highest proportion of zero-scores were observed in the Reading Comprehension task with half of Grade 3 students (54.9 percent) and one-quarter of Grade 5 students (24.0 percent) unable to correctly answer a single comprehension question on Grade 2-level text. The third highest proportion of zero-scores was observed in the Phonics task with almost half of Grade 3 students (43.3 percent) and almost one-quarter of Grade 5 students (21.7%) unable to answer a single item correctly.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 32 TABLE 14: EGRA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE AND TASK

SRP EGRA Tasks Grade 3 Grade 5 Core Reading Skills (Task Number and Task Name) (% students) (% students) 1. Phonemic Awareness 1. Phoneme Isolation 87.0% 82.0% 2. Phonics 2. Non-word Reading (Decoding) 43.3% 21.7% 3. Vocabulary 3. Expressive Vocabulary 7.9% 8.6% 4. Fluency (Grade 2-level text) 4a. Passage Reading 38.6% 16.2% 5. Reading Comprehension 4b. Reading Comprehension (Grade 2-level text) 54.9% 24.0% 6. Listening comprehension 6. Listening Comprehension 12.7% 9.1% 7. Fluency (Grade 4-level text) 6a. Passage reading -- 15.8% 8. Reading Comprehension (Grade 6b. Reading comprehension 4-level text) -- 22.7%

Table 15 disaggregates the proportion of zero-score students by gender. Mirroring the overall EGRA results by gender, this analysis shows that the proportion of zero-scores were higher among Grade 3 girls than boys in Phonics, Fluency and Listening Comprehension tasks. In Grade 5, the proportion of zero- scores were higher among girls than boys in Listening Comprehension and Fluency (grade 4-level text) tasks.

TABLE 15: EGRA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE, GENDER AND TASK

Core Reading Skills SRP EGRA Tasks Grade 3 Grade 5 (Task Number and Task Name) Boys Girls Boys Girls 1. Phonemic Awareness 1. Phoneme Isolation 87.3% 86.0% 82.0% 81.8% 2. Phonics 2. Non-word Reading (Decoding) 40.6% 49.4%* 20.0% 25.2% 3. Vocabulary 3. Expressive Vocabulary 9.2% 5.4% 7.6% 10.7% 4. Fluency (Grade 2-level text) 4a. Passage Reading 34.8% 47.1%* 14.8% 19.0% 5. Reading Comprehension 54.3% 56.8% 23.4% 25.2% 4b. Reading Comprehension (Grade 2-level text) 6. Listening comprehension 6. Listening Comprehension 10.0% 17.9%* 7.4% 12.8%* 7. Fluency (Grade 4-level text) 6a. Passage reading -- -- 13.4% 20.7%* 8. Reading Comprehension -- -- 21.0% 26.0% 6b. Reading comprehension (Grade 4-level text) *Indicates that the performance of the group was significantly higher, p< 0.05

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 33 3.8 EGRA Summary Scores across Districts

Figure 16 below shows the EGRA summary scores (the average of the scores for the seven EGRA tasks23), for each of the seven Sindhi language districts, and the mean score across all districts (note that the EGRA instrument for Karachi was in Urdu language and so the results cannot be directly compared or presented here). This comparison of district-level scores is provided merely for contextual purposes; these aggregate scores do not provide the level of detail required for improving teaching and learning.

FIGURE 15: EGRA SUMMARY SCORES BY DISTRICT

53.2 46.9 43.9 42.0 45.1 45.7 40.2 37.7 33.5 33.7 28.8 29.1 31.1 25.0

Kambar Jacobabad Larkana Sukkur Dadu Khairpur Mirs Kashmore

Grade 3 Grade 5 Grade 3 Mean Grade 5 Mean

In the broader context, other assessments of literacy and mathematics have been conducted in Pakistan in recent years, including a nationwide EGRA for the PRP baseline. Each of these assessments used a different tool since the instruments used must respond to that project’s specific research questions. As a result, the methodology and purpose of the nationwide EGRA and PRP projects differed from SRP, including research questions, the ways in which results can and have been used, the tasks, sampling, and data collection procedures. These differences in approach mean that results are not directly comparable across assessments. However, each of these measures do provide a broader context for understanding reading and mathematics performance among students in Pakistan, and provide different vantage points. The extent of comparability, and the circumstances under which these results are comparable, however, is outside the scope of this report.

3.9 EGRA Summary and Recommendations

The SRP EGRA baseline results provide useful information to help guide interventions to improve reading for Kashmore district students. A summary of findings, by grade, is presented below followed by recommendations. Summary Grade 3: At baseline, students in Grade 3 had the highest scores in Vocabulary, followed by Listening Comprehension. The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores. Across all tasks, the average percent correct was 37.7 percent. Promisingly, the highest scores – in the areas of Vocabulary and Listening Comprehension – are indicators of prior knowledge and precursors to a student’s future success as a reader.

23 Grade 5 students were assessed on Grade 4-level reading and mathematics as well as Grade 2-level. Only the Grade 2-level results are included in the Summary Scores. The Summary Scores thus show the average percent correct for each grade for the same tasks.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 34 Grade 5: Students in Grade 5 scored highest on both Fluency tasks although approximately 15 percent of students could not read a single word (i.e., zero-scores). The lowest scores were observed in Phonemic Awareness, which also had the highest proportion of zero-scores. Recommendations Grade 3 and Grade 5 students had relatively high scores in vocabulary and listening comprehension, suggesting that students’ oral foundation and background knowledge is an area of relative strength. Provisional fluency standards taken from the PRP baseline suggest that Grade 2 students are in the top of the “early readers” range and Grade 4 students are in the top of the “intermediate readers” range. These fluency rates, and the low performance of students in Phonemic awareness, suggest that generally, students in Kashmore need continued support to build both fluency and comprehension but struggle most with sound identification of words in isolation. To this end, following recommendations are proposed in order to focus SRP interventions on areas of need for students, teachers, and head teachers: 1. Recommendations to strengthen reading instruction:

• Capitalize on students’ relative strengths in the areas of Vocabulary and Listening Comprehension throughout reading instruction. For example, book selection should incorporate topics where students have prior background, where possible. • Ongoing formative assessment should guide instructional decision-making for the teacher to ensure that students’ instructional skills are identified and targeted for instruction. • Small group instruction is provided to students via reading groups that have students together of a similar reading level. Leveled books at the student’s “instructional level” are used in small group reading instruction. “Instruction level” is with text where the student can read between 90-95% of the words with automaticity. • Students should be provided with adequate time each day to practice independent reading skills, regardless of their reading level. Independent reading practice should be with text where the student can read at least 95% of the words with automaticity, have prior knowledge of the topic, and is of high interest to the student. Students of similar reading levels can pair up and read to each other aloud, to practice reading text at their “independent level.” Students can also read independently to themselves to practice reading. Students’ reading endurance is to be increased as they development, starting with 5-10 minutes for younger grades and 20 or more minutes for older grades. • To strengthen reading Fluency, Reading Comprehension, and Vocabulary development, daily read alouds should be incorporated by the classroom teacher. Book selection for read alouds should be at a reading level higher than what the students can read independently. This allows for exposure to different genres, content vocabulary, and builds prior knowledge in students. • Ensure phonemic awareness is taught regularly and explicitly, particularly in Grade 1, as it will provide a foundation for later phonics instruction. • Daily, explicit, structured phonics instruction should be taught in an engaging, multisensory, authentic manner. Adequate practice is incorporated, such that mastery of each decoding skill is attained prior to moving on to the next skill. Phonics skills include asking students to identify the relationship between letters and sounds. Activities may include writing new words and word parts on the blackboard and asking students to decode them. • Incorporate reading practice to enhance reading fluency with a “repeated reading” strategy, whereas a student re-reads a passage more than one time. This enhances comprehension and allows the student to experience reading with enhanced fluency. This strategy can also be used as “paired reading,” where two students of similar reading levels are paired and engage in “repeated reading” with each other. • Incorporate regular and explicit practice with “sight words” to enhance reading automaticity. Sight words are high frequency words that a student can read by sight” (i.e., not by sounding out). Using a “Drill Sandwich” strategy to learn sight words is an effective method for

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 35 mastery of sight words, whereas adequate challenge is incorporated into the learning set. Typically, a learning set where 70% of the words are mastered and 30% of the words are new is considered to be adequate challenge. If this proves too difficult, the learning set can be made easier (e.g., 90% mastered and 10% unknown). • Provide teachers with explicit strategies for strengthening listening and reading comprehension skills such as pre-reading (asking questions before reading a story, teaching vocabulary, discussing the subject of the story before reading), read-alouds (the teacher reads a story and asks questions along the way to draw students’ attention to important details, query their comprehension, or ask them to predict) and post-reading (such as group, pair, and independent reading, drawing pictures, continuing or dramatizing the story) to bolster comprehension and reading skills, simultaneously.

2. Recommendations to strengthen school support for reading: • Develop a support system for reading teachers so they can share best practices and promising practices across and between colleagues for reading instruction. • Provide high quality, targeted professional learning for teachers regarding the implementation of best practices in reading instruction. Included in this professional learning is a framework to use formative assessment to guide instruction. • Expand time spent reading in and out of class. For example, designate class time for reading each day, ensure that students have access to reading materials at school and at home, develop highly engaging reading events such as a community “reading marathon,” and use volunteers or “cross-age buddies” to build language skills. • Create a library within each classroom so that students have ready access to high interest reading materials. • Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult. Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests.

3. Recommendations for further research or actions by SRP staff:

• Establish standards for reading fluency in the Sindhi language. • Identify critical areas of intervention to focus on schools and/or sub-groups where large proportions of children are being left behind (i.e., those with zero-scores). Targeted interventions may include pullout of non- and struggling readers for intensive teaching of core competencies/foundation skills, early enrichment activities in formal educational care or via informal mother training to help develop pre-reading foundational skills, implementation of procedures for diagnosing reading difficulties, remediation, and the use of formative assessment to ensure students who lack foundational reading skills are identified early. • Examine factors that may contribute to girls lagging behind boys in the development of reading skills in Grade 3. Provide targeted interventions for girls where needed.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 36 CHAPTER 4: EGMA RE S ULT S – KASHMORE DISTRICT

This section provides an overview of EGMA results for students in Kashmore district. First, overall results across all tasks are presented as a distribution of students by the percent correct score in Grade 3 and Grade 5. Second, results are disaggregated by task and gender with results summarized by percent correct and fluency (for timed tasks). Third, the proportion of students who were unable to answer a single question on the task (or zero-scores) are reported. The section concludes with recommendations for improved mathematics outcomes as suggested by the EGMA results.

Additional details on score calculations, including percent correct, timed task scores (fluency) and task score ranges and calculations are provided in Annex 2. EGMA item-level statistics, by task and grade, are provided in Annex 3.

4.1 Overall EGMA Results

The graphs below show the number of students by the percent of total items correct on the EGMA. For example, the tallest bar in the first graph (Grade 3) shows that approximately 42 students had correctly answered 12 percent of the items correctly. This graph (or histogram) provides a snapshot of the number of students at each level of performance, as well as a profile of the group as a whole. The black trendline summarizes the overall pattern of percent correct for the group based on a calculation of averages over the entire group.

As shown in the first graph in Figure 17, the trendline for Grade 3 peaks at the center (i.e., normal or similar to a bell curve), indicating average performance, overall, on the test. This is also reflected in the mean score for Grade 3 students at 46.8 percent. The trendline for Grade 5 peaks substantially to the right; on average, Grade 5 students correctly answered 65.4 percent of items (second graph in Figure 17).

FIGURE 16: DISTRIBUTION OF EGMA SCORES

Grade 3 Grade 5

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 37 4.2 EGMA Results by Task

While the overall results are useful to determine how students in Grades 3 and 5 performed across all EGMA tasks, the remainder of this section focuses on EGMA results by task. Table 16 shows the average percent correct score, by task.24 The average score for students in Grade 3 in Kashmore ranged from 28.2 percent to 66.3 percent correct; the average score for students in Grade 5 ranged from 27.7 percent to 84.4 percent correct.

TABLE 16: EGMA PERCENT CORRECT SCORES BY GRADE AND TASK

Core Mathematics SRP EGMA Task Grade 3 Grade 5 Skills (Number and Task Name) (% correct) (% correct) 1. Number Identification 1. Number identification up to 999 66.3% 84.4% 2. Number 2. Number discrimination up to 999 Discrimination 58.6% 81.0% 3. Missing Number 3. Missing Numbers up to 999 28.2% 45.3% 4. Addition, level 1 4. Addition level 1 within 10 39.6% 59.4% 5. Subtraction, level 1 5. Subtraction level 1 within 10 33.9% 53.9% 6. Word Problems 6. Word Problems 54.4% 68.4% 7. Simple Multiplication 7. Multiplication up to 2 digits (Grade 5 only) -- 27.7% Summary Score (all tasks) 46.8% 65.4%

Fluency Basic fact fluency in mathematics, also called computation fluency, is defined as “the efficient, appropriate, and flexible application of single-digit calculation skills and…an essential aspect of mathematical proficiency.”25 Moreover, “computation fluency in the elementary grades is a strong predictor of later success in higher-level mathematics coursework.”26 Critical components of fluency measures include the ability: 1) to quickly and accurately add and subtract or multiply and divide; and 2) know from memory all sums and products of problems using two one-digit numbers. Fluency rates were calculated for three EGMA tasks: addition, subtraction and multiplication. 27 The average fluency rates, by grade, are shown in Table 17 with maximum scores shown in parentheses.28 Fluency rates are also shown in box-plot diagrams in Annex 4.

24 The score for each task was calculated using the total number correct and dividing by the number of items. 25 NCTM. Assessing Basic Fact Fluency. Teaching children mathematics. April 2014. Vol. 20, No. 8 26 Gersten, R., Jordan, N. C., & Flojo, J. R. (2005). Early identification and interventions for students with mathematics difficulties. Journal of Learning Disabilities, 38, 293-304. 27 Though Task 1: Number identification was timed, it is not considered a mathematics fluency task because no computation is required. 28 The maximum figures are much higher than the percent correct calculations because they are based on the number of words successfully read in a minute. These maximum scores should provide a reference for comparing the mean scores also listed. Please note that maximum scores can contain extremely high scores, called outliers. In addition, adjustments were made to the raw scores for those students who finished the task before the end of one minute. (See Annex 4 for detailed calculations).

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 38 TABLE 17: MATHEMATICS FLUENCY RATES BY GRADE

Grade 3 Grade 5 Mathematics Fluency Tasks (Task Average Correct Average Correct Number and Name) Responses per Responses per Minute (Max) Minute (Max) 4. Addition 8.7 (80.0) 14.1 (63.2) 5. Subtraction 7.0 (39.3) 11.7 (38.7) 7. Multiplication up to 2 digits (Grade -- 6.1 (40.0) 5 only)

4.3 Discussion of EGMA Results by Task

Task 1: Number Identification To assess students’ ability to identify numbers, a timed number identification task was included in the EGMA (Task 1) that presented students with up to 20 numbers up to 999. Grade 3 students’ average score was 66.3 percent which translates to the average Grade 3 student being able to identify (read) 13 numbers out of 20 in one minute. Grade 5 students’ average score was 84.4 percent which translates to the average Grade 5 student being able to correctly read 17 numbers in one minute.

Task 2: Number Discrimination To assess students’ ability to discriminate between numbers up to 999, an untimed task was included in the EGMA (Task 2). On this task, students were presented with a pair of numbers and asked to identify (read) the larger number. Grade 3 students’ average score was 58.6 percent; in other words, the average Grade 3 student could identify the larger number in 12 of the 20 pairs presented to them. Grade 5 students’ average score was 81.0 percent; in other words, the average Grade 5 student correctly identified the larger number in 16 of the 20 pairs.

Task 3: Missing Number To assess students’ ability to identify a missing number when given a pattern, an untimed task was included on the EGMA (Task 3). Of all EGMA tasks, Grade 3 students had the lowest average score on this task; Grade 5 had the second lowest average score in comparison to all other tasks. The average score for Grade 3 students was 28.2 percent which translates to the average Grade 3 student being able to correctly identify missing numbers in three out of 10 patterns. Grade 5 students’ average score was 45.3 which translates to the average Grade 5 student identifying the missing number correctly for almost five out of 10 number patterns.

Tasks 4 and 5: Addition and Subtraction29 To assess students’ computation skills, two timed tasks on addition and subtraction were included on the EGMA. On these tasks, students were provided with a printed list of 20 addition problems consisting of two single-digit numbers with sums no greater than 10, and a printed list of 20 subtraction problems consisting of two single-digit numbers under 10 (e.g. 9 - 2 = …. ).

For students in Grade 3, the average score in addition (Task 4) and subtraction (Task 5) was 39.6 percent and 33.9 percent, respectively. In other words, the average Grade 3 student correctly solved 8 addition problems and 7 subtraction problems out of 20 problems for each task in one minute.

For students in Grade 5, the average score in addition (Task 4) and subtraction (Task 5) was 59.4 percent and 53.9 percent, respectively. In other words, the average Grade 5 student correctly solved 12 addition and 11 subtraction problems out of 20 problems for each task in one minute.

29 Level 1.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 39 Task 6: Word Problems To assess students’ ability to solve mathematical computations in the context of a word problem, an untimed task was included on the EGMA (Task 6). Students were given six word problems for which one or more computations were needed in order to arrive at the correct answer. Grade 3 students’ average score was 54.4; in other words, the average Grade 3 student was able to solve three out of six problems. Grade 5 students’ average score was 68.4; in other words, the average Grade 5 student was able to solve four out of six problems correctly.

Task 7: Simple multiplication (Grade 5 only) To assess Grade 5 students’ multiplication skills, a timed task was included on the EGMA (Task 7). For this task, students were provided with a printed list of 20 multiplication problems in which they had to mentally multiply two single-digit numbers (e.g. 4 x 3 =…) to arrive at the correct answer. This task proved to be the most difficult for Grade 5 students – their average score was 27.7 percent correct. In other words, Grade 5 students, on average, were able to compute just under six out of 20 multiplication problems correctly in one minute.

4.4 EGMA Results by Gender

EGMA results were further disaggregated by gender to determine how girls’ and boys’ performance on each task compared. At the task level, boys in both grades significantly outperformed girls in each of the EGMA tasks. Figure 18 shows the average score, by gender, for Grade 3 and Figure 19 shows the average score, by gender, for Grade 5. In Grade 3, boys had a higher overall score on the EGMA than girls by 12 percentage points. In Grade 5, boys outperformed girls, overall, by 8.5 percentage points. At the task level, boys in both grades significantly outperformed girls in all tasks except the Missing Numbers task.

FIGURE 17: GRADE 3 EGMA SCORES BY GENDER

73.4% 64.7% 56.7% 52.0% 45.8% Gr.3 Boys Avg. Score 50.7% 49.3% Gr. 3 Girls Avg. Score 42.7% 38.7% 37.7% 32.8% 29.0% 26.2% 26.0%

Number Number Missing Number Addition* Subtraction* Word Problems* Identification* Discrimination*

G3 Boys G3 Girls G3 Boys Average G3 Girls Average

*Indicates that the performance of the group was significantly higher, p< 0.05

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 40 FIGURE 18: GRADE 5 EGMA SCORES BY GENDER

88.6% 84.4% 75.7% 74.2% 71.6% Gr.5 Boys Avg. 61.9% Score 68.2% 61.4% 56.7% 46.8% 48.5% Gr. 5 Girls Avg. 42.5% 55.6% Score 59.7%

28.6% 26.0%

Number Number Missing Number Addition* Subtraction* Word Problems* Multiplication Identification* Discrimination*

G5 Boys G5 Girls G5 Boys Average G5 Girls Average

*Indicates that the performance of the group was significantly higher, p< 0.05

For timed tasks, fluency rates were also disaggregated by gender (Table 18). In Grade 3, boys significantly outperformed girls in both Addition and Subtraction tasks. In Grade 5, boys outperformed girls in Addition and Subtraction tasks.

TABLE 18: MATHEMATICS COMPUTATION FLUENCY RATE MEANS BY GRADE AND GENDER

Mathematics Fact Fluency Grade 3 Grade 5 Subtest (Task Number and Name) Boys Girls Boys Girls 4. Addition 9.5* 7.0 (-2.5) 14.9* 12.5 (2.4) 5. Subtraction 7.8* 5.3 (-2.5) 12.4* 10.4 (2.0) 7. Multiplication -- -- 6.3 5.5 (0.8) *Indicates that the performance of the group was significantly higher, p< 0.001

4.5 EGMA Proportion of Zero-Scores

The results presented thus far examine the percent of items answered correctly, or fluency rates – i.e., what students were actually able to do, on average. The proportion of zero-scores, however, provides information on the number of students who were unable to correctly answer even a single item on a task.

Table 19 shows the proportion of zero-score students by grade and EGMA task. As noted with the EGRA results, zero-scores provide critical warning bells to a system, highlighting students who will continue to be left behind if not provided significant remedial help.

In both grades, some of the highest proportion of zero-scores were observed in addition and subtraction compared to all other tasks. Approximately two in five Grade 3 students had zero-scores in addition (37.5 percent) and in subtraction (40.8 percent). In Grade 5, half of students had zero-scores in multiplication (50.3 percent), and approximately one-quarter had zero-scores in addition (22.0 percent) and in subtraction (25.5 percent). The fewest number of zero-score students in both grades were observed in Number Identification and Word Problems tasks.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 41 TABLE 19: EGMA PROPORTION OF STUDENTS WITH ZERO-SCORES BY GRADE AND TASK

Grade 3 Grade 5 Task (Subtest) (% students) (% students) 1. Number identification 4.9% 2.0% 2. Number discrimination 11.9% 3.5% 3. Missing Number 25.6% 13.2% 4. Addition 37.5% 22.0% 5. Subtraction 40.8% 25.5% 6. Word Problems 9.0% 3.1% 7. Multiplication -- 50.3%

Disaggregating the proportion of zero-score students (Table 20) by gender shows that in Grade 3 and in Grade 5, the proportion of girls with zero-scores was significantly higher in Number Identification, Discrimination and computation tasks.

TABLE 20: EGMA PROPORTION OF ZERO-SCORES BY GRADE, GENDER AND TASK

Grade 3 Grade 5 Task (Subtest) Boys Girls Boys Girls 1. Number identification 2.1% 10.5% * 1.0% 4.1%* 2. Number discrimination 8.0% 19.8* 2.2% 5.8%* 3. Missing Number 25.2 26.5 11.8% 16.1% 4. Addition 33.6% 45.5%* 21.0% 24.0% 5. Subtraction 35.7% 51.4% * 22.8% 31.0%* 6. Word Problems 7.2% 12.8% 2.6% 4.1% 7. Multiplication -- -- 49.5% 52.2% *Indicates that the performance of the group was significantly higher, p< 0.05

4.6 EGMA Summary and Recommendations

The SRP EGMA baseline results provide useful information to help guide interventions to improve mathematics for Kashmore district students. A summary of findings, by grade, is presented below followed by recommendations. Summary Grade 3: Grade 3 students scored highest on number identification, number discrimination, and word problems; the same tasks in which the proportion of zero-scores was lowest. Notably, results for word problems were stronger than the results for Tasks 3-5 (Missing Number, Addition and Subtraction) which suggests that, for many students, computations in the context of word problems were easier than isolated computation problems. Grade 3 students scored lowest on the Missing number and computation tasks (Addition and Subtraction); computation tasks had the highest proportion of students with zero- scores.

Grade 5: Grade 5 students scored highest in number identification, number discrimination and word problems, while they scored lowest on multiplication and missing number. The relatively higher

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 42 performance in word problems compared to computation is notable among Grade 5 students as with Grade 3 students. Furthermore, half of Grade 5 students were unable to answer a single question correctly on the Multiplication task, and approximately one-quarter of students could not answer a single addition or subtraction problem correctly. Recommendations The SRP EGMA baseline results provide useful information to help guide interventions to improve mathematics skills among students in Kashmore district. The following recommendations are based on these results to strengthen mathematics instruction:

• Provide teachers with strategies for building number sense and computational approaches, which would enable students to identify numbers and patterns and build computational skills (addition, subtraction, and multiplication). • Provide teachers with strategies for diagnosing students experiencing difficulties in number sense and computation, and formative assessment to ensure students who lack foundational mathematics skills are identified early. • Examine trends by sub-groups and geography to determine root causes for the poor performance among girls in both Grades 3 and 5 relative to that of boys. Use these findings to develop targeted intervention for girls. • Incorporate the premise that the learning of all students is enhanced when there is adequate challenge (“not too easy, not too hard”). Recognize that behavioral concerns will most often be displayed when there is not an instructional match for the student (the work being given to the student is either too easy or too difficult). Providing teachers with professional learning in “instructional assessment” is key to student success. “Instructional assessment” incorporates the notion that instruction and materials are matched to the student’s individual skills, prior knowledge, and interests.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 43 CHAPTER 5: CONTEXTUAL VARIABLES AND EGRA AND EGMA PERFORMANCE

The SRP baseline study also included questionnaires in which a series of questions were posed to the students taking the EGRA and EGMA tests and to their teachers and head teachers. The questions were designed to capture contextual variables, including student, teacher, and Head Teacher characteristics to examine the relationships between these contextual variables and students’ reading and mathematics performance. To this end, responses to the questionnaire were summarized and correlated with the EGRA and EGMA summary scores. The goal of this analysis was to answer the following type of question: “does having reading materials at home correspond to higher EGRA/EGMA scores?” Analyses of these variables did not reveal many significant relationships, but this may have been due to the small sample size of questionnaire results, particularly with the teacher and head teacher data.30 While these relationships are useful to examine as contextual factors that may positively or negatively influence students reading and mathematics performance, the generalizability of these data is limited and, therefore, the results must be interpreted with caution. Key Findings Student questionnaire: • Reading materials at home: One-quarter of Grade 3 and one-fifth of Grade 5 students said they had no reading materials at home. Students who had reading materials at home had significantly higher EGRA and EGMA scores. • Practice reading at home: About half of students in both Grades reported that someone at home read stories aloud to them; this was associated with higher EGRA scores among Grade 5 students. Slightly more than half of students reported that they practiced reading stories aloud to someone in their home; this was associated with significantly higher EGRA scores among Grade 3 students. Grade 5 students who practiced reading silently at home had significantly higher EGRA scores as did Grade 3 students who reported reading the Quran at home. • Work with peers: About 60% of students reported completing classwork with peers and this was associated with higher assessment scores. • Absenteeism: Just under one-third of students missed days from school the previous week; in Grade 5, these students’ EGRA and EGMA scores were lower than those who were not absent. • Work outside of school: Almost all students reported working before or after school, but this was associated with higher EGRA scores among Grade 3 students only. • Access to Communication Technology: About a third of students reported listening to the radio at home; Grade 5 students who listened had slightly higher EGMA scores. Teacher and head teacher questionnaires: • Teacher qualifications: No significant differences were associated with teacher qualifications in EGRA or EGMA scores. • Multi-grade classroom: More than three quarters of teachers taught multi-grade classrooms. Teaching multi-grade classrooms was not associated with student assessment scores. • Timetable implementation: Seventy-one percent of head teachers indicated that their schools followed a timetable; of those who did follow a timetable, about half set aside time for teaching reading and math. Including reading in the school timetable was not associated with student scores. • Head Teacher Support: Almost all head teachers reported supporting teachers in reading and math instruction, those who supported teachers on how to teach reading skills had higher Grade 3

30 Reasons that responses from the Questionnaire must be treated with caution include: the questionnaire was not piloted; small samples can lead to a high margin of error (the number of teachers and head teachers surveyed in each district is relatively small). Note also that there were some students, teachers, and head teachers who did not respond to certain questionnaire items; they were labeled as missing.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 44 EGRA scores. Interestingly, head teachers monitoring student progress was associated with lower EGMA scores in Grade 5. • School Infrastructure: Over half of schools had facility improvements such as clean water, electricity, and toilet facilities; 42% of schools had gender separate toilet facilities at mixed schools; only one school reported having a library. Facility improvements were not associated with higher assessment scores.

Detailed data from student, teacher and head teacher questionnaires are provided in Annex 5 and summarized in the tables below. Table 21 and Table 22 summarize the student and the teacher and head Teacher responses that corresponded with higher scores on the EGRA or EGMA – these results are analyzed and presented by language of the test and not by district (note that results for Kashmore are presented in the following sections). For results by language of the test, results are presented only if the difference was statistically significant; if the result was only statistically significant for one language (Urdu or Sindhi), then the language where significant findings were observed is noted.

TABLE 21: CONTEXTUAL VARIABLES FROM STUDENT QUESTIONNAIRE ASSOCIATED WITH EGRA OR EGMA PERFORMANCE

Grade 3 Grade 5 Student Responses that Corresponded with Higher Scores on the EGRA or EGMA EGRA EGMA EGRA EGMA

Language at home is same as test language * (Urdu) * (Sindhi) Type of reading material at home: Newspapers * *

Type of reading material at home: Magazines * *

Type of reading material at home: Books *

Stories Read Aloud to Someone at Home * Practice Reading Stories Aloud to Someone at Home * * Practice Silent Reading at Home * * * * Read Quran at Home * * * Study or Do Class-work with Classmates * * * * Did Not Miss Any Days of School in the Previous * * * * Week Do Not Work Before or After School * * * Watch Television at Home * Have a Computer at Home * * Type of Vehicle at Home: Motorcycle * Type of Vehicle at Home: Car * * *Summary scores were significantly higher (p < .05)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 45 TABLE 22: CONTEXTUAL VARIABLES FROM TEACHER AND HEAD TEACHER QUESTIONNAIRE ASSOCIATED WITH EGRA OR EGMA PERFORMANCE

Teacher and Head Teacher Responses that Grade 3 Grade 5 Corresponded with Higher Student Scores on the EGRA or EGMA EGRA EGMA EGRA EGMA

Teachers Do not teach multi-grade classroom * * Use flashcards * (Sindhi) Use charts * (Sindhi) * (Sindhi) * (Sindhi) Use novels/books * (Sindhi) * (Sindhi) Use magazines * (Sindhi) * (Sindhi) Have individual meetings with parents * (Sindhi) * (Sindhi) Updating skills through peer support * (Urdu) Updating skills by attending seminars or * (Sindhi) educational conferences Updating skills through in-service training * (Sindhi) * (Sindhi) * (Sindhi) Have language or reading teacher guide * (Urdu) Have mathematics teaching guide * (Urdu) Head Teachers Reviewing student evaluations by supervisors for * (Urdu) Union Council’s *Summary scores were significantly higher (p < .05)

The sections below provide results from the student, teacher and head teacher questionnaires for Kashmore district.

5.1 Student Questionnaire

The 1,519 students participating in the EGRA and EGMA assessments in the Kashmore district were also surveyed with the student questionnaire. Student questions related to language spoken at home; availability of reading materials at home; study habits; access to communication technology; and vehicles at home. Language Spoken at Home Students were asked what primary language was spoken in the home. About two-thirds of families spoke Sindhi and another 30% spoke Balochi; the remaining 10 percent spoke Siraeki, Punjab, Urdu or another language at home. Although the assessments were in Sindhi, more than a third of Kashmore students in both Grades spoke a different primary language at home. Overall, the language spoken in the home was not associated with performance on EGRA and EGMA. Reading at Home Students were asked what types of reading materials were found at their homes (newspapers, magazines, books, or nothing). One-quarter of Grade 3 and one-fifth of Grade 5 students said they had no reading materials at home. Students who had reading materials at home had significantly higher EGRA and EGMA scores.

Students were also asked about their reading habits at home (does someone read aloud to them; do they practice reading aloud to anyone; do they practice silent reading; do they read the Quran at home). About half of students in both Grades reported that someone at home read stories aloud to them; this was associated with higher EGRA scores among Grade 5 students. Slightly more than half of students reported that they practiced reading stories aloud to someone in their home; this was associated with significantly higher EGRA scores among Grade 3 students. Grade 5 students who practiced reading

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 46 silently at home had significantly higher EGRA scores as did Grade 3 students who reported reading the Quran at home. Study Habits, School Attendance, Parents’ Knowledge of School, Work Duties Students were asked questions about study habits, school attendance, parents’ knowledge of school performance and work-related duties. About 60% of students reported completing classwork with peers and this was associated with higher EGMA scores among Grade 3 students. Just under one-third of students missed days from school the previous week; in Grade 5, these students’ EGRA and EGMA scores were lower than those who were not absent.

When students were asked about their parents’ knowledge of their performance in school, slightly less than a half of students reported that parents had no knowledge of their school performance, which was associated with a small negative difference in EGRA and EGMA scores in Grade 3.

When asked about work obligations or recreation time outside the classroom, almost all students (over 85 percent) reported working before or after school, but this was associated with higher EGRA scores among Grade 3 students only. Communication Technology and Vehicles at Home When asked about access to communication technology at home, about 70% of students reported watching television at home and in Grade 5. About a third of students reported listening to the radio at home; Grade 5 students who listened had slightly higher EGMA scores. Only ten percent of students reported having computers at home, but this did not correspond to higher test scores. (Ninety percent of students did not respond to the question about internet access at home, so this data is not presented.)

The final section of the student questionnaire asked what types of vehicles were at home (bicycle, motorcycle, or car). About one-fifth of students reported having none of these vehicles at home, but this did not correspond with a statistically significant difference in test results.

5.2 Teacher and Head Teacher Questionnaires

In Kashmore district, 125 teachers and 69 head teachers were surveyed on questions pertaining to qualifications, attendance at training sessions, and school-related factors such as scheduling, teacher support, head teacher’s knowledge of student performance, teacher observation procedures, School Management Committee (SMC) authority, and facility improvements. Of the 125 teachers surveyed, 62 were Grade 3 teachers (40 female and 21 male), and 63 were Grade 5 teachers (39 female and 24 male). Of the 70 head teachers surveyed, 25 were female and 44 were male teachers. Teacher Qualifications Roughly half of teachers reported having a B.A. /B.Sc. and one quarter of teachers a F.A/F.Sc. as their highest academic qualification. Most teachers had masters or bachelors academic and professional qualifications, but teacher qualifications did not have an impact on assessment scores. More than three quarters of teachers taught multi-grade classrooms. Teaching multi-grade classrooms was not associated with student assessment scores. Teaching Materials and Practices About 30% of teachers reported using books other than text books in the classroom but this was not associated with student assessment scores. Using other classroom tools did not correspond with higher test scores.

Almost all teachers conduct individual meetings with parents, had head teachers who encourage them to use a variety of activities, and give extra time to slow learners after class. These practices however, were not associated with higher student performance.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 47 Updating Teaching Methods Teachers were asked to select the method in which they keep current on teaching methods. Almost all respondents noted reading books as their method of updating knowledge; seeking peer support, in- service training and attending seminars were identified by 15-30% of respondents. Very few teachers (6%) turned to the internet for updating their pedagogical knowledge. None of these methods corresponded with higher test scores. The sample size was very small for this topic and results should be interpreted with caution. Head Teacher Qualifications As with teachers, most head teachers had B.A./B.S degrees and a B.Ed. as certification; no significant differences in the assessments scores were found related to head teacher academic or professional qualifications. Head Teacher Classroom Experience and Training Most head teachers (81%) reported teaching classes. Under 30% of head teachers reported having received any school management training in the previous year; about sixty percent had received training for implementing reading programs and 36% for math programs. There were no significant differences in assessment scores between head teachers who reported receiving in-service training those who did not. Timetable Implementation The next series of questions pertained to the use of timetables and assigning specific periods to reading and mathematics. Seventy-one percent of head teachers indicated that their schools followed a timetable; of those who did follow a timetable, about half set aside time for teaching reading and math. Including reading in the school timetable was not associated with student scores. Head Teacher Support and Knowledge of Student Progress Almost all head teachers reported supporting teachers in reading and math instruction, those who supported teachers on how to teach reading skills had higher Grade 3 EGRA scores. Interestingly, head teachers monitoring student progress was associated with lower EGMA scores in Grade 5. Head Teacher Observation of Teachers More than half of head teachers reported observing teachers’ lesson plans and this was associated with significantly higher EGMA scores in Grade 5. About half of respondents indicated that the head teacher had responsibility for observing teachers (others said no one or the Deputy Teacher). SMC Decision Making Authority and Facility Improvement Sixty-seven head teachers indicated that their school had an SMC. No SMC involvement or responsibility was associated with consistently higher assessment scores.

Over half of schools had facility improvements such as clean water, electricity, and toilet facilities; 42% of schools had gender separate toilet facilities at mixed schools; only one school reported having a library. Facility improvements were not associated with higher assessment scores.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 48 ANNEXES

Annex 1. Quality of Assessment Tools – Additional Data

i. Cronbach’s alpha

TABLE 23: KASHMORE RELIABILITY ESTIMATES

Assessment Grade Level Tasks Number Alpha Grade 3 6 744 0.79 Reading Grade 5 8 745 0.89 Grade 3 6 744 0.90 Mathematics Grade 5 7 745 0.88

ii. P-values and Item-total correlations

TABLE 24: EGRA TASKS STATISTICS FOR KASHMORE

Grade 3 Grade 5 Task (Subtest) Item- Item- p-value p-value Total Total 1. Phoneme Isolation .07 .36 .09 .25 2. Non-word Reading (Decoding) .30 .78 .49 .76 3. Expressive Vocabulary .59 .32 .61 .45 4a. Passage Reading (Grade 2-level) .47 .74 .75 .82 4b. Reading Comprehension .26 .79 .52 .84 5. Listening Comprehension .58 .34 .67 .46 6a. Passage reading (Grade 4-level) -- -- .76 .80 6b. Reading comprehension (Grade 4-level) -- -- .55 .85 Reading Summary Score .38 -- .52 --

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 49 TABLE 25: EGMA TASKS STATISTICS FOR KASHMORE

Grade 3 Grade 5 Task (Subtest) p-value Item-Total p-value Item-Total 1. Number identification up to 999 .66 .78 .84 .67 2. Number discrimination up to 999 .59 .79 .81 .72 3. Missing Number up to 999 .28 .65 .45 .60 4. Addition level 1 within 10 .40 .78 .59 .72 5. Subtraction level 1 within 10 .34 .78 .54 .79 6. Word Problems .54 .64 .68 .64 7. Multiplication up to 2 digits -- -- .28 .61 EGMA Summary Score .47 -- .65 --

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 50 Annex 2: Score Calculation Definitions Percent Correct The results of the EGRA testing were calculated using task and summary scores. Table 25 lists the tasks, stimuli, raw score ranges, and the method for calculating the task and summary scores on the test. For each of the tasks, the stimuli (items) (i.e., questions, letters, sounds, words, and non-words) were worth one score point. The score points were added and since the range of raw scores varies across the tasks, the percent of correct scores was used to report all results. No weighting was used with the tasks to calculate the summary scores. Each task summary score was calculated using the total of the percent correct scores and dividing it by the number of items (i.e., the average percent correct). The overall Reading Summary Score was calculated by adding all of the task summary scores and dividing by nine (total number of tasks) to arrive at the average. Timed Tasks Scores The scores on the timed tasks were calculated (adjusted) by taking the number of correct responses (i.e., the raw score) times 60 seconds then dividing that number by the number of seconds used to read the stimulus. For instance, if a student read 25 letters correctly in 30 seconds, their letters-correct-per-minute score would be 50 (25 words x 60 seconds/30 seconds). Given another example, if a student completes 12 additions correctly in 30 seconds, his or her timed task score would be 24 additions per minute (12 additions x 60 seconds/30 seconds). EGRA Score Ranges and Calculation

TABLE 26: EGRA GRADE 3 SCORE RANGES AND CALCULATIONS

Task (Subtest) Stimuli Score Range Calculation 1. Phoneme Isolation 10 questions (untimed) 0-100 Percent correct of answers 2. Non-word Reading (Decoding) 50 non-words (timed) >100 Words correct per minute 3. Expressive Vocabulary 10 pictures (untimed) 0-100 Percent correct of words 4a. Passage Reading (Grade 2-level) 60 words (timed) >100 Words per minute 4b. Reading Comprehension 5 questions (untimed) 0-100 Percent correct of answers 5. Listening Comprehension 3 questions (untimed) 0-100 Percent correct of answers Reading Summary Score 6 tasks 0-100 Average of percent correct

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 51 TABLE 27: EGRA GRADE 5 SCORE RANGES AND CALCULATIONS

Score Task (Subtest) Stimuli Calculation Range 1. Phoneme Isolation 10 questions (untimed) 0-10 Percent correct of answers 2. Non-word Reading (Decoding) 50 words (timed) >100 Words correct per minute 3. Expressive Vocabulary 10 pictures (untimed) 0-100 Percent correct of words 4a. Passage Reading (Grade 2-level) 60 words (timed) >100 Words correct per minute 4b. Reading Comprehension 5 questions (untimed) 0-100 Percent correct of answers 5. Listening Comprehension 3 questions (untimed) 0-100 Percent correct of answers 6a. Passage reading (Grade 4-level) 60 words (timed) >100 Words correct per minute 6b. Reading comprehension 5 questions 0-100 Percent correct of answers (Grade 4-level) Reading Summary Score 6 tasks 0-100 Average of percent correct

An example of percent correct scores for each of the tasks and as a summary score is provided below. The raw score is divided by the maximum score (the highest score possible in the score range) to produce the percent correct score for each task. Then, the task scores are averaged to produce the summary score. Note that each of the task percent correct scores is weighted equally to provide the summary score.

TABLE 28: EXAMPLE OF EGRA PERCENT CORRECT AND SUMMARY SCORES

Task (Subtest) Maximum Score Raw Score % Correct Score 1. Phoneme Isolation 10 6 60% 3. Expressive Vocabulary 10 5 50% 4b. Reading Comprehension (Grade 2-level) 5 3 60% 5. Listening Comprehension 3 3 100% 6b.Reading Comprehension (Grade 4-level) 5 4 80% 7b. Passage comprehension 5 2 40% Reading Summary Score -- -- 55.0%

An example of timed task scores (adjusted) is provided below for the five fluency tasks. The formula explained above is used (timed task score = raw score x 60 seconds/seconds used).

TABLE 29: EXAMPLE OF EGRA TIMED TASK SCORES

Task (Subtest) Raw Score Seconds Used Timed Task Score 2. Non-word reading 25 40 37.5 4a or 6a. Passage reading 50 40 75.0

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 52 EGMA Score Ranges and Calculation

TABLE 30: EGMA GRADE 3 SCORE RANGES AND CALCULATIONS

Score Task (Subtest) Stimuli Calculation Range Percent correct numbers per 1. Number Identification up to 999 20 numbers (timed) >100 minute 2.Number Discrimination up to 10 questions 0-100 Percent correct of numbers 999 (untimed) 10 numbers 3. Missing Number up to 999 0-100 Percent correct of numbers (untimed) 4. Addition level 1 within 10 20 problems (timed) >100 Number correct per minute 5. Subtraction level 1 within 10 20 problems (timed) >100 Number correct per minute 6 Problems 6. Word Problems 0-100 Percent correct of answers (untimed) Mathematics Summary Score 6 tasks 0-100 Average of percent correct

TABLE 31: EGMA GRADE 5 SCORE RANGES AND CALCULATIONS

Score Task (Subtest) Stimuli Calculation Range 1. Number Identification up to Percent correct numbers per 20 numbers (timed) >100 999 minute 2.Number Discrimination up to 10 numbers 0-100 Percent correct of numbers 999 (untimed) 10 numbers 3. Missing Number up to 999 0-100 Percent correct of numbers (untimed) 4. Addition level 1 within 10 20 problems (timed) >100 Number correct per minute 5. Subtraction level 1 within 10 20 problems (timed) >100 Number correct per minute 6 Problems 6. Word Problems 0-100 Percent correct of answers (untimed) 7. Multiplication up to 2 digits 20 problems (timed) >100 Number correct per minute Mathematics Summary Score 7 tasks 0-100 Average of percent correct

An example of percent correct scores for each of the tasks and as a summary score is provided below. The raw score is divided by the maximum score (the highest score possible in the score range) to produce the percent correct score for each task. Then, the task scores are averaged to produce the summary score. Note that each of the task percent correct scores is weighted equally to provide the summary score.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 53 TABLE 32: EXAMPLE OF EGMA PERCENT CORRECT AND SUMMARY SCORES

Task (Subtest) Maximum Score Raw Score % Correct Score 1. Number Identification up to 999 20 6 30.0% 2.Number Discrimination up to 999 10 8 80.0% 3. Missing Number up to 999 10 5 50.0% 4. Addition level 1 within 10 20 42 42.0% 5. Subtraction level 1 within 10 20 14 70.0% 6. Word Problems 6 3 50.0% 7. Multiplication up to 2 digits 20 18 90.0% Mathematics Summary Score -- -- 55.0%

An example of timed task scores (adjusted) is provided below for the five timed accuracy tasks. The formula explained above is used (timed task score = raw score x 60 seconds/seconds used).

TABLE 33: EXAMPLE OF EGMA TIMED TASK SCORES

Task (Subtest) Raw Score Seconds Used Timed Task Score 1. Number Identification 25 40 37.5 4. Addition Level 1 within 10 50 40 75.0

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 54 Annex 3: EGRA Item Statistics by Grade

Table 33 presents statistics for each EGRA item on the untimed tasks. The untimed tasks each have multiple items. For instance, task 1 (phoneme isolation) has ten items, and each has item statistics. Note that the timed tasks are lists of letters, sounds, and words, i.e., not items, so it is not necessary to calculate item statistics for them.

Task statistics are presented in the body of the report. These item statistics are calculated in the same way. They show the difficulty and quality of the items. Recall that when constructing a test, we strive for tasks and items that have difficulty values (p-values) that are spread across the range from about 0.1 to 0.90. For Kashmore, the difficulty values ranged from 0.04 to 0.76, indicating a strong range of item difficulties. At Grade 3, total of 17 out of the 18 items had high item-total correlations. At Grade 5, 22 out of the 23 items were high quality items.

TABLE 34: COMPLETE EGRA ITEM STATISTICS BY GRADE FOR KASHMORE

Grade 3 Grade 5 Task (Subtest) Item P-Value Item-Total P-Value Item-Total Q1 .06 .73 .10 .72 Q2 .08 .76 .10 .76 Q3 .06 .81 .10 .81 Q4 .09 .80 .13 .77 Q5 .09 .72 .11 .77 1. Phoneme Isolation Q6 .06 .72 .09 .76 Q7 .06 .77 .10 .76 Q8 .04 .63 .06 .56 Q9 .08 .81 .11 .82 Q10 .04 .70 .06 .56 2. Non-word reading (decoding) ------3. Expressive Vocabulary ------Q1 .33 .71 .63 .67 Q2 .29 .72 .63 .66 4. Passage Reading Q3 .19 .60 .39 .51 (Grade 2-level) Q4 .31 .64 .57 .61 Q5 .17 .58 .37 .47 Q1 .39 .14 .46 .27 5. Reading Comprehension Q2 .74 .30 .84 .34 Q3 .61 .23 .71 .35 Q1 -- -- .65 .63 6. Passage Reading Q2 -- -- .66 .70 (Grade 4-level) Q3 -- -- .60 .69 Q4 -- -- .41 .53 For the mathematics assessment the difficulty values ranged from 0.05 to 0.94, indicating a very strong range of item difficulties. All items had high item-total correlations for both grades indicating high quality items.

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 55 TABLE 35: COMPLETE EGMA ITEM STATISTICS BY GRADE FOR KASHMORE

Grade 3 Grade 5 Task (Subtest) Item P-Value Item-Total P-Value Item-Total 1. Number Identification ------Q1 .85 .51 .95 .52 Q2 .64 .65 .83 .67 Q3 .66 .76 .88 .70 Q4 .73 .73 .89 .67 2. Number Discrimination Q5 .59 .79 .81 .73 Task 2.3 Q6 .54 .80 .78 .69 Q7 .38 .61 .65 .53 Q8 .45 .73 .71 .65 Q9 .50 .81 .79 .76 Q10 .54 .83 .80 .75 Q1 .50 .32 .65 .42 Q2 .55 .57 .76 .58 Q3 .41 .61 .58 .62 Q4 .41 .62 .59 .62 Q5 .13 .38 .25 .42 3. Missing Number Q6 .34 .60 .61 .61 Q7 .11 .49 .26 .48 Q8 .13 .51 .33 .57 Q9 .19 .63 .39 .63 Q10 .05 .32 .13 .34 4. Addition, Level 1 ------5. Subtraction, Level 1 ------Q1 .83 .36 .91 .33 Q2 .67 .53 .80 .35 Q3 .32 .47 .52 .36 6. Word Problems Q4 .55 .35 .65 .20 Q5 .41 .47 .58 .42 Q6 .50 .56 .64 .47 7. Simple Multiplication ------

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 56 Annex 4: Box Plots for Phonics and Reading-rate Fluency Tasks

Recently, greater emphasis has been placed on the results of the fluency (timed) tasks over other tasks. In addition to the descriptive statistics presented in the body of the report, we show box plots for the different fluency tasks. Widely used since their development in the 1960s, box plots are a convenient way for graphically presenting numerical data.

Box plots have two characteristics: 1) central tendency (i.e., the median, or the middle score in the data) and 2) variation (i.e., the range, with scores grouped by quartile). The boxes (which are actually rectangles) represent the two middle quartiles of the scores and the “whiskers” represent the upper and lower quartiles. The small circles on the ends of the whiskers represent outliers. Figure 20 provides a more detailed explanation for interpreting box plots.

FIGURE 19: EXPLANATION OF BOX PLOTS

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 57 Box plots are presented below for reading fluency and mathematics-fact fluency for both grades. At Grade 3, non-word reading and Grade level 2 passage reading fluency are presented. At grade 5 an additional task of Grade 4-level reading was generated. The EGMA mathematics fluency scores follow. At Grade 3 number identification, addition and subtraction are plotted. Simple multiplication was also included at Grade 5. Reading Fluency Box Plots The fluency rates for non-word reading (decoding, the far left box plot) were much lower for both grades. The decoding variation (i.e., the range of scores, without outliers) for both was less than 30 words per minute. The reading fluency variation was 40 words at Grade 3. At Grade 5 there was a much larger reading rate variation, over 80 words per minute, indicating a wider range of reading fluency abilities.

FIGURE 20: PHONICS AND READING- FIGURE 21: PHONICS AND READING- RATE FLUENCY BOX PLOTS FOR RATE FLUENCY BOX PLOTS FOR GRADE 3 GRADE 5

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 58 Mathematics Fluency Box Plots Mathematics fluency scores are plotted below. The box plots infer that most Kashmore students solve fewer than 20 mathematics problems per minute. A few students (third quartile) can solve over 20 at Grade 3 and over 75 problems at Grade 5. At Grade 3 addition and subtraction problem solving rates are much lower and indicate that many students solve basic mathematics problems slowly. At grade 5, box plots are relatively higher inferring faster problem solving rates. Students were quicker in identifying numbers. Box plot scores for addition and subtraction show over 25 percent of students at zero or very low rates because the lower quartile line is not visible.

FIGURE 22: MATHEMATICS FACT-RATE FIGURE 23: MATHEMATICS FACT-RATE FLUENCY BOX PLOTS FOR GRADE 3 FLUENCY BOX PLOTS FOR GRADE 5

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 59 Annex 5: Questionnaire Results Student Questionnaire Tables

TABLE 36: DISTRIBUTION OF STUDENTS BY PRIMARY LANGUAGE SPOKEN AT HOME

Grade 3 Grade 5 Language Number Percent Number Percent Sindhi 515 66.7% 468 60.6% Siraeki 20 2.6% 48 6.2% Balochi 230 29.8% 217 28.1% Punjabi 0 0.0% 3 0.4% Urdu 2 0.3% 1 0.1% Other 5 0.6% 5 0.6% Missing 2 0.3% 3 0.4% Total 772 100.0% 745 100.0%

TABLE 37: DISTRIBUTION OF STUDENTS SPEAKING THE SAME LANGUAGE AT HOME AS THE TEST

Home Grade 3 Grade 5 Language Same as Number Percent Number Percent Test No 259 34% 277 37% Yes 515 67% 468 63% Missing 3 0.4% 0 0.0% Total 774 100.0% 745 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 60 TABLE 38: AVERAGE EGRA/EGMA SCORES BY READING MATERIAL AT HOME

Grade 3 Grade 5 Avg Avg Avg Avg Reading EGRA EGMA EGRA EGMA Material Type Number Number Sum Sum Sum Sum Score Score Score Score 295 Yes 41.9%** 53.2%** 291 (39%) 55.3%* 67.6% Newspapers (38%) No 479 35.2% 42.9% 454 50.4% 64.0% Yes 18 (2%) 46.6% 55.9% 13 (2%) 67.7%* 68.9% Magazines No 756 37.6% 46.6% 732 52.0% 65.3% 234 Yes 40.3%* 50.2%* 306 (41%) 56.6%** 70.0%** Books (30%) No 540 36.6% 45.4% 439 49.3% 62.2% 204 Yes 30.9% 38.5% 141 (19%) 43.9% 58.4% Nothing (26%) No 570 40.2%** 49.8%** 604 54.3%** 67.1%** Total 774 37.7% 46.8% 745 52.3% 65.4% *Category summary scores were significantly higher (p < .05)

TABLE 39: PERCENTAGE OF STUDENTS BY STORIES READ ALOUD TO STUDENT AT HOME

Stories Read Aloud Grade 3 Grade 5 to Student at Home Number Percent Number Percent No 430 56% 379 51% Yes 337 44% 365 49% Missing 7 0.9% 1 0.1% Total 774 100.0% 745 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 61 TABLE 40: AVERAGE EGRA/EGMA SCORES BY PRACTICE READING STORIES ALOUD TO SOMEONE AT HOME

Grade 3 Grade 5 Avg Practice Reading Avg Avg Avg EGRA Stories Aloud to EGRA EGMA EGMA Number Number Sum Someone at Home Sum Sum Sum Score Score Score Score

Yes 430 (55.6%) 38.2% 45.7% 617 (82.8%) 53.7%* 66.6%* No 337 37.4% 48.6% 125 45.9% 60.3% Missing 3 -- -- 0 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher (p < .05)

TABLE 41: AVERAGE EGRA/EGMA SCORES BY SILENT READING AT HOME

Grade 3 Grade 5 Avg Avg Avg Avg Practice Silent EGRA EGMA EGRA EGMA Reading at Home Number Number Sum Sum Sum Sum Score Score Score Score Yes 582 (75.2%) 38.3% 47.5% 617 (82.8%) 53.7%** 66.6%* No 186 36.2% 45.0% 125 45.9% 60.3% Missing 6 -- -- 3 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher (p < .05) **Summary scores were significantly higher (p < .01)

TABLE 42: AVERAGE EGRA/EGMA SCORES BY READ QURAN AT HOME

Grade 3 Grade 5 Avg Avg Avg Avg Read Quran at EGRA EGMA EGRA EGMA Home Number Sum Sum Number Sum Sum Score Score Score Score Yes 663 (85.7%) 39.4%** 47.6%* 694 (93.2%) 53.4%** 66.2%* No 105 27.6% 41.6% 43 37.5 54.8% Missing 2 -- -- 8 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher (p < .05) **Summary scores were significantly higher (p < .01)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 62 TABLE 43: AVERAGE EGRA/EGMA SCORES BY STUDY OR COMPLETE CLASSWORK WITH CLASSMATES

Grade 3 Grade 5 Study or Do Class- Avg Avg Avg Avg work with EGRA EGMA EGRA EGMA Classmates Number Sum Sum Number Sum Sum Score Score Score Score Yes 366 (47%) 40.8%* 49.3%* 516 (69%) 53.4%* 66.6% No 388 35.5% 45.4% 220 49.8% 63.1% Missing 15 -- -- 4 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher (p < .05)

TABLE 44: AVERAGE EGRA/EGMA SCORES BY STUDENTS MISSING SCHOOL DAYS DURING THE PREVIOUS WEEK

Grade 3 Grade 5 Missed Any Days of Avg Avg Avg Avg School in the EGRA EGMA EGRA EGMA Previous Week Number Sum Sum Number Sum Sum Score Score Score Score Yes 236 (31%) 37.9% 46.5% 478 (64%) 49.4% 61.2% No 532 37.9% 47.3% 267 53.9%* 67.8%* Missing 6 -- -- 0 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher than not ticked (p < .05)

TABLE 45: AVERAGE EGRA/EGMA SCORES BY PARENT’S KNOWLEDGE OF SCHOOL PERFORMANCE

Grade 3 Grade 5 Parents Knowledge Avg Avg Avg Avg of School EGRA EGMA EGRA EGMA Number Number Performance Sum Sum Sum Sum Score Score Score Score Yes 313 (40%) 40.6%* 49.5% 482 (65%) 54.0%* 66.4% No 419 35.8% 45.9% 239 50.0% 64.4% Missing 42 -- -- 24 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Category summary scores were significantly higher (p < .05)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 63 TABLE 46: DISTRIBUTION OF STUDENTS BY STUDENTS REPORTING WORKING BEFORE OR AFTER SCHOOL

Working Before or Grade 3 Grade 5 After School Number Percent Number Percent No 612 79.1% 679 91.1% Yes 155 20.0% 59 7.9% Missing 7 0.9% 7 0.9% Total 774 100.0% 745 100.0%

TABLE 47: AVERAGE EGRA/EGMA SCORES BY STUDENTS WHO WATCH TELEVISION AT HOME

Grade 3 Grade 5 Avg Avg Avg Avg Watch Television at EGRA EGMA EGRA EGMA Home Number Number Sum Sum Sum Sum Score Score Score Score Yes 515 (67%) 38.3% 47.4% 523 (70%) 54.4%* 67.4%* No 259 36.7% 45.7% 221 47.3% 60.6% Missing 0 -- -- 1 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4% *Summary scores were significantly higher (p < .05)

TABLE 48: AVERAGE EGRA/EGMA SCORES BY STUDENTS LISTENING TO THE RADIO AT HOME

Grade 3 Grade 5 Avg Avg Avg Avg Listen to the Radio EGRA EGMA EGRA EGMA at Home Number Number Sum Sum Sum Sum Score Score Score Score Yes 240 (31%) 38.8% 47.9% 321 (43%) 52.9% 67.4%* No 528 37.4% 46.5% 423 51.9% 63.9% Missing 6 -- -- 1 -- -- Total 774 37.7% 46.8% 745 52.3% 65.4%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 64 TABLE 49: DISTRIBUTION OF STUDENTS BY COMPUTER ACCESS AT HOME

Computer Access Grade 3 Grade 5 at Home Number Percent Number Percent Yes 75 9.7% 87 11.7% No 692 89.4% 651 87.4% Missing 7 0.9% 7 0.9% Total 774 100.0% 745 100.0%

TABLE 50: DISTRIBUTION OF STUDENTS BY TYPE OF VEHICLES AT HOME

Grade 3 Grade 5 Vehicle Number Percent Number Percent Bicycle 40 5.2% 64 8.6% Motorcycle 515 66.5% 522 70.1% Car 77 9.9% 77 10.3% None 185 23.9% 150 20.1% Total 774 100.0% 745 100.0%

Teacher Questionnaire Tables

TABLE 51: DISTRIBUTION OF TEACHERS’ ACADEMIC QUALIFICATIONS

Grade 3 Grade 5 Degree Number Percent Number Percent M.A./M.Sc/M.Phil. 16 25.8% 19 30.2% B.A./B.Sc. 38 61.3% 41 65.1% F.A/F.Sc. 5 8.1% 2 3.2% Matric 2 3.2% 0 0.0% Other 1 1.6% 1 1.6% Total 62 100.0% 63 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 65 TABLE 52: DISTRIBUTION OF TEACHERS’ PROFESSIONAL QUALIFICATIONS

Professional Grade 3 Grade 5 Qualification Number Percent Number Percent M.Ed/M.A. Education 6 9.7% 11 17.5% B.Ed. 35 56.5% 35 55.6% ADE 0 0.0% 0 0.0% C.T. 2 3.2% 1 1.6% P.T.C. 16 25.8% 13 20.6% Other 3 4.8% 3 4.8% Total 62 100.0% 63 100.0%

TABLE 53: AVERAGE EGRA/EGMA SCORES BY TEACHING MULTI-GRADE CLASSES

Grade 3 Grade 5 Teaching Multi- Avg Avg Avg Avg EGRA EGMA EGRA EGMA Grade Classroom Number Number Sum Sum Sum Sum Score Score Score Score Yes 45 (73%) 36.0% 44.8% 53 (84%) 51.5% 67.5% No 17 43.8%* 53.1% 10 56.1% 65.0% Total 62 37.7% 46.8% 63 52.3% 65.4% *Summary scores were significantly higher (p < .05)

TABLE 54: AVERAGE EGRA/EGMA SCORES BY TEACHERS USING BOOKS OTHER THAN TEXTBOOKS IN THE CLASSROOM

Grade 3 Grade 5 Using Books Other Avg Avg Avg Avg Than Textbooks in EGRA EGMA EGRA EGMA Number Number Classroom Sum Sum Sum Sum Score Score Score Score Yes 19 (30.6%) 41.7% 48.6% 19 (30.6%) 58.1%* 71.2%* No 41 36.3% 45.6% 40 50.3% 63.6% Missing 2 -- -- 4 -- -- Total 62 37.7% 46.8% 63 52.3% 65.4% *Category summary scores were significantly higher (p < .05)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 66 TABLE 55: DISTRIBUTION OF CLASSROOM TOOLS USED BY TEACHERS

Grade 3 Grade 5 Classroom Tools Number Percent Number Percent Blackboard 60 96.8% 62 98.4% Chalk 60 96.8% 63 100.0% Whiteboard 4 6.5% 3 4.8% Marker 13 21.0% 13 20.6% Writing Paper 20 32.3% 26 41.3% Textbooks 57 91.9% 59 93.7% Books/Storybooks 6 9.7% 11 17.5% Posters 6 9.7% 10 15.9% Flashcards 2 3.2% 0 0.0% Charts 17 27.4% 25 39.7% Math Instruments 2 3.2% 6 9.5% Counters 2 3.2% 2 3.2% Enough Textbooks 57 91.9% 57 90.5% Do you have language or 38 43 reading teacher guides 61.3% 68.3% Do you have mathematics 34 38 teacher guides 54.8% 60.3% Total 62 100.0% 63 100.0%

TABLE 56: DISTRIBUTION OF TEACHING PRACTICES

Grade 3 Grade 5 Teacher Practices Number Percent Number Percent Head Teacher Encourages use of a variety of 59 61 class activities 95.2% 96.8% Give extra time to slow learners after class 51 82.3% 59 93.7% Conduct individual meetings with parents 56 90.3% 55 87.3% Total 62 100.0% 63 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 67 TABLE 57: METHODS BY WHICH TEACHERS’ UDATE THEIR KNOWLEDGE OF TEACHING

Grade 3 Grade 5 Updating Knowledge Number Percent Number Percent Reading books 59 95.2% 61 96.8% Going on the Internet 4 6.5% 4 6.3% Through peer support 9 14.5% 13 20.6% Attending seminars or educational conferences 9 14.5% 12 19.0% In-service training 15 24.2% 20 31.7% Total 62 100.0% 63 100.0%

Head Teacher Questionnaire Tables

TABLE 58: DISTRIBUTION OF HEAD TEACHERS’ ACADEMIC QUALIFICATIONS

Head Teacher Degree Number Percent M.A./M.Sc/M.Phil. 20 29% B.A./B.Sc. 45 65% F.A/F.Sc. 3 4% Matric 1 .01% Total 69 100%

TABLE 59: DISTRIBUTION OF HEAD TEACHERS’ PROFESSIONAL QUALIFICATIONS

Head Teacher Certification Number Percent M.Ed/M.A. Education 11 16% B.Ed. 42 61% C.T. 1 1% P.T.C. 14 20% Missing 1 1% Total 69 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 68 TABLE 60: DISTRIBUTION OF HEAD TEACHERS TEACHING CLASSES

Head Teacher Teaching Class Number Percent Yes 56 81.2% No 12 17.4% Total 69 100.0%

TABLE 61: DISTRIBUTION OF HEAD TEACHERS WHO RECEIVED TRAINING FOR IMPLEMENTING READING PROGRAMS

Training in Head Teacher Implementing Reading Programs Number Percent Yes 40 58% No 25 36% Missing 4 6% Total 69 100%

TABLE 62: DISTRIBUTION OF HEAD TEACHERS WHO RECEIVED TRAINING FOR IMPLEMENTING MATHEMATICS PROGRAMS

Training in Head Teacher Implementing Math Programs Number Percent Yes 25 36.2% No 37 53.6% Missing 7 10% Total 69 100.0%

TABLE 63: DISTRIBUTION OF SCHOOLS HAVING A TIMETABLE

Head Teacher School Timetable Number % Yes 49 71.0% No 20 29.0% Total 69 100.0%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 69 TABLE 64: AVERAGE EGRA/EGMA SCORES BY SCHOOL TIMETABLE

Grade 3 Grade 5 Avg EGRA Avg EGMA Avg EGRA Avg EGMA Timetable Number Sum Sum Sum Sum Score Score Score Score Yes 49 (71%) 39.2% 47.8% 54.6%* 68.0% No 20 34.5% 45.4% 48.0% 60.9% Total 69 37.7% 46.8% 52.3% 65.4%

TABLE 65: AVERAGE EGRA/EGMA SCORES BY INCLUDING READING IN THE TIMETABLE

Grade 3 Grade 5 Avg EGRA Avg EGMA Avg EGRA Avg EGMA Timetable Number Sum Sum Sum Sum Score Score Score Score Yes 34 (49%) 40.4% 50.0% 57.8%* 72.0%* No 14 35.9% 41.9% 46.1% 58.0% Omit (no timetable) 21 ------Total 69 37.7% 46.8% 52.3% 65.4%

TABLE 66: SCHOOLS BY INCLUDING MATH IN THE TIMETABLE

Head Teacher Math Period Number Percent Yes 44 64% No 4 6% Omit (no timetable) 21 30% Total 69 100%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 70 TABLE 67: DISTRIBUTION OF HEAD TEACHERS SUPPORTING READING INSTRUCTION

Support Reading Head Teacher Instruction Number % Yes 64 94.1% No 4 5.9% Total 68 100.0%

TABLE 68: DISTRIBUTION OF HEAD TEACHERS SUPPORTING MATHEMATICS INSTRUCTION

Support Head Teacher Mathematics Instruction Number % Yes 62 92.5% No 5 7.5% Total 67 100.0%

TABLE 69: AVERAGE EGRA/EGMA SCORES BY HEAD TEACHER’S KNOWLEDGE ABOUT STUDENT PROGRESS

Grade 3 Grade 5 Avg Avg Avg Avg Progress Method EGRA EGMA EGRA EGMA Number Sum Sum Sum Sum Score Score Score Score Yes 34 (49%) 38.1% 48.0% 53.0% 65.1% Classroom observation No 35 37.6% 46.2% 52.3% 66.8% Monitor student test Yes 40 (58%) 37.4% 44.7% 49.5% 61.7% results No 29 38.4% 50.5% 57.2%* 71.9%* Evaluate students orally Yes 45 (65%) 38.0% 47.5% 54.4% 66.9% myself No 24 37.6% 46.4% 49.6% 64.1% Review students’ Yes 22 (32%) 37.0% 45.9% 53.7% 65.5% assignments No 47 38.2% 47.7% 52.2% 66.2% Teacher-provided progress Yes 19 (28%) 36.7% 45.8% 52.5% 66.2% reports No 50 38.3% 47.6% 52.8% 65.8% Review End-of-Year Test Yes 34 (49%) 39.0% 48.7% 52.3% 66.7% Results No 35 36.7% 45.6% 53.0% 65.2% Union Council supervisors Yes 37 (54%) 37.1% 46.3% 53.4% 66.2% evaluate students No 32 38.6% 48.1% 51.9% 65.7% Total 69 37.7% 46.8% 52.3% 65.4% *Category summary scores were significantly higher (p < .05)

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 71 TABLE 70: AVERAGE EGRA/EGMA SCORES BY HEAD TEACHERS’ OBSERVING TEACHERS’ USE OF LESSON PLANS

Grade 3 Grade 5 Observation of Avg Avg Avg Avg Teachers Lesson Number EGRA EGMA EGRA EGMA Plans Sum Sum Sum Sum Score Score Score Score Yes 40 (58%) 38.4% 47.3% 55.7%* 69.5%* No 24 37.3% 47.9% 48.9% 61.4% Total 69 37.7% 46.8% 52.3% 65.4% *Category summary scores were significantly higher (p < .05)

TABLE 71: DISTRIBUTION OF THE PERSON RESPONSIBLE FOR OBSERVING TEACHERS IN THE CLASSROOM

Head Teacher

Number % Head Teacher 31 66.0% Deputy Head Teacher 8 17.0% No one 8 17.0% Total 47 100.0%

TABLE 72: DISTRIBUTION OF SMC DECISION MAKING AUTHORITY/RESPONSIBILITY

Head Teacher SMC Responsibility Number Percent Management Problems 39 56.5% Students’ problems/solutions 37 53.6% School improvement planning 38 55.1% Review school improvement progress 29 42.0% Review Finances 28 40.6% Participate in SMC-fund spending 34 49.3% Manage infrastructure/equipment 31 44.9% Discuss school curriculum 11 15.9% Raise funds 11 15.9% Manage procurement/distribution of 19 textbooks 27.5%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 72 TABLE 73: DISTRIBUTION OF FACILITY IMPROVEMENTS

Head Teacher Type of Student Observation Number Percent Clean water 41 59.4% Electricity 41 59.4% Toilet facilities 45 65.2% Gender separate toilet facilities at 29 42.0% mixed-gender schools Library 1 1.4%

SRP BASELINE REPORT, SINDH PROVINCE, KASHMORE DISTRICT-USAID SINDH SEPTEMBER 2015 73