READ FOUNDATION - Baseline Assessment Study

USAID-funded Reading Project 1 READ FOUNDATION - Baseline Assessment Study

READ FOUNDATION Baseline Assessment Study (Azad Jammu & Kashmir and Gilgit Baltistan) July, 2018

This report was made possible by the support of the American people through the United States Agency for International Development (USAID) funded Pakistan Reading Project. The contents are the sole responsibility of International Rescue Committee (IRC) and do not necessarily reflect the views of USAID or the United States Government.

USAID-funded Pakistan Reading Project 2 READ FOUNDATION - Baseline Assessment Study

Table of Contents

Glossary ...... 5 Acknowledgement ...... 6 Background ...... 7 Executive Summary ...... 8 Major Findings ...... 9 Study Methodology ...... 12 Baseline Questions ...... 12 Method ...... 13 Participants ...... 13 Languages Spoken by Children Tested in Urdu ...... 14 Instruments ...... 15 1. EGRA ...... 15 2. Teacher Interview ...... 17 3. Teacher Classroom Observation ...... 17 4. Student Interview ...... 19 Process ...... 19 Results ...... 22 What are the levels of literacy skills among grade 1 and 2 children in READ Foundations schools at baseline? ...... 22 Summary of EGRA Scores ...... 22 Zero Scorers of EGRA subtasks: ...... 24 EGRA Subtasks Scores Compared with National Performance Standards: ...... 25 Subtasks by Region ...... 28 Comparison of Regional Results with Drafted National Reading Standards ...... 29 National Reading Standards ...... 29 What are the current demographics and perceptions of teachers regarding teaching in early grades? .. 34 Like reading Urdu book: ...... 35 What teacher does when student made mistake: ...... 35 Provision of Library at School ...... 36 Limitations ...... 36

USAID-funded Pakistan Reading Project 3 READ FOUNDATION - Baseline Assessment Study

List of Table

Table 1.READ Foundation Sample Schools by district proportion ...... 13 Table 2 Number of assessed Students and observed Teachers by grade and region ...... 14 Table 3 EGRA Tool task wise and overall Reliability ...... 16 Table 4 Teacher competencies wise Mean Score and internal reliability statistics ...... 19 Table 5 Both grades students average Percentage scores for different subtasks ...... 22 Table 6 Grade wise % zero scorers for reading subtasks ...... 24 Table 7: Zero Scorer by grades for subtasks ...... 24 Table 8.EGRA Subtasks percentage mean scores and National Performance Standards ...... 25 Table 9: EGRA Untimed Tasks Result by Region ...... 28 Table 10: EGRA timed tasks scores ...... 28 Table 11: Student percentage meeting reading standards ...... 30 Table 12:Task wise students average percentage scores for both regions ...... 31 Table 13: Teacher academic qualification by region ...... 34 Table 14: Teacher professional qualification by region ...... 34

List of Figure

Figure 1: Percentage of language spoken by Students at home ...... 14 Figure 2: Grade 1 students reading skills results by gender ...... 23 Figure 3: Grade 2 students reading skills results by gender ...... 23 Figure 4: Un-Timed Tasks Mean Score by grade ...... 26 Figure 5: Timed Task Mean Scoresby grades ...... 26 Figure 6: Readers’ categorization by grades ...... 27 Figure 8: Grade wise average mean score and percentage of students meeting reading standards for different tasks ...... 31 Figure 9: Student average mean score for passage reading with national standards and percentage of students meeting standards ...... 33 Figure 10: Student average mean score and percentage of student meeting RC standard ...... 33 Figure 11: Teacher average competency score by region ...... 34 Figure 12: Student who like and dislike reading Urdu books Figure 13: Comparison between students like & dislike reading book ...... 35 Figure 14: Teachers actions towards students' mistake ...... 35 Figure 15: Students response on School Library ...... 36

USAID-funded Pakistan Reading Project 4 READ FOUNDATION - Baseline Assessment Study

Glossary

ACRONYMS

AJ&K Azad Jammu and Kashmir BHS Boys High School BPS Boys Primary School CRO Classroom Observer DDE Deputy Director Education DOE Department of Education DPM District Program Manager EGRA Early Grade Reading Assessment GB Gilgit Baltistan HT Head Teacher PRP Pakistan Reading Project RF READ Foundation WL World Learning

USAID-funded Pakistan Reading Project 5 READ FOUNDATION - Baseline Assessment Study

Acknowledgement

We would like to acknowledge the cooperation of READ Foundation’s schools’ management for their openness and welcoming attitude for the USAID-funded Pakistan Reading Project (PRP) to work for the improvement of Urdu reading with their grade one and two students as well as teachers.

PRP also wants to thanks Urdu language teachers and grade 1 & 2 students who participated in the study along with Head Teachers who facilitated PRP enumerators and classroom observers during data collection. Special thanks to the enumerators and classroom observers for collecting assessment data.

Furthermore, we would also like to acknowledge the role of each PRP department who directly or indirectly provided support to conduct this baseline study.

USAID-funded Pakistan Reading Project 6 READ FOUNDATION - Baseline Assessment Study

Background

The USAID-funded Pakistan Reading Project (PRP) was conceived by USAID and the Government of Pakistan to address the reading deficit in Pakistani schools. PRP aims to improve children’s Urdu and Sindhi reading skills in grades 1 and 2 and through pilot interventions improve reading in Pashto. During the seven years of program implementation, approximately 1.3 million children will be targeted for reading skills improvement. Project focus areas are: Azad Jammu & Kashmir (AJ&K), Balochistan, Federal Administered Tribal Areas (FATA), Capital Territory (ICT), (KP), Gilgit Baltistan (GB) and . Applying an evidence- based and evidence-generating approach to teaching reading,

PRP will:

1. Deliver a robust in-service training model, including teacher inquiry groups, coaching and face-to-face training, that will help teachers effectively teach the five component skills of reading (phonemic awareness, phonics, vocabulary, fluency, and comprehension, with writing and print concepts integrated); 2. Develop rich, age appropriate children’s literature and supplemental learning materials to engage children in the learning process, spurring both immediate and lasting improvements in student reading ability; and 3. Build on Year 1 pre-service work in select teacher training institutes (TTIs) to strengthen the preparation of Pakistan’s teaching workforce, with a specific focus on reading, and to demonstrate to the Higher Education Commission how to take this work forward.

Improving policies and systems at the provincial and district levels to better enable teachers and students to succeed is also critical to reading achievement outcomes. PRP’s strategy to improve Pakistan’s policies and systems underpinning its national reading initiative is to align the many elements of the in-service and pre-service programs with policy discussions at the national, provincial, district, and school levels.

Based on practical experience and global evidence regarding the importance of community- based support to reinforce children’s learning and reading skills acquisition, PRP will solicit support from a variety of Pakistani NGOs through small community grants and a few larger grants focused explicitly on community-based reading activities in support of PRP’s goals.

USAID-funded Pakistan Reading Project 7 READ FOUNDATION - Baseline Assessment Study

Executive Summary

The USAID-funded Pakistan Reading Project (PRP), implemented by the International Rescue Committee (IRC) and its partners (Creative Associates International, World Learning, and Institute of Rural Management), is a seven-year project with the objective to support the provincial and regional Departments of Education (DoE) and low cost private schools throughout Pakistan to improve reading skills of children in grades one and two.

PRP is interested in exploring cost effective and sustainable options for improvement of reading in the country other than the public sector schools to expand its intervention and increase number of beneficiaries. For this purpose, PRP has initiated the process to coordinate with the management of several foundations working across the country through low fee foundation schools.

PRP successfully negotiated with READ Foundation a collaborative project to implement its reading model in their schools in AJK and GB from second quarter of FY 2016-17. As part of this initiative, a baseline study was carried out in May 2017. Trained enumerators administered EGRA assessments using tablets to a total of 483 students from grade 1 (B=116, G=118) and grade 2 (B=132, G=117) from 39 mixed gender Urdu-medium schools in four districts of AJK (Bagh, Bhimber, Muzaffarabad, Kotli, ) and two districts of GB (Gilgit, Astore). Enumerators also observed a total of 59 Urdu language teachers (22 grade I, 24 grade II and 13 grade I and II) while they were conducting their classes.

USAID-funded Pakistan Reading Project 8 READ FOUNDATION - Baseline Assessment Study

Major Findings

A summary of each subtask’s result follows:

! Orientation to Print - This subtask assessed students’ pre-knowledge before reading. It tested students’ ability to hold a book in the right direction, identify where the title of the story is located, show the direction of the text and identify starting and ending letters or words in a paragraph. Grade I students achieved 52.9 correct percentage average score whereas Grade II achieved 58.40. In grade I boys are slightly better than girls by achieving 53.1 in comparison to 52.60 percentage correct scores. Grade II has better results than grade I where boys achieved 57.2 and girls 59.80 percent correct scores. Hence boys are performing slightly lesser than girls of grade II. READ Foundation’s students have good understanding of orientation to print and there is no student with zero score in both grades.

! Phonemic Awareness – Phonemic awareness refers to the ability to explicitly identify and manipulate the sounds of language. Phonemic awareness has been found to be one of the most robust predictors of reading acquisition, as this skill can be used to put the sounds of letters together to make a simple word. In this subtask, children heard three words sets spoken aloud, and were asked which of the three words began with a different sound. Amongst administered students, 23.3% of grade I students and 18.2% of the grade II students achieved zero scores. Initial sound identification was challenging for students, because on average, grade I students could identify the initial sound of 4.3 (out of 15 sets) words, whereas grade II students could identify the initial sounds of 5.4 words.

! Letter Sound Knowledge – Letter-sound knowledge assessed children’s automaticity in their knowledge of the sounds associated with each letter. This skill is critical, as it enables children to decode, or sound out, new and unfamiliar words. In this subtask, students were given a chart with 100 letters and were asked to produce the sound associated with each letter. More than half of students - 53.7% in grade 1 - and almost

USAID-funded Pakistan Reading Project 9 READ FOUNDATION - Baseline Assessment Study

near to half of the students - 48.2% in grade 2 - did not know the sound of a single letter. Among the students who knew at least one letter sound, students knew the sounds of an average of 11.2 letters within one minute (grade 1, 10.6 CLPM and grade 2, 11.8 CLPM), in both grades, girls are more fluent than boys. In grade I fluency for girls is 11.59 CLPM whereas boys are 9.68 CLPM and in grade II fluency of girls is 14.01 CLPM and boys is 9.85 CLPM. Although girls showed more automaticity in their knowledge of letter sounds than boys, these differences were small.

! Reading Familiar Words – This subtask assessed children’s skill at reading high frequency words. Recognizing familiar words is critical for developing reading fluency. In this subtask, children were given 50 simple familiar words drawn from grade 1 & 2 textbooks of different provinces/regions to read within 1 minute. Approximately 19.1% of students in grade 1 and 5.3% in grade 2 could not read a single word. Among the students who could recognize at least one word, grade I students could read an average of 27.7 words within one minute (boys, 22.05 CWPM and girls, 33.19 CWPM), and students of grade II on average of 45.0 familiar words within one minute (boys, 40.56 CWPM and girls, 50.10 CWPM). Once again, girls could correctly read more familiar words than boys, with a good margin of differences between boys and girls.

! Reading Fluency and Comprehension – This sub-task assessed children’s fluency in reading a passage of grade-level text aloud and their ability to understand what they had read. In this subtask, children first read aloud a short story consisting of 60 words, with a simple plot. Children then answered 5 questions related to the story. Reading the story was a timed task for which the enumerator had to record the number of correct words read per minute.

o Reading Fluency – paragraph reading – Reading the grade-level passage was not very challenging for grade II students whereas it was a bit challenging for grade I students. Almost 12.6% of grade I students and 3.3% grade II students could not read a single word of the passage. Among the students who could recognize at least one word, students could read an average of almost 39.2 words within one

USAID-funded Pakistan Reading Project 10 READ FOUNDATION - Baseline Assessment Study

minute (CWPM) (grade 1, 26.6 CWPM and grade 2, 51.8 CWPM). Once again, girls could correctly read more connected words than boys in grade I (boys 21.28 CWPM, girls 31.80 CWPM) as well as in grade II (boys 43.48 CWPM, girls 61.25 CWPM).

o Reading Comprehension – Given children’s challenges in reading the passage, it is not surprising that almost 15.55 children could not answer any questions about the story. Among these children in grade 1, 21.9% whereas 9.2% grade 2 students got zero scores on the reading comprehension test. Typically, good reading comprehension for a grade II student is thought to occur when a child could answer at least 4 out of 5 comprehension questions correctly. The data shows that, on average, grade 1 student are able to respond 1.91 correct questions (out of 5) whereas grade 2 students are able to respond 2.87 correct questions. Although girls are leading as compared to boys in responding correct answers but difference is small.

o Student Demographics –

o Languages - A majority of the students who were surveyed in Urdu language at READ Foundations schools spoke Pahari (33.1%), Urdu (39.2%), Shina (15.4%), Punjabi (6.1%) and Gojri (3.4%) at home. Almost 2.9% students reported 06 additional languages as their mother tongue (such as Burushaki, English, Kashmiri, Khaur, and Hindko).

Teacher & Student Teaching and Learning Behaviors

o What teachers do when students make mistakes during an Urdu lesson – 16.2% students reported that when they make a mistake, their teacher encourages them to correct their mistake, 56.8% students responded that teacher corrected their mistake, and almost 38% students responded that their teacher punished

USAID-funded Pakistan Reading Project 11 READ FOUNDATION - Baseline Assessment Study

them. Only 0.2% students responded that their teacher talked with their mother too.

o Like reading own Urdu book – It was shared by 98.3% of students that they like reading their Urdu book and only 1.7% said that they do not like reading their Urdu book. It was observed that students who like reading their Urdu books have an average of 32.6 CWPM.

Limitations

A limitation of the present study is that without a control group, at endline we will not be able to conclude that the changes observed are due to PRP, because other factors that we are not controlling for will also be driving change, such as children’s development, differential levels of motivation, etc. Another limitation of the study is that the levels of performance documented here only reflect performance of students in READ Foundation schools, so caution should be taken when trying to generalize findings to students from other settings, such as public schools.

Study Methodology

The study uses a pre-post-test research design to compare the reading outcomes of READ Foundation schools before and after PRP intervention. A random sample of students and Urdu language teachers was selected from the grades I and II student to assess students’ baseline reading performance and teachers’ baseline instructional practices. At endline, we will track down the same students and teachers (if available) to evaluate students’ improvement in their reading performance and teachers’ improvement in their teaching behaviors.

Baseline Questions Our baseline report will aim to answer the following questions:

1. What are the baseline levels of literacy skills among children in grades 1 and 2?

USAID-funded Pakistan Reading Project 12 READ FOUNDATION - Baseline Assessment Study

2. What are the general demographics of children who are targeted by PRP? 3. What are the demographic characteristics and baseline perceptions of READ Foundation teachers regarding teaching in early grades?

Method

Participants Keeping our resources into consideration, our sample schools was selected from available READ Foundation’ schools population. Four districts were randomly selected from AJK along with both districts of GB (Gilgit and Astore), where READ Foundation has their schools. Schools were selected as per available proportion mentioned below. Ratio of rural/urban & summer/winter schools from available district school population was also considered. Enumerators assessed four randomly selected students from each grade and collected electronic data using android based tablet where as CROs observed teachers during their Urdu classroom teaching in their respective grades (I, II). CROs also interviewed those Urdu language teachers and filled the teacher’s questionnaire.

Table 1.READ Foundation Sample Schools by district proportion

District wise School proportional Selection

District Total Proportion Selected Rounded School School #

Muzaffarabad 39 0.17 5.4 6 Bagh 69 0.30 9.6 10 Kotli 60 0.26 8.3 8 Bhimber 63 0.27 8.7 8 Astore 4 0.33 2.33 2 Gilgit 8 0.67 4.67 5 Total 243 39

USAID-funded Pakistan Reading Project 13 READ FOUNDATION - Baseline Assessment Study

Table 2 shows number of students & teachers that were administered from 39 randomly selected schools from both regions.

Table 2 Number of assessed Students and observed Teachers by grade and region

Region Schools Students Teachers Grade 1 Grade 2 Grade I Grade II Both Grade B G Total B G Total M F Total M F Total M F Total AJK 32 92 97 189 110 95 205 2 17 19 2 18 20 1 9 10 GB 7 24 21 45 22 22 44 0 3 3 0 4 4 0 3 3 Total 39 116 118 234 132 117 249 2 20 22 2 22 24 1 12 13

Total 483 students (Boys=248, Girls=235) assessed from both grades and 59 Teachers (Male=5, Female=54) are observed

Languages Spoken by Children Tested in Urdu A majority of the students surveyed in the baseline spoke Pahari and Urdu language at home. In addition to the languages shown in the graph below, students reported 6 additional languages as their mother tongue (Burushaski, English, Kashmiri, Khuar, Hindako, Kohistani, and so on in small numbers). Following graph shows percentage of students with their mother language spoken at their home.

Figure 1: Percentage of language spoken by Students at home

Students Language (%) Urdu Pahari 45.0% 39.2% Shina 40.0% 33.1% Punjabi 35.0% Gojri 30.0% BURUSHASKI 25.0% English 20.0% 15.4% 15.0% Kashmiri 10.0% 6.1% KHUAR 3.4% 5.0% 0.6% 0.6% 0.4% 0.4% 0.4% 0.2% 0.2% Do not know 0.0% Hindako KOHISTANI

USAID-funded Pakistan Reading Project 14 READ FOUNDATION - Baseline Assessment Study

Instruments

PRP assessment team developed EGRA Urdu language tool along with other questionnaires with the participation of national and provincial/regional government education officials— subject specialists, language experts and assessment experts—along with PRP Technical Unit, Teacher Training and Material Development team. These tools/instruments were used for other the baseline, midline and end line studies of PRP after necessary pilot testing and revision. The details of the instrument used are as below:

1. EGRA The EGRA assessment used in the present study has the following subtasks:

a. Orientation to print – This subtask assessed students’ knowledge of orientation to print and written text. For this subtask, a total of 11 questions were given for testing students’ ability to hold a book in the right direction, identify where the title of the story is located, show the direction of the text and identify starting and ending letters in a paragraph. The final score was the number of questions children successfully answered, with a maximum possible score being 11. b. Phonemic awareness – This section of the test assessed student’s ability to differentiate the initial sounds of different letters in a simple word. Phonemic awareness has been found to be one of the most robust predictors of reading acquisition, as this skill can be used to put the sounds of letters together to make a simple word. In this subtask, children heard three words spoken aloud, and were asked which of the three words began with a different sound. Children were presented fifteen sets of words. The final score was the number of words which children successfully identified as having a different initial sound, with the maximum possible score being 15. c. Letter sound knowledge: Assessed children’s automaticity in their knowledge of the sounds associated with each letter. This was a timed subtask, in which children were shown a chart containing 100 letters arranged in 10 rows each with 10 random letters. Students were asked to

USAID-funded Pakistan Reading Project 15 READ FOUNDATION - Baseline Assessment Study

produce the sounds associated with each letter as quickly and accurately as they could within one minute, yielding a score of correct letter sounds per minute (CLPM). d. Familiar word identification: Assessed children’s skill at reading high-frequency words that they may have seen in their textbooks. Recognizing familiar words is critical for developing reading fluency. In this timed subtask, children were presented a chart of 50 familiar words. Children were asked to read as many words as they could within one minute, yielding a score of correct words per minute (CWPM). e. Oral reading fluency (ORF): The ability to read passages fluently is considered a necessary component for reading comprehension. In this subtask, children were given a 60-word story and were asked to read it aloud in one minute. Before starting, each child was instructed to pay attention to the story as they read because he or she would be asked questions about the story. The oral reading fluency score was the number of correct words read per minute (CWPM). The scores on the timed tasks were calculated by taking into consideration the correct words per minute and the time remaining until the end of 1 minute. f. Reading comprehension: After the children finished the passage, or the one minute ended, the passage was removed away from children and they were orally asked questions that required them to either recall basic facts or infer information based on the passage or the part they had read. The reading comprehension score was the number of correct answers, with a maximum possible score of 5.

Table 3 shows the means, standard deviations and Cronbach’s Alpha of the EGRA subtasks at baseline of READ Foundation schools. Overall reliability of tool is 0.99 which is excellent.

Table 3 EGRA Tool task wise and overall Reliability

EGRA Subtasks No of Items Mean SD Cronbach’s alpha Orientation to Print 11 6.1 1.64 0.32* Phonemic Awareness 15 4.9 3.51 0.61 Letter Sound Fluency 100 11.2 17.71 0.98 Familiar Word 50 36.6 25.08 0.98 Oral Reading Fluency 60 39.6 32.09 0.99 Reading Comprehension 5 2.4 1.66 0.78 Overall 241 - - 0.99 *Alpha value is not into acceptable ranges.

USAID-funded Pakistan Reading Project 16 READ FOUNDATION - Baseline Assessment Study

2. Teacher Interview This instrument used 92 questions to assess teachers’ qualifications, in service and pre-service reading training. It takes 15-20 minutes to complete a teacher questionnaire. For baseline, the instrument also gathered information about the PRP materials that teachers had received and their usage, PRP school follow-up support, and the reading activities that teachers had conducted.

3. Teacher Classroom Observation

This instrument was used to conduct on average 20-30 minute classroom observations of teachers while they are conducting a lesson, with the aim of rating their competencies using a matrix that describes how different levels of proficiency look like in the classroom on a scale from 1 to 4 (1=no evidence of the desired behavior or not yet started, 2=emerging evidence of the behavior, 3=good effort or proficient, and 4= exemplary evidence of the behavior). The measure captured the following 5 dimensions:

a) Lesson plan preparation with lesson plan objectives: 8 items that capture the degree to which the teacher demonstrates lesson preparation and lesson plan objectives (e.g.: The teacher prepares lessons with the following elements: objectives, introduction, learning activity, practice activity and evaluation; the lesson addresses the component skills of reading (print concepts, phonemic awareness, phonics, vocabulary, comprehension, fluency, writing, the lesson uses examples and concepts from the local environment, etc.) b) Teaching methods and resources: 6 items that capture the degree to which the teacher uses teaching methods/techniques and resources appropriately and effectively (e.g.: The teachers uses different types of question (both closed and open to engage pupils, teacher uses/displays a variety of resources to reinforce content (alphabet chart, environmental print, books, objects, visual aids, etc.) c) Promotion of student participation and wellbeing: 7 items that capture the degree to which the teacher ensures that all students participate and that their well-being is

USAID-funded Pakistan Reading Project 17 READ FOUNDATION - Baseline Assessment Study

supported (e.g.: teacher creates an inclusive, participatory classroom culture for (all) student (linguistically, culturally diverse, learning differences, gender, disability, etc.), the teacher provides opportunities for student-led activities and project based learning, teacher supports student wellbeing by creating a safe and secure learning environment and employing positive discipline techniques, etc.) d) Formative and summative assessments: 4 items that capture the degree to which the teacher uses formative and summative assessments to gauge student learning; give meaningful feedback to students and families; and keep assessment records (e.g.: the teacher uses techniques to check for understanding throughout the lesson (questioning, checklists use of slates, exit tickets) and adjusts the lesson according to student needs and lesson objective, the teacher gives meaningful feedback (verbal, checking copy books, written comments) from which students can learn, etc.). e) Teaching reading and writing: 7 items that capture the degree to which the teacher teaches reading and writing effectively (e.g.: the teacher uses techniques (picture support, thematic word banks, context clues) to teach vocabulary, teacher uses techniques (modeling, sight word practice, re-reading, partner reading, punctuation instruction) to teach fluency, etc.)

USAID-funded Pakistan Reading Project 18 READ FOUNDATION - Baseline Assessment Study

Table 4 shows the means, standard deviations and Cronbach’s alpha of the Classroom observation tools subtasks at baseline of READ Foundations schools. The overall reliability of the tool is excellent, with a Cronbach’s alpha of 0.92. Table 4 Teacher competencies wise Mean Score and internal reliability statistics

Teacher Competency No of Items Mean SD Cronbach’s Alpha Reading Lesson Preparation 8 1.8 0.56 0.88 with Lesson Plan Objectives Teaching Methods and 6 1.7 0.50 0.78 Resources Student Participation and 7 1.7 0.44 0.71 Well being Use of Formative and 4 1.8 0.48 0.58 Summative Assessments Teaching Reading and 7 1.4 0.37 0.71 Writing Overall 32 1.7 0.38 0.92

4. Student Interview This instrument included 36 questions about students’ home language, reading activities at home, in the community and in the classroom. The interview conducted right after completing EGRA tasks and student responded the questions which were recorded by enumerators.

Process

Given that PRP had conducted the Endline study for the Urdu sample in the month of April 2017 and the enumerators and CROs were trained on the administration of EGRA instruments, the team chose to work with some of the same enumerators as they had already been trained and gained experience administering the EGRA. Enumerators were chosen on the basis their performance and to work in baseline of READ Foundations schools in both AJ&K & GB regions.

USAID-funded Pakistan Reading Project 19 READ FOUNDATION - Baseline Assessment Study

One day training refresher was held in the month of May, 2017 to refresh their knowledge to administer the questionnaires, EGRA assessment and classroom observations in both AJ&K and

Assessment specialist is conducting Enumerators training in AJ & K while the GB Enumerators are doing pair practice during the orientation

GB regions. 8 Enumerators (4 male, 4 female) and 8 CROs (4 male, 4 female) services were hired in AJ&K while 2 Enumerators (1 male, 1 female) and 2 CROs (1 male, 1 female) services were hired in GB. The refresher training covered all aspects of the data collection including review of the basic instruction, selecting students, rapport building with the student, use of the tangerine, marking rules, data saving and uploading and other logistical aspects of implementing EGRA.

Data collection started soon after the orientation in May 2017. Android based tablets were used for student assessment as well as teacher interview using Tangerine software, but the lesson observation was conducted using paper and then transcribed to Tangerine at a later point. This was done to better allow classroom observers to flip back and forth in the instrument while the lesson was ongoing.

The assessment ‘team’ consisted of one enumerator and one CRO supervised by regional and national assessment team. One vehicle was provided to each team so that they could visit the schools. The PRP teams ensured that the READ Foundations schools management were

USAID-funded Pakistan Reading Project 20 READ FOUNDATION - Baseline Assessment Study informed prior to their visit so that on the day of assessment team will face no issue to carry out assessment activities. READ Foundations schools monitoring teams were remained available and visited schools and observed the assessment activities being carried out by PRP enumerators. READ Foundation management cooperated with the assessment team through the data collection.

It was ensured that all tablets are fully charged before going to school for data collection. In case of power shortage on any tablet, it was also ensured that an extra tablet with rendered EGRA tools may also be available to each team as alternative data collection device. These rendered tools provided necessary data validation checks for each input. Total number of grade wise assessed students and teachers was shared with assessment team manager/officer before completion of data collection. After completing reading assessment with students, all team members sat together at a centralized location where internet was available. The data was uploaded on Tangerine server. Daily tracking sheet of assessed students and observed teachers was also maintained and shared with all concern people on daily bases. During the data collection necessary technical feedback was given to enumerator after reviewing the uploaded data by national assessment team. This feedback improves the enumeration during data collection.

USAID-funded Pakistan Reading Project 21 READ FOUNDATION - Baseline Assessment Study

Results

What are the levels of literacy skills among grade 1 and 2 children in READ Foundations schools at baseline?

Summary of EGRA Scores

Table 5 shows average percent of items answered correctly by first and second grade students in all EGRA tasks. Students in grade II comparatively perform better than grade I students and good progression can be seen in all tasks, except a slightly low progression observed in Orientation to Print and Letter Sound knowledge tasks.

Table 5 Both grades students average Percentage scores for different subtasks

Sub Task Grade I Avg. % Grade II Avg. % Correct Progression Correct Grade I to Grade II (%)* Orientation to Print 52.9 58.4 5.5 Phonemic Awareness 28.5 36.0 7.5 Letter Sound Knowledge 10.5 11.7 1.2 Familiar Word Reading 47.8 72.2 24.4 Passage Reading 39.6 66.7 27.1 Reading Comprehension 36.2 57.3 21.1 *raw mean score progression

Figure 2 shows gender wise percentage correct score for grade I students in all literacy sub tasks. It is observed that girls are performing better than boys in familiar word reading as well as passage reading & reading comprehension. Boys are almost performing at par with girls in the rest of tasks.

USAID-funded Pakistan Reading Project 22 READ FOUNDATION - Baseline Assessment Study

Figure 2: Grade 1 students reading skills results by gender

Grade 1 - Students reading skills results by Gender

60 53.1 52.60 54.60 50 46.30 40.8 38.90 40 28.40 32.8 33.3 28.5 30

20 11.40

Correct Score (%) 9.6 10

0 Orientation to Phonemic Letter Sound Familiar Word Passage Reading Print Awareness Knowledge Reading Reading Comprehension

Boys Girls

Whereas in grade II, girls are performing better than boys in all tasks shown in Figure 3

Figure 3: Grade 2 students reading skills results by gender

Grade 2 - Students reading skills results by Gender

90 78.70 80 75.30 66.5 66.40 70 57.2 59.80 59.2 60 40.80 49.2 50 40 31.8 30 Correct Score (%) 20 13.90 9.8 10 0 Orientation to Phonemic Letter Sound Familiar Word Passage Reading Print Awareness Knowledge Reading Reading Comprehension

Boys Girls

USAID-funded Pakistan Reading Project 23 READ FOUNDATION - Baseline Assessment Study

Zero Scorers of EGRA subtasks: Table 6 shows the percentage of students who were not able to give a single correct answer. We observe the highest percentages of zero scores in the letter sound knowledge and phonemic awareness subtasks. As expected, we also observe less zero scores in second grade than in first grade.

Table 6 Grade wise % zero scorers for reading subtasks

Sub Task % of Grade 1 % of Grade 2 Orientation to Print 0.0 0.0 Phonemic Awareness 23.3 18.2 Letter Sound Knowledge 53.7 48.2 Familiar Word Reading 19.1 5.3 Passage Reading 12.6 3.3 Reading Comprehension 21.9 9.2

Table 7 shares grade wise percentage of students attaining zero scores in different tasks.

Table 7: Zero Scorer by grades for subtasks

Zero Scores Decrease EGRA (Subtasks) Grade 1 Grade 2 Zero Scorer (G1 to G2)

Orientation to Print 0 0 0 Phonemic Awareness 23.3 18.2 5.1 Letter Sound Fluency 53.7 48.2 5.5 Familiar Word 19.1 5.3 13.8 Oral Reading Fluency 12.6 3.3 9.3 Reading Comprehension 21.9 9.2 12.7

USAID-funded Pakistan Reading Project 24 READ FOUNDATION - Baseline Assessment Study

EGRA Subtasks Scores Compared with National Performance Standards: Table 8 shows first and second grade students’ scores in each EGRA sub task. It can be observed that students are performing at moderate level to low level in sub tasks. On average, grade 2 students got around 60 percentage correct responses for 11 orientations to print questions and 36 percent correct for phonemic awareness. Additionally, on average, students answered correctly approximately 60 percent reading comprehension questions. Students had fluency of 12 correct letters per minutes and 45 correct words per minutes for disconnected paragraph and 40 correct words per minutes for read passage.

Table 8.EGRA Subtasks percentage mean scores and National Performance Standards

Sub Task # of Description Grade I Target for Grade II Target Items Grade 1 for grade 2 Orientation to Print 11 Percentage 52.9 70 58.4 70 Correct Answers Phonemic 15 Percentage 28.5 70 36.0 70 Awareness Correct Answers Reading 05 Percentage 36.2 80 57.3 80 Comprehension Correct Answers Letter Sound 100 CLSPM* 10.5 100 11.7 100 Knowledge Familiar Word 50 CWPM* 47.8 30 72.2 50 Reading Passage Reading 60 ORF* 39.6 30 66.7 60

*CLSPM=Correct Letter Sounds per minutes, CWPM=Correct words per minutes, ORF=Oral reading fluency

USAID-funded Pakistan Reading Project 25 READ FOUNDATION - Baseline Assessment Study

Graphical representations of Un-timed (See Figure 4) and timed (See Figure 5) tasks are given below for both grades students. Students in grade 2 are performing better in all tasks, but have exhibit almost the same performance than grade 1 students in letter sound knowledge.

Figure 4: Un-Timed Tasks Mean Score by grade

Grade wise Un-Timed Task Mean Score

7.0 6.4 5.8 6.0 5.4 5.0 4.3 4.0 2.9 3.0 1.8 2.0 Correct answers answers Correct 1.0 0.0 Orientation to Print Phonemic Awareness Reading Comprehension

Grade I Grade II

Figure 5: Timed Task Mean Scoresby grades

Grade wise Timed Task Mean Score

50.0 45.0 45.0 40.0 40.0 35.0 27.7 30.0 23.8 25.0 20.0 15.0 10.6 11.8 10.0

Correct Per LS/W Minute 5.0 0.0 Letter Sound Knowledge Familiar Word Reading Oral Reading Fluency

Grade I Grade II

USAID-funded Pakistan Reading Project 26 READ FOUNDATION - Baseline Assessment Study

Figure 6 provide percentage of non-readers (zero scores in oral reading fluency), emergent readers (scores from 1 to 59 in oral reading fluency) and fluent readers (scores of 60 and above in oral reading fluency) for both grades.

Figure 6: Readers’ categorization by grades

Grade wise Readers

100.0 11.3 90.0 80.0 40.7 70.0 Non Reader 60.0 Emerging Reader 50.0 76.1 40.0 Fluent Reader 30.0 56.1

Percentage of Reader 20.0 10.0 12.6 0.0 3.3 Grade 1 Grade 2 7

Note: Non-reader cannot read single word and Emerging reader may read from 1-29 CWPM for grade I and 1-59 CWPM for grade II whereas Fluent readers reads 30+CWPM for grade I and 60+CWPM for grade II.

As the figure 6 shows, at baseline most students are emerging readers. In second grade, we observe a large number of fluent readers, and a small proportion of non-readers i.e. 3.3%.

USAID-funded Pakistan Reading Project 27 READ FOUNDATION - Baseline Assessment Study

Subtasks by Region In this section, we shared region wise EGRA subtasks scores of the assessed students.

Table 9 shows that there is not much difference in the orientation to print mean scores of students in both regions where as in phonemic awareness and reading comprehension tasks GB students have high mean raw score than AJK students.

Table 9: EGRA Untimed Tasks Result by Region

Province / EGRA AJK GB Subtasks Grade I Grade II Grade I Grade II Mean (SD) Mean (SD) Mean (SD) Mean (SD) Orientation to Print 5.9 (1.6) 6.4 (1.5) 5.5 (1.6) 6.5 (2) Phonemic Awareness 4.1 (2.9) 5 (3.6) 4.8 (4) 7.2 (3.8) Reading Comprehension 1.8 (1.4) 2.7 (1.7) 2.0 (1.8) 3.5 (1.5)

Table 10 shows that GB students have higher raw mean score than AJK students in all sub tasks for both grade students.

Table 10: EGRA timed tasks scores

Province / EGRA AJK GB Subtasks Grade I Grade II Grade I Grade II Mean (SD) Mean (SD) Mean (SD) Mean (SD) Letter Sound Knowledge 5 (9) 6 (10) 37 (20) 40 (22) Familiar Word Reading 27 (23) 44 (24) 30 (23) 51 (26) Passage Reading 26 (24) 50 (32) 31 (29) 62 (37)

USAID-funded Pakistan Reading Project 28 READ FOUNDATION - Baseline Assessment Study

Comparison of Regional Results with Drafted National Reading Standards

National Reading Standards In 2015, PRP, in collaboration with Ministry of Federal Education and Professional Training, organized a five-day national workshop on Performance Standards Setting for Urdu reading skills among early grade students that was attended by representatives from the seven provinces/regions under PRP’s scope. The overall purpose of the workshop was to develop draft student performance standards in reading at the national level, develop compliance at provincial/regional levels for early grade students in Urdu PRP also worked with the Sindh Reading Program (SRP) and supported the Sindh government in setting the performance standards for Urdu and Sindhi. The provincial compliance level standards developed during consultative workshops held in respective provinces were approved by the Department of Education. Following are the national reading standards for different reading skills: Orientation to Print: 70% items answered correctly Phonemic Awareness: 70% items answered correctly Letter sound knowledge: 100% Correct Letter Sounds Identified Per Minute Familiar word reading: 30 familiar words per minute for first graders and 50 for second graders Oral reading fluency: 30 Correct Words per Minute for first graders and 60 Correct Words per Minute for second graders. Reading Comprehension: 60% items answered correctly for first graders and 80% or above for second graders These standards reflect optimal standards that students would achieve at respective grade levels after sustained systematic and explicit reading instruction. Reading initiatives within Pakistan are in a nascent form. PRP is working with the GOP to establish and maintain these reading standards in PRP target schools nationwide, but achievement of these new and ambitious standards will be attainable as a long-range goal for students who have access to sustained explicit, systematic reading instruction by trained, seasoned reading teachers.

USAID-funded Pakistan Reading Project 29 READ FOUNDATION - Baseline Assessment Study

The following table shows the both regions combined percentage of students for both grades that meet above mentioned reading standards.

Table 11: Student percentage meeting reading standards

EGRA (Subtasks) % of Grade 1 Students % of Grade 2 Students Orientation to Print 15.0 23.0 Phonemic Awareness 3.0 11.0 Letter Sound Fluency 0.0 0.0 Familiar Word 41.0 46.0 Oral Reading Fluency 33.0 40.0 Reading Comprehension 29.0 45.0

It can be observed zero percent of students which meet letter sound fluency standard in both grades where as in the rest of reading skills there is small to moderate percentage of students who meet the reading standards. Better results can be seen for familiar and oral reading fluency in comparison to orientation to print and phonemic awareness. It is also observed that grade II students ahead than grade I students.

Figure 7: Grade wise percentage of students meeting reading standards

Grade wise students meeting reading Standards

50.0 46.0 45.0 45.0 41.0 40.0 40.0 33.0 35.0 29.0 30.0 23.0 25.0 20.0 15.0 15.0 11.0 Grade 1 10.0

Percentage of Students Students of Percentage 3.0 5.0 0.0 0.0 Grade 2 0.0

USAID-funded Pakistan Reading Project 30 READ FOUNDATION - Baseline Assessment Study

The following table shows the grade wise average percentage score of both regions for different tasks in comparison to drafted National Reading standards

Table 12:Task wise students average percentage scores for both regions

AJK GB NRS S # Tasks G 1 G2 G 1 G2 G1 G2 1 Orientation to Print 54 58 50 59 70 70 2 Phonemic Awareness 28 34 32 48 70 70 3 Letter Sound Knowledge* 5 6 37 40 100 100 4 Familiar Word Reading* 47 71 51 77 30 50 5 Oral Reading Fluency* 39 65 44 73 30 60 6 Reading Comprehension 36 55 40 70 80 80 *: Fluency, NRS: National Reading Standards

Figure 8 shows average mean of score for orientation to print, phonemic awareness, letter sound knowledge and familiar word reading along with percentage of students meeting reading standards for both regions and both grades

Figure 8: Grade wise average mean score and percentage of students meeting reading standards for different tasks

USAID-funded Pakistan Reading Project 31 READ FOUNDATION - Baseline Assessment Study

*No one student met letter sound knowledge standard

USAID-funded Pakistan Reading Project 32 READ FOUNDATION - Baseline Assessment Study

Figure 9 shows oral reading fluency comparisons of both grades in both regions.

Figure 9: Student average mean score for passage reading with national standards and percentage of students meeting standards

Figure 10 shows the percentage of correct responses against the asked questions after reading a passage in comparison to national reading comprehension standards (left) and the percentage of student meeting that standard (right).

Figure 10: Student average mean score and percentage of student meeting RC standard

USAID-funded Pakistan Reading Project 33 READ FOUNDATION - Baseline Assessment Study

What are the current demographics and perceptions of teachers regarding teaching in early grades?

Trained classroom enumerators observed Urdu teachers of grade I and grade II during their classroom teaching. Figure 11 shows teacher’s average competencies by region.

Figure 11: Teacher average competency score by region

Region wise teacher competency average mean score

2 1.8 1.6 1.9 1.8 1.8 1.4 1.7 1.6 1.7 1.2 1.4 1.5 1.5 1 1.2 0.8 0.6 0.4 AJK 0.2 0 GB Reading Teaching Student Use of Teaching Lesson Methods and Participation Formative Reading and Preparation Resources and Well and Writing with Lesson being Summative Plan Assessments Objectives

*1: Not yet started, 2: Emerging, 3: Proficient, 4: Exemplary

Table 13 and Table 14 show their academic and professional qualification statistics for both regions teachers.

Table 13: Teacher academic qualification by region

Academic Qualification AJK GB Total Matric 1 1 2 Inter (Science/Artis) 10 0 10 B.A / B.Sc 27 9 36 M.A / M.Sc 11 0 11 Table 14: Teacher professional qualification by region

Professional Qualification AJK GB Total No Professional Education 23 1 24 PTC 2 1 3 CT 3 0 3 A.E.D 0 1 1 B.Ed 19 5 24 M.Ed / M.A (Ed) 1 0 1 Other 1 2 3

USAID-funded Pakistan Reading Project 34 READ FOUNDATION - Baseline Assessment Study

Like reading Urdu book: It was observed that almost all students of both grades like reading their Urdu book, which ultimately can contribute toward good oral reading fluency.

Figure 12: Student who like and dislike reading Urdu books Figure 13: Comparison between students like & dislike reading book

Like Reading Urdu Books % ORF Mean Score in Like Reading 98.3% Urdu Books 100.0% 80.0% 40 32.6 60.0% 30 40.0% 20 3.8 Percentage Percentage 20.0% 1.7% 10 0.0% 0 Yes No Yes No Mean ORF Score Axis Title Like Reading Urdu Books

What teacher does when student made mistake: Students were asked about how teachers react when they make a mistake while reading their lesson. More than half of students responded that their teacher correct their mistake and make correction for them, 16.2% students report that their teachers encourage them to correct their mistakes, and more than 35% responded that their teachers punish them.

Figure 14: Teachers actions towards students' mistake

Talk with Mother 0.2% Tolerate 0.2% No Mistakes 0.4% Punish 36.8% Anger 1.1% Mock 2.1% Fix 56.8% Nothing 2.5% Encourage 16.2% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0%

USAID-funded Pakistan Reading Project 35 READ FOUNDATION - Baseline Assessment Study

Provision of Library at School Students from both regions and both grades were asked about existence of library at their schools. Almost 80% students replied that they do not have school library, approximately 16% students said that they have library and approximately 2% do not know.

Figure 15: Students response on School Library

Library at School

90.0 79.6 80.0 70.0 60.0 50.0 40.0 30.0 15.7

Percentage of Students Students of Percentage 20.0 10.0 1.8 2.9 0.0 No Yes Do not know Missing

Limitations The present study uses a pre-posttest design to identify the changes observed, from baseline to endline, in the reading skills of students and the instructional practices of teachers in READ Foundation schools. One limitation of the study is the lack of comparison group. Without a comparison group, at endline we will not be able to conclude that the changes observed are due to PRP. In fact, some of them will also be due to children’s development, differential levels of motivation, etc.

Another limitation is that the levels of performance documented only reflect performance of students in READ Foundation schools, so caution should be taken when trying to generalize findings to students from other settings, such as public schools. In fact, other PRP reports conducted with other populations in Pakistan show that students in READ Foundation start ahead than students in public schools.

USAID-funded Pakistan Reading Project 36