USAID-Funded Pakistan Reading Project 0

USAID-funded Pakistan Reading Project 0

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

This study/report/ publication is made possible by the support of the American people through the United States Agency for International Development (USAID) funded Pakistan Reading Project. The contents are the sole responsibility of International Rescue Committee (IRC) and do not necessarily reflect the views of USAID or the United States Government.

USAID-funded Pakistan Reading Project 1 ACKNOWLEDGEMENTS

APEX Consulting Pakistan (APEX) would like to express their gratitude to the 246 respondents from 21 schools of seven districts belonging to seven organizations who participated in this evaluation upon whose opinion and thoughts this report is based. They provided their valuable inputs; observations and experiences which enabled the consultants to draw a picture of effectiveness of CRP grant initiative which is one of the components of USAID funded Pakistan Reading Project (PRP).

PRP is supporting the provincial and regional education departments throughout Pakistan to improve the reading skills of children in grades one and two. PRP under its community based support component awarded Complementary Reading Project (CRP) grants to Pakistani Non- Governmental Organizations under two thematic areas i.e. “Reading Events to Promote Parental Engagement” and “Children with Special Needs”.

Omer Azmat, Rizwan Mehmood and Junaid Tahir from PRP directed the assignment. Syed Sardar Ali (APEX) was the overall lead for the assignment. Whereas Dr. Zafar Iqbal led the team of this evaluative study that included Mudabbir Haneef, Hassaan Khan, Varda Rasheed, Aftab Awan, Adil Pasha and Muhammad Jamil Arshad.

USAID-funded Pakistan Reading Project 2 ABBREVIATIONS

AF Azat Foundation

APEX APEX Consulting Pakistan

CRP Complementary Reading Project

CSN Children with Special Needs

CSO Civil Society Organizations

DDEO Deputy District Education Officer

DEO District Education Officer

FGD Focused Group Discussions

FIELD Foresight Institute for Education and Leadership Development

FP Focal Person

GB Gilgit Baltistan

HDF Human Development Foundation

HF Hashoo Foundation

HI Hearing Impaired

IRC International Rescue Committee

KAEED Karakoram Associates for Educational and Economic Development

KII Key Informant Interviews

M&E Monitoring and Evaluation

MRC Material Review Committee

PRP Pakistan Reading Project

RWF Root Work Foundation

SMC School Management Committee

SRPO Sindh Rural Partners Organization

TOR Terms of References

USAID United States Agency for International Development

VI Visually Impaired

USAID-funded Pakistan Reading Project 3 TABLE OF CONTENTS

EXECUTIVE SUMMARY ...... 5 1. INTRODUCTION OF THE COMPLEMENTARY READING PROJECT (CRP) GRANTS ...... 8 2. METHODOLOGY ...... 9 3. INTRODUCTION OF THE ASSIGMENT ...... 12 4. CONSOLIDATED THEMATIC ANALYSIS ...... 13 4.1. Targets Set vs. Achieved: ...... 13 4.2. Relevance ...... 13 4.3. Effectiveness ...... 17 4.4. Efficiency ...... 24 4.5. Sustainability ...... 26 4.6. Parental Engagement ...... 27 4.7. Children with Special Needs ...... 28 4.8. Student Evaluation ...... 28 4.9. Changes occurred ...... 29 4.10. Advocacy Index Scores: ...... 29 4.11. Success Stories ...... 32 4.12. Limitations ...... 33 4.13. Conclusion ...... 33 4.14. Recommendations ...... 34

LIST OF FIGURES

Figure 1: Phases of Evaluation………………………………………………………………………………………………………………..……….5 Figure 2: Advocacy Index Score of All Grantees ...... 30 Figure 3: Advocacy Cohort Score of All Grantees ...... 31

LIST OF TABLES

Table 1: Planned and Achieved Sample Size ...... 10 Table 2: Advocacy Methodology ...... 11 Table 3: Advocacy Index Score for all Grantees ...... 30

Annexure ...... 36

USAID-funded Pakistan Reading Project 4 Evaluation of Complementary Reading Project (CRP) Grants Initiatives

EXECUTIVE SUMMARY The USAID-funded Pakistan Reading Project (PRP), is being implemented by the International Rescue Committee (IRC) and its consortium partners Creative Associates International, World Learning, and Institute of Rural Management. The project is supporting the provincial and regional education departments throughout Pakistan to improve the reading skills of children in grades I and II. PRP under one its component “Community Based Support for Reading” awarded Complementary Reading Project (CRP) grants under two themes “Reading Events to Promote Parental Engagement” and “Children with Special Needs” with the purpose to reinforce children’s learning and reading skills acquisition. PRP commissioned this study to APEX Consulting Pakistan (APEX) to evaluate the effectiveness of completed CRP grants awarded to registered Pakistani non-governmental organizations. The study further assessed the satisfaction of key stakeholders and developed an advocacy index to gauge the grantees’ advocacy skill level.

By adopting qualitative research method, the study was designed around through an evaluation of individual grant initiative with special consideration to elements such as; Relevance, Effectiveness, Efficiency, and Sustainability. The thematic focus of the awarded grants covered two broad categories i.e. i) Reading Events to Promote Parental Engagement, and ii) Children with Special Needs (Visual Impairment), with the evaluation tools designed according to the focus of the implemented initiatives. Achievement of the agreed outputs and outcomes, extent of the effect of strategies/tools used, extent of the grants on changing the perception of parents on the school system, sustainability, and best practices of the initiative, level of parental engagement achieved and attention to children with special needs, and overall changes occurred in the reading enhancement through the grants were incorporated into the evaluation study.

A total of 21 schools (three from each grantee district) were randomly selected to collect data from principals, teachers, students, and parents. Besides this, interviews were conducted with District Education Officer (DEOs), and focal persons of the grantees. Focus Group Discussions (FGDs), Key Informant Interviews (KIIs), Student Evaluation, and filling Output Evaluation Checklists were the employed techniques of data collection. Some unforeseen circumstances during the data collection in the target districts are considered as the study limitations. In order to avoid any biases as to the objectivity of the report they are mentioned in the methodology section of the report.

The analysis for evaluating the effectiveness of the grants was performed according to the designed evaluation criteria. Furthermore, any changes resulted due to the grant interventions were also assessed from all stakeholders affected by the initiatives. The report demonstrates the effectiveness of these grants in the consolidated section regarding the overall effectiveness of CRP grants, with specific attention to the nature of the grants awarded by IRC in terms of achieving the desired thematic objectives, whereas, grantee basis section is included in the annexure.

Relevance The initiatives implemented by the grantees were largely relevant to the project objectives. Teachers reported positive effects by learning about differential treatment of students having varying abilities particularly those who are referred to as Children with Special Needs (CSN). Teacher training imparted through the project initiatives remained highly relevant in capacity building and particularly in learning new techniques. Most of the teachers that participated in the study were working in single room schools with multi-grade teaching system, dealing with a large number of students from different classes simultaneously. These initiatives were reported to be the first of their kind in several districts, including Kharan, Skardu and Mardan, where lack of basic facilities and low motivation among teachers were pressing issues. Prior to the CRP, it was difficult for them to identify problems among children. However, after the training under CRP, they were able to identify

USAID-funded Pakistan Reading Project 5

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

and help the children easily. It was found that the varying design of the initiatives by different grantees affected the relevance respective to CRP objective. Furthermore, the grantees were able to increase parental awareness through sensitization sessions and workshops.

Effectiveness Generally effectiveness outcomes of the project were harnessed by almost all the grantees as the various activities in the milestones set in the grant agreements were implemented to achieve the thematic objectives. Although activities designed to increase parental engagement were useful in changing parents’ perceptions regarding children’s reading & learning habits, low literacy and poverty of parents prevented the effectiveness of the grants at some extent. Grants were able to change the perception of parents on the school system at least in the CRP grants intervention schools. Ownership of the programs by the communities, especially in Mardan and Kharan, were considered to be good indicators of project effectiveness. The strategies and tools in the form of training, motivation sessions, reading corners, and provision of glasses to the students proved to be highly effective to achieve the intended results of establishing an environment in and outside of schools conducive for learning and development of children with special needs.

Sustainability Parents were less aware on the importance of the communication with their children prior to these initiatives but this communication gap was minimized through trainings and sensitization sessions. However, in most of the schools no proper mechanism existed for formal and permanent engagement of parents with schools. Most of the parents visited the schools only when their children had issues or when the school called them. In some schools in Sajawal and Tando Allahyar, books given to schools had either been lost, or had been worn out and could not be used for long. Permanent mechanism for the placement of such books was needed. The need for more books on a regular basis was also reported by schools. On the other hand some activities like teachers trained on the initial eyesight screening and the parents and communities sensitized on the importance of reading and their role in providing conducive environment for their children were sustainable.

Efficiency The programs were kept cost and time efficient through design, using low-cost to no-cost protocols while implementing the project activities. Strategies adopted and implementation mechanisms were thoroughly appreciated by all the grantees. All the participants found different trainings and sensitization sessions to be in line with the norms of the efficacy pyramid. Above all, all the project activities were lined in the budgets, work plans and grant documents. The different trainings and their curriculum was simple and according to the level of participants. Parents in general found the different sensitization sessions to be fruitful and efficiently working. Short duration of the implementation was predominant observation from almost all the participants, especially teachers, who felt that a large amount of learning was imparted in a relatively brief period.

Parental Engagement CRP grants were able to increase parental engagement with schools, teachers, and children which was one of the major thematic areas of the project. Moreover, initiatives resulted in increasing awareness in parents and their capacity building as well as enhancing their motivational levels and greater school-parent-teachers liaison. None of the earlier development initiatives had focused on parental engagement and enhancing the reading skills of the children, and outreach to the homes of children was a novel aspect of these initiatives. In Sajawal and Tando Allahyar, it was reported that the interaction between parents and the schools had weakened over the last two decades and needed to be strengthened. Prior to these initiatives, it was reported that parents did not bother coming to school even during the annual result ceremony, in all the regions covered by the grants. At the end of the CRP initiative, majority of the parents started taking interest, attending the school meetings for discussing the learning progress of the children. In Skardu, Ghizer and

USAID-funded Pakistan Reading Project 6

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Mardan, teachers reported that the frequency of meetings with parents had increased from only once or twice a year to 2-3 times per month.

Children with Special Need Providing glasses to the children in remote areas like Ghizer and Sajawal was a well received component of the project focusing on Children with Special Needs. According to a study by U.S. Department of Education, National Center for Education Statistics (2016) about 10% of the students in the normal class rooms of the schools had vision impairment.1 The grantee not only identified such students through qualified eye specialist, but also provided the remedy. The main outcome of this activity was that instead of punishing the children for their slow learning, teachers were able to identify and resolve their reading issues. The socio-economic status of the parents in both the areas was not very high. It would have been difficult for the parents in both the areas to go for the screening tests and subsequently look for the means to resolve the issues. The strengthened capacity of the teachers to screen the children was a step in the right direction as they could continue screening the children in schools and identify their problems at an early stage.

Student’s Evaluation The primary objective of the awarded grants was focused towards active parental involvement in children’s learning and establishing an environment in and outside of schools conducive for learning and development of children with special needs. The grant activities were planned accordingly to meet the thematic objectives. Reading material was also provided to the students for developing interest in reading. Under this evaluation, student’s reading abilities were also evaluated, which showed a satisfactory level of reading abilities of individual students from all schools, primarily due to weak reading skills prior to the implementation of the project.

Advocacy Index The majority of grantees were able to improve their advocacy capacity as a result of the CRP grants. While no Advocacy Index measurement of the grantees was implemented prior to the start of the grants, there is sufficient evidence to conclude that the CRP grants improved the advocacy performance because of technical and financial assistance. The technical requirements of the grants for evidence-led and planned activities ensured that the grantees devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners.

Challenges The grants in this project proved to be highly relevant to the needs of the targeted communities. While being quite effective and efficient, sustainability was highlighted in some of the places as a major challenge. At some places such as Sajawal and Tando Allahyar districts the student activities conducted during the grant period had slowed down and the level of interaction between the parents and the teachers had faded. The reason behind this may be linked to weak follow up and lack of resources available within the schools.

Recommendations As the project has been most successful in achieving its objectives, this may be replicated in other schools and districts. Being very cost effective project, the District Education Authorities should introduce project activities using their own resources. For continuation of activities in sample schools involving finances, Project authorities may request District Education Authorities to arrange finances from regular budget.

1 U.S. Department of Education, National Center for Education Statistics. (2016). Digest of Education Statistics, 2015 (NCES 2016-014), Chapter 2.

USAID-funded Pakistan Reading Project 7

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1. INTRODUCTION OF THE COMPLEMENTARY READING PROJECT (CRP) GRANTS The USAID-funded Pakistan Reading Project (PRP), is being implemented by the International Rescue Committee (IRC) and its consortium partners Creative Associates International, World Learning, and Institute of Rural Management, and is supporting the provincial and regional education departments throughout Pakistan to improve the reading skills of children in grades I and II.

PRP under one its component “Community Based Support for Reading” through a transparent and competitive process awarded Complementary Reading Project (CRP) grants under two themes “Reading Events to Promote Parental Engagement” and “Children with Special Needs” with the purpose to reinforce children’s learning and reading skills acquisition.

CRP Grants were awarded to 9 Pakistani Non-Governmental Organizations (NGOs) on the two above mentioned thematic areas in FY 2015-16 and out of the 9 awarded grants, 7 grants were completed and 02 were terminated due to unavailability of No Objection Certificate from respective government authorities.

Following are the themes, grantees and districts where the projects were implemented:

- “Reading Events to Promote Parental Engagement” Theme

Ø Hashoo Foundation (HF) – Islamabad Capital Territory

Ø Human Development Foundation (HDF) – Dist. Mardan, KPK

Ø Azat Foundation (AF) - Dist. Kharan, Balochistan

Ø Sindh Rural Partners Organization (SRPO) - Dist. Tando Allahyar, Sindh

Ø Foresight Institute for Education and Leadership Development (FIELD) - Dist. Skardu, Gilgit- Baltistan

- “Children with Special Needs (Visual Impairment)” Theme

Ø Root Work Foundation (RWF) - Dist. Sajawal, Sindh

Ø Karakoram Associates for Education and Enterprise Development (KAEED) - Dist. Ghizer, Gilgit- Baltistan

USAID-funded Pakistan Reading Project 8

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

2. METHODOLOGY For evaluation of CRP grants, a qualitative research method was employed. Under this method, FGDs and KIIs were conducted to gather qualitative data. Besides these, APEX also assessed the achievement level of the outputs of the grants through administration of a checklist with grantees, parents, and teachers and conducted a basic evaluation of the students.

Figure 1: Phases of Evaluation

Assignment Structuring

Desk Review

Focus Group Students Checklist for Key Informant Discussions Reading Output Interviews (FGDs) Assessment Validation (KIIs)

Data Management, Transcriptions and Analysis

Feedback from Feedback on Feedback from Other Research IRC, PRP MEL Stakeholders Findings & USAID

Report Development and Presentation

Evaluation instruments were developed around Development Assessment Criteria (DAC) of evaluation, relevance, effectiveness, efficiency, and sustainability, as well as around grants main goal, and targets. Instruments were further customized for each grantee under relevant CRP themes implemented. FGDs and KIIs instruments were consisting open ended direct and probing questions which provided qualitative data to gauge effectiveness of the grants. A sub-section for measuring advocacy skills of grantees was also added in KIIs tool for grantees. Following are the instruments applied under this Assignment. § FGD guidelines for parents and teachers; § KII guidelines for grantee, principals, DEO; § Student evaluation tool; and § Checklist to measure the grant output. Using purposive sampling, three beneficiary schools per grantee were selected randomly from provided list by PRP. Then each school was considered one research site to collect data from students, parents, teacher and principal. The associated persons/officials including members of SMC, DEO, and grantee were also consulted. A total of 21 schools were selected for evaluation of seven grantees.

An attempt was made to include targeted parents, teachers and students who were direct beneficiary of training and advocacy through the grant initiatives. Gender balance was kept under consideration throughout

USAID-funded Pakistan Reading Project 9

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

the sample. All the FGDs and KIIs were well planned and the participants were timely informed through respective grantees. Due to some personal reasons, parents for 5 out of 21 planned FGDs couldn’t participate in the data collection process.

Following are details on planned and achieved sample size of administered tools for data collection.

Table 1: Planned and Achieved Sample Size FGDs KIIs Students Evaluation Parents Teachers Principal Grantee EDO/DEO

Grantees Planned Planned Planned Planned Planned Planned Achieved Achieved Achieved Achieved Achieved Achieved

FIELD 3 3 1 1 3 3 1 1 1 1 30 12 KAEED 3 3 1 1 3 3 1 1 1 1 30 25 HF 3 3 1 1 3 3 1 1 1 1 30 30 HDF 3 3 1 1 3 3 1 1 1 1 30 30 SRPO 3 2 1 1 3 3 1 1 1 1 30 30 RWF 3 0 1 1 3 3 1 1 1 1 30 14 AF 3 2 1 1 3 3 1 1 1 1 30 27 Total 21 16 7 7 21 21 7 7 7 7 210 168

To conduct field data collection, Team Leader, six moderators along with note takers were deployed. The moderators held discussions with the participants, while the note takers took detailed notes as well as recorded the audio after securing the permission from the participants. After conducting FGDs, and KIIs, the moderators prepared transcriptions using notes and the audio recordings.

USAID’s Advocacy Index was implemented to measure the advocacy performance of the seven grantees during the grant cycle. This measurement captured advocacy as the active process through which these CSOs sought to influence political or social change. Beyond the measurement, a key area of interest was whether the advocacy performance of these CSOs improved as a result of the grant, both during the course of implementing the grant and afterwards. Based on USAID’s Advocacy Index, the advocacy performance of these CSOs was measured around the following eight dimensions:

1. Ensuring the issue is timely and important; 2. Devising a strategy or action plan; 3. Collecting information about the issue like through research and talking to stakeholders; 4. Formulating a position or recommendations on the issue; 5. Securing sufficient resources for the initiative; 6. Building partnerships or coalitions with other interested groups; 7. Taking actions or activities to promote your position or recommendations on the issue; and 8. Monitoring any changes resulting from your efforts to ensure that they are implemented well.

USAID-funded Pakistan Reading Project 10

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Methodology of Advocacy Index The measurement of the Advocacy Index was based on a two-step process, featuring one instrument for each step as follows: Table 2: Advocacy Methodology Sequence Steps Instrument First Qualitative, in-depth interviews with key informants Advocacy Index Interview Guide Second Scoring debrief by research team Advocacy Index Scorecard

The Advocacy Index Interview Guide and the Scorecard used for this measurement followed the standard guidelines, questions and scoring scales provided by USAID’s Handbook of Democracy And Governance Program Indicators (1998), while taking into consideration more recent adaptations of the Advocacy Index, particularly the EWMI CGP CSO Organizational and Advocacy Capacity Assessment (2013) and the Civil Society: Mutual Accountability Project (CS: MAP) (2016). These more recent adaptations were taken into account in order to incorporate ‘heuristic updates’ to the Advocacy Index that have taken place over the course of the past two decades since its introduction as a qualitative performance measurement indicator by USAID for CSOs in 1998.

A lead interviewer facilitated and documented these in-depth interviews while a second interviewer, where available, was also used for further documentation and asking additional questions. According to respondent consent, the interviewer(s) asked questions verbally while writing the responses, while following-up and probing through the same medium. The interviewers underwent trainings on the interview guidelines and the scorecard, and also acquire knowledge of the context for each interview in order to acquire familiarity with the CSOs and their activities. The interviewers, as well as the entire research team, also conducted onsite and offsite review of project and programme data and documents, in order to validate the interview findings through triangulation with other sources of evidence. Immediately after each, the interviewer(s) scored the CSOs using the Advocacy Index Scorecard. They entered scores into Excel files for the next step. All interviews were conducted between November and December 2017 with each of the CSOs in their relevant offices. All interviews were reviewed for completeness and missing information and one respondent was followed up for clarifications beyond the original process of follow-up and probing.

During the first step, the research method involved in-depth interviews with key informants – usually one to two representatives of CSOs. Key informants were selected and interviewed, given their special knowledge and experience of the CRP grant. A total of 11 key informants were interviewed. The interviews lasted two hours on average, and were initiated with an explanation of the purpose of the research and requesting respondents’ oral consent to participate.

For the second step, a debrief session was conducted by the entire research team with the lead and second interviewers shortly after each in-depth interview. During this session, the interviewers as well as the remaining research team reviewed and refined the initial scores by mapping all the interview findings across the series of right advocacy dimensions (and their components) of the Advocacy Index Scorecard. It was ensured that the cumulative requirements of the scale were met i.e. to allocate the CSO one score on the scale would necessitate ensuring that the CSO also fulfills the requirements of the preceding score on the scale. In order to control for validity and reliability, the research team reviewed all scores with their supporting evidence, requesting clarifications from the interviewers as needed.

USAID-funded Pakistan Reading Project 11

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Using transcribed data, the analysis was conducted around two CRP themes and DAC criteria of evaluation. Similarly, student’s evaluation was also analyzed by computing aggregate score of correct words read per minute by the evaluated students. This enabled us to set range and boundaries at grant level. As no concrete baseline data was available, APEX collected and analyzed data on certain indicators to conduct a situational analysis (on recall basis) of communities’ needs, attitudes and behaviors before and after implementation of Grantee’s interventions.

As mentioned in the TORs, separate results are produced and reported for each grantee. Besides this, a consolidated chapter covering overall results of CRP is presented in this report.

Methodology of Student Evaluation The tools used for student evaluation included Familiar word reading as well as passage reading. A meeting was held among data collectors in APEX office where they held an orientation session to ensure uniform methodology in all regions. Passage and word reading task consisted of 60 and 50 words. Students were first required to read aloud the familiar words and the time was noted till the end and then the passage reading was done by the student and the time of this task was also noted till the end. Stop watches were used to note down the time. If the student was finding it hard to read, data collector told the child to skip on to the next word. However, a note taker was separately counting right words and was noting them down. The average mean score was taken out of the aggregate of both scores for each student. This evaluation was not based on early grade reading assessment (EGRA) methodology. The purpose of the evaluation was not to assess students by using the EGRA time based method of calculating the fluency.

Methodology of Obtaining Success Stories The source of obtaining success stories was through FGDs with teachers and parents. During the FGDs, parents and teacher were asked about any success stories they specifically like to share regarding positive changes in children after the interventions. Collective comments by teachers and parents were penned down as success stories with a proper background of the child in regard to his/her previous performance and the performance after the intervention.

3. INTRODUCTION OF THE ASSIGMENT APEX Consulting Pakistan was engaged by PRP to evaluate the effectiveness of the grants provided to the above mentioned organizations.

The evaluation assessed the satisfaction of key stakeholders, i.e. children, parents, teachers, and school administrators on the effectiveness of the grants in improving the reading skills and habits in children. The evaluation further collated and analyzed lessons learnt, challenges faced and best practices obtained during implementation in order to apprise IRC while designing the next phase of CRP grants. The evaluation also assessed the change in the capacity of grantees in effective grant management and advocacy skills on broader educational issues.

The evaluation was done by obtaining in-depth qualitative data on various aspects of the project through pre- designed tools for relevant populations. These aspects included Relevance, Effectiveness, Efficiency, and Sustainability. These aspects cover the need for the project intervention in the area, its alignment with overall project outcomes and objectives, the extent of their impacts on the communities, the level of achievement of desired results, short and long-term effects, as well as covering issues of community ownership of such projects, lessons learnt and best practices.

USAID-funded Pakistan Reading Project 12

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

4. CONSOLIDATED THEMATIC ANALYSIS

4.1 Targets Set vs. Achieved The results indicated that the majority of the targets set were met with an excellent level of achievement. All targets set by FIELD, HDF and HF mainly focused on enhancing children’s reading habits and improving parental engagement were met with outstanding level of achievement. As for KAEED, all target activities met satisfactory levels of achievement in identifying and facilitating the visually impaired children along with the capacity building of teachers, SMC members and parents. RWF team on the other hand, was expected to organize different types of activities to achieve the targets of identifying and supporting visually impaired children, identification and training of teachers for visual screening of impaired children and provisioning of vision screening kits to schools. They were able to achieve all the targets and completed all the activities required for achievement of the targets. According to the analysis of the results obtained through the target checklist, SRPO and AF team were also able to achieve all the targets and the activities were completed on time that involved parents, teachers and communities.

4.2 Relevance The grants provided by PRP as part of its program were all used by various grantees to implement specific, pre- designed projects all over Pakistan. These grants were implemented in 1 district of Balochistan, 2 districts of Sindh, 1 each in KPK and Islamabad, and 2 districts of GB. Overall the initiatives implemented by the grantees were largely relevant to the project objectives and achieved maximum agreed outputs and outcomes.

Low engagement of parents with schools, lack of awareness of teachers in improving reading skills of children and lack of interest of children in reading itself were commonly found in the areas that were evaluated for this study.

The underlying reasons for low involvement in the reading skills and general education of children in primary school were found to be poverty, illiteracy and lack of awareness among parents, lack of available facilities in schools and inadequate capacity among the teaching staff to be able to address the needs of the children.

Despite slight variations in the design of the projects implemented by the grantees, they all managed to address these underlying issues in one way or the other. Awareness was raised amongst parents on the importance of developing reading skills and the ‘provision of a favorable environment for children to be able to read and study at home’. Sensitization sessions with parents stressed on their role & responsibilities in their child’s education and their engagement with the school management.

Reading facilities in target schools were provided in most cases, improving their general condition and helping to improve the learning environment through the use of colorful charts and displays at minimal cost to the community, with the involvement of the students themselves. These charts, along with the developed reading materials, small libraries and multimedia aids, addressed the shortcoming in material infrastructure that were preventing the development of a reading culture in the target areas.

Capacity building of teachers, head teachers and community members was done through several awareness sessions and workshops. The training provided to teachers enabled the identification of special needs, such as visual impairment. It also trained the teachers in using modern methods of teaching reading to primary school children, such as the use of phonetics. Furthermore, all teachers included in the study displayed a change in

USAID-funded Pakistan Reading Project 13

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

attitude towards teaching, using increased involvement with the children and parents instead of punishing deviating children, as was traditionally being done. Furthermore, all the study schools reported that despite the immense need for such projects, the initiatives taken by the respective grantees were the first of their kind. Most other education related interventions focus on infrastructure development or teachers’ training, but choose to ignore parental engagement with schools and their children. By directly going to the households of the children and engaging their parents, majority of the grantees were able to fulfill a critical gap in the effectiveness of such interventions.

Azat Foundation was working in Kharan, which is one of the most underdeveloped districts of Balochistan. Kharan displayed very poor educational indicators, including the class specific reading skills. Primary schools don’t have any facilities like libraries and reading rooms. Keeping in view the existing conditions, most of the stakeholders agreed that the project was quite relevant to the needs of the people and addressed the felt needs.

It was the first project of its kind in Kharan district. Due to a high degree of sensitivity and security concerns, not many NGOs and INGOs had implemented any projects in the area. It was found that parents, being mostly uneducated, were not able to support children in addressing their educational needs. It was found that interaction between the school and the community was also limited, which was related to the motivation and commitment of the teachers.

The project in Kharan was found to have contributed in creating an enabling environment in the schools for the children, which encouraged them to engage in reading skills enhancement activities. “Earlier the environment in the schools and in the homes was not conducive for encouraging children to improve their reading habit & skills. The schools had no libraries and reading material. The parents were mostly uneducated and not able to assist and support their children in their studies.” The interaction between the schools and the community was limited and not very proactive. The project helped in improving coordination and engagement with the parents.

Many schools in the area were found to lack even basic facilities and the lack of support and cooperation from the (education) department in providing those facilities. Many schools lacked basic facilities like washrooms and drinking water. In the absence of even basic amenities, the provision of adequate reading facilities such as books, libraries, etc. would not have been possible without outside assistance.

HDF implemented the grant in Mardan, where it was reported that most of the communities in Mardan were illiterate and had no awareness about libraries, story books or reading corners. Children had a tough time in reading and communicating their issues to teachers & parents due to lack of interest. A non-conducive environment had led to lack of confidence in children. The DEO said that the teachers were not skilled enough to motivate children and parents, as most of the teachers preferred the same old traditional way of teaching.

Principals in the selected schools of Mardan said that prior to this initiative, “presence of children was not regular, teachers were not motivated enough and parents had no involvement in their children’s school matters.” As a result of the project activities, including awareness sessions and workshops, confidence had been raised among parents and they had started taking a keen interest in the school activities and in improving children’s reading habit & skills. It was also reported that the provision of libraries had significantly impacted the community, as before CRP there was no concept of reading story books and watching animated learning videos in school.

USAID-funded Pakistan Reading Project 14

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

All teachers agreed that before this intervention, children had difficulty in Urdu reading and speaking, they did not know pronunciation; and a number of students did not attended the school regularly. They also said that libraries were not functional before this project. As for parents, all teachers said that they never used to visit school as they had lack of awareness, relying mostly on the old ways of teaching (rote learning, scolding and punishment) before the training sessions which introduced them to the concept of engagement and involvement in the child’s learning processes.

FIELD implemented the project in Skardu District. Even though agencies such as AKDN, UNICEF and MAARIF had worked in the region, their focus had largely been restricted to teachers’ training and infrastructure development. It was felt that most education related interventions focus on improving English language skills in students, leaving a gap in Urdu language proficiency. The CRP project was solely designed for Urdu reading, which was thought to be an area that greatly needed improvement. The current curriculum being implemented in Govt. schools does not fulfill all the needs of the community. A care towards standards of quality and regional requirements of the area was taken, suggesting a high relevance of the project intervention in the district.

The project focal person in Skardu felt that previously, no intervention had gone directly to the households for increasing parental engagement. With a high prevalence of poverty and illiteracy among parents of students studying in the project schools, this aspect was thought to be addressing some very important needs that had a direct impact on increasing parental engagement with their children in educational matters.

The teachers participating in the Focus Group Discussions were of the opinion that Urdu reading had been a major weakness in primary education before FIELD’s intervention. Children were having difficulty in joining letters to make words.

As for the availability of reading & learning material in schools, only one high school in Skardu had a library. The rest of the schools did not possess any reading or learning materials which could aid in developing children’s interest in Urdu reading. The provision of small libraries and vocabulary books to the schools as part of the CRP implementation was thought to be very relevant in addressing this need. Most of the schools in the area have been getting only the most basic facilities, with some even lacking electricity and chairs for all the students.

It was further noted that 80% of the children’s parents belonged to areas outside of the city, with the students working in other people’s homes as domestic servants. Parents were largely of the opinion that there was “some difficulty in reading for children before the program and not much awareness among parents regarding their children’s problems in improving reading.” Home activities were mostly restricted to children doing homework without any parental supervision. The role of supervising the children’s education was relegated mostly to mothers, the majority of whom could not read to them. They also felt that they were facing problems which became apparent only after FIELD’s intervention through the awareness sessions.

HF worked on implementing the CRP in Islamabad District. Children in the area were said to have lack of confidence prior to this intervention. They said to have a tough time in reading and sharing their issues to teachers & parents. Teachers were not skilled enough to motivate children and parents to increase their level of engagement with the school and spend more time in reading activities.

USAID-funded Pakistan Reading Project 15

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The provision of books was made relevant to the target age groups by making them colorful and attractive for all children to take more interest. Parent’s involvement in these reading activities boosted the children’s confidence, having an effect on making the home environment more conducive for promoting reading. Prior to this initiative, it was reported that children’s attendance was unsatisfactory, teachers were not motivated enough and parents had no involvement in their children’s school matters. Children had difficulty in Urdu reading and speaking, “Torr Jorr” i.e. constructing/deconstructing words, they did not know pronunciation; and could not identify Urdu letters. Teachers also said that due to high strength of children in schools, it was not easy to pay attention to each child. Children were also very shy, so they were hesitant to share their issues. They also said that they did not have libraries in school before this project and all the focus in schools was on completing syllabus, so no book reading activities were implemented.

Parents appreciated the change in teachers attitudes in cooperating with regard to any issue their children were facing. All of the parents agreed that the provision of libraries had led the children to take keen interest in reading in general, as well as studying and attending school regularly.

KAEED was working in Ghizer, where due to the remoteness of the area, illiteracy and poverty are prevalent. In terms of effectiveness of the project, school principals were of the view that most of the project goals were met with a positive impact in the schools and the community. Learning about modern pedagogical techniques, including differential treatment of differently abled students, had changed the environment in the schools and its effect could be seen even among parents. Teachers were positively affected by learning about differential treatment towards children having fewer abilities than others, or suffering from specific problems. They felt that previously, their skill was not sufficient to be able to understand and address the various problems faced by students.

A Teacher from Kuchdeh said: “This project was the first time they were taught how to use phonetics to teach Urdu, as well as the identification of visually impaired children, which was something they never knew existed before.”

As a result of the project, almost 70% of the targeted schools have prepared detailed lists of differently abled children, the nature of their problems and the special attention and treatment they require during lessons, which are now displayed in the classrooms for the aid of current and new teachers.

KAEED’s focal person for the project was of the view that prior to the program, parental involvement in schools was almost non-existent. No interaction with parents was visible, who sent the children to school because of low fees. He added that BEC Schools, which did not have any fees, showed even lesser involvement, and only 2-3% of such schools had SMCs in which parents were active, while Government schools displayed less than 20% active SMCs.

RWF was working in District Sajawal, where a number of factors were preventing the increase in parental involvement in enhancing the children’s reading skills. These factors included the socio-economic status of the parents, low levels of information, lack of awareness about the issues of visual disability among parents, lack of capacity among teachers and low level of nutrition among the children.

Furthermore, parents had little or no knowledge about the issues of visual disability and were not able to afford checkups and buying visual impairment correction glasses for their children. Even teachers said that previously they were not aware of such issues among their students. Since most of the teachers were working in single room schools with multi-grade teaching system, they had to deal with a large number of students

USAID-funded Pakistan Reading Project 16

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

from different classes simultaneously. This made it difficult for them to identify such issues among children. However, after the training, they were able to identify and help the children easily. When they received the training, they were able to identify many children with visual disabilities in their classes and refer them for further testing and glasses. The coordination between the schools and parents was also quite weak earlier and there was little or no communication between the two.

None of the projects previously implemented had focused on supporting visually impaired children to improve their reading skills. Thus, the project was not just unique; it was according to the needs to children. Prevalent poverty in the area and the lack of education among the parents prevented them from being able to identify the issues of their children and this project helped them. Thus the project addressed a need, which was previously overlooked.

SRPO was implementing the CRP initiative in Tando Allahyar, which suffered from extremely high levels of poverty, coupled with low literacy rates. In the studied schools, most of the teachers complained the parents did not take much interest in the education of the children. Once the child was admitted in the school, the parents felt that it was no more their responsibility to support and help the child. The environment in the schools was also not vey conducive for learning and increasing their reading skills. Schools had no libraries and no specific reading periods. Teachers also agreed that previously they were not using or aware of any special strategies to help the children improve the reading skills.

None of the earlier development initiatives in the region had focused on increasing parental engagement and enhancing the reading skills of the children. The earlier projects had focused on improving disaster preparedness among the children and the teachers and building infrastructure. Lack of capacity among teachers was also cited as one of the reasons behind weak reading skills of the children.

The interaction between the schools and the parents had weakened over the last decade and it needed to be re-strengthened. Moreover, the environment in the schools was not conducive for improving the reading skills of the children. The teachers lacked capacity and the help provided to the students was not also up to the mark.

We can see from the above that the basic objective of the project was enhancement of reading habits of students of grade I & II. The interventions like provisioning of reading glasses and books in a colourful material to students and holding of reading activities at school met its objective in the target areas. The students not only developed their habit of reading at school but at home as well, as witnessed by parents. And this is also true that inputs such as distribution of reading glasses, story books, library enrichment, participation in classrooms and reading corner development resulted in desired outputs and outcomes in the form of reading activities at school changing reading habits not only at school but at homes as well.

4.3 Effectiveness In order to meet the planned outcomes of the project, the grantees implemented various activities outlined in the milestones set in the Grant agreements. These activities were designed to primarily improve the reading skills of children by increasing the level of parental engagement in the children’s reading activities and the provision of specially designed reading material for primary school students. Moreover, 2 out of the 7 grantees also chose to work on addressing reading abilities among children with special needs, such as visual impairment.

USAID-funded Pakistan Reading Project 17

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

These activities were centered on setting-up of small libraries in the school and reading corners at home. Furthermore, awareness sessions with the parents and trainings for schools’ teaching staff was designed to instill a realization of their role in the child’s learning and importance of providing a conducive environment for enhancing their reading abilities. The most crucial barriers towards the effectiveness of these grants were poverty and illiteracy, which were circumvented through the design of the materials and ensuring that no financial burden is placed on the communities as a result of this project.

AF, implementing the CRP grant in Kharan was especially effective in carrying out orientation sessions, which were received quite well by the parents and the teachers. These sessions helped the parents and teachers understand the objectives of the program and the activities to be carried out throughout the program. All the sessions were completed in time and attended by the desired number of participants.

The material review committee was constituted in collaboration with education department. The representative of the education in material review committee took keen interest in the development and review of the material. Effectiveness can also be measured by the degree of ownership the community takes in the project, in this regard.

The focal person of AF said that: “Department of Education took such a keen interest in the review and development of the material that we felt that it was their project and we were just supporting him”

The material was appreciated and liked by the children and their parents. 20 schools were provided steel bookshelves and reading materials for organizing different reading activities in the school. All the schools included in the survey had books shelves in them. However in some places the reading material was not available. It was said that it was almost a year since the end of the project, therefore most of the books had either been taken away by the children, who had not returned them and some of the books were worn out due to excessive use.

All the competitions and community based activities proved quite effective in increasing the interest of the parents and community in the project and encouraged the children and schools to take active part in them. The schools took keen interest in the reading room decoration competitions and even after a year, the rooms were still decorated. The competitions also helped in developing a story culture among the students. The initiative of decorating the reading corners in homes was well received in the community, as it was designed in such a manner that it did not put any burden of the financial resources of the parents. It was reported that in most of the houses the corners were still decorated and could still be visited.

It was felt that the project had been successful in increasing interest of the children in enhancing their reading skills and had increased parental engagement. However it was also pointed out that project could have proven even more successful if the teachers had also been given detailed training instead of just orientation about the project objectives and activities. This would result in increased capacity to help children in improving their skills.

HDF was working in Mardan, where the DEO was of the view that the CRP initiative had definitely contributed to the capacity building of participants. Young staff and teachers had become more competent and showed keen interest in learning new techniques. This programme developed students’ reading skills and creativity.

USAID-funded Pakistan Reading Project 18

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Community requirements were put into consideration while material development, which made it more appropriate and effective.

According to the focal person from HDF, awareness was developed in parents and teachers. Motivation level of teachers was boosted. Children improved their Urdu reading skills and developed story reading habits even at home. Parent-teacher communication was enhanced and regular meetings started taking place. One rupee library was appreciated by all participants as per the focal person’s opinion.

Principals observed that the CRP initiative was effective in developing a keen interest in children with regard to their reading habits, Urdu speaking skills, word construction had mainly improved their attendance at school. Teachers agreed that they gained knowledge from the training sessions, learning new techniques such as flash cards; silent reading, methods of teaching word structuring and promoting group activities enhanced their communication skills. Due to these communication skills, children started sharing their problems more comfortably. Moreover, teachers were of the opinion that parents also developed interest in their children’s matters and started inquiring about their children’s performance.

Parents said that with the training sessions, they learnt techniques which helped them teaching their children at home. Parents also said that with the sessions, they also learnt the importance of Urdu reading, as before the project most emphasis was laid on English. They realized that before this intervention, they had a huge communication gap with their children. But after the sessions parents learnt how to deal with this issue, moreover friendly relation was developed with teachers as well.

FIELD implemented the project in Skardu, where the effectiveness of the CRP intervention was determined to a great degree by its relevance- i.e. the need for such a program to raise awareness among parents in engaging with their children to improve reading skills.

The Deputy Director of Education in Skardu said “initiatives undertaken by NGOs and other development organizations generally generate more interest in the public, being considered more attractive compared to government sponsored projects, which are thought to have low ownership and lacks monitoring protocols”. He added, “NGOs also use alternative channels, such as community notables and ‘ulema’ to increase their traction in the communities, which contributes to the effectiveness of these initiatives”.

Indicators reflecting the effectiveness of the CRP initiative were thought to be: increased enrollment in CRP schools, greater fluency in reading and an increase in parent’s trust in schools abilities to increase reading skills in children, reflected in the adoption of learning techniques advocated by the Complementary Reading Project. The focal person of FIELD in Skardu was of the view that positive effects were visible in school children immediately after the introduction of new techniques. A notable difference was felt in the parent’s attitudes towards their children’s reading skills.

In terms of home-based activities, the introduction of story reading hours with the parents, household items labeling and the display of charts at home were thought to be the most effective in developing an environment conducive to increasing parental engagement with children. At school levels, the establishment of small libraries was thought to be the most effective method, followed by indigenous story compilation, which generated an interest in the local culture and developed interest among the children in learning through reading. Furthermore, flash cards were seen to be an effective tool in getting the children to physically engage in the classroom environment. It was evident in the discussion with teachers, that materials distribution in the form of stationary packets was helpful in generating parent’s interest towards the project.

USAID-funded Pakistan Reading Project 19

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The focal person for FIELD told the research team. “When we gave them the stationary packets to take home, it convinced the children’s parents that this was not a meaningless project and it was earnestly trying to help the children in improving their abilities.”

After the implementation of the project, teachers feel that more parents are willing to come to school to attend meeting which are held twice or thrice every month, as well as coming on their own if they feel that their child is facing some difficulties to discuss it with the teachers. The provision of magazines, cultural story books and the vocabulary books provided under the CRP initiative helped create an interest among both students and teachers, by making them aware of their own culture and the outside world. Identification of students facing difficulty in reading has been made easier after the project, and such students are now said to be given more attention.

The parents of children who were part of the CRP program felt that earlier, their lack of education proved to be a barrier to their engagement with their children. After receiving the sensitization sessions and learning new techniques, they felt more motivated to participate in the child’s learning process and usually spent one to three hours with the child as he/she studied at home. Whereas both children and parents viewed education and learning as a passive process, it was now generally felt that they were proactively taking a part in improving the children’s reading habit & skills. Despite being illiterate, the parents sought the help of teachers to discuss the child problems and accepted their responsibility in their child’s education. Profound effects in the home environment were expressed by parents, who had become more engaged in their child’s reading after they sat with them as he/she read story books, and this effect was being passed on to other children in the household who displayed more enthusiasm about reading and going to school.

Parents expressed happiness on the fact that their child asked them to bring them more books after they had finished the ones they had been reading, and this was increasing their own enthusiasm at the child’s improvement in reading Urdu, which had earlier been either very weak or non-existent.

HF, worked in underprivileged areas of Islamabad where the Govt. representative was of the view that any programme that had been implemented in the past was not long term as children would have to return the books after a specific period of time. She also was of the view that colorful books gained more children’s attention than curricular books did. The programme had lasting impression especially on parents and teachers, as they showed keen interest in learning new techniques. She also spoke about lack of confidence in children and how first they used to feel shy but through these activities they started sharing their views and ideas openly. Practical demonstration of the taught techniques in the awareness session was highly appreciated by parents because they were illiterate so the activities were really helpful for them. Sensitization session further increased the efficacy by demonstrating the use of anything that was given to children and their parents.

Awareness was developed in parents and teachers after the intervention. She said that the design of project was pre-planned, all developed material reviewed by MRC, orientation session with parents and teachers helped to identify gaps in the project design.

Focal Person of HF said that: “Motivation level of participants was average before the initiation of project, but later, involvement of parents and teachers increased, and they all were happy and welcoming”-.

USAID-funded Pakistan Reading Project 20

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Activities like flash cards and reading corners made a lasting impression on parents, children and teachers. Parents motivated their children to read more enthusiastically. Communities had a positive response to parent-teacher meetings and stories development activities. After this project, children had libraries, separate reading hours at home, parent-teacher communication increased, story-reading periods were introduced in school and donate a book campaigns were introduced at community level. Teachers adopted innovative techniques in promoting children reading habits to some extent.

As per the principals, CRP was an excellent initiative. Motive for this project was to promote Urdu reading skills especially for non-Urdu speaking children. Before the intervention, the parents undermined and underestimated their importance in the education of children. However, after the project’s implementation, this attitude improved quite a bit. There was a satisfactory change in the participant’s attitude.

It developed students’ reading skills and creativity. It developed keen interest in children with regard to their Urdu speaking skills, word construction and improved their attendance at school. They also thought that all steps undertaken in the programme have been very effective in the community. Overall, a lot of improvement in children’s attitude and interest level was observed. Besides, the skill set of teachers had been polished as well. Parent-teacher communication was enhanced and regular meetings started taking place.

Teachers agreed that they learnt from the training sessions, like skills of engaging with children in class in an interesting way, promoting group activities, increasing interest of student in reading, parental engagement and most of all enhanced their communication skills. After training, gap between student and teacher reduced leading to students sharing their problem. Regular library periods started taking place, quotes were shared daily in morning assembly; reading activities would take place and mainly teacher-parent gap reduced.

After the programme training, parents learnt the importance of reading and communicating with their children. They also said that their children had learnt to be capable of communicating with their parents more confidently. They all had made reading corners at home and had labeled them too, which made the children gain interest towards sitting in that corner and studying/reading. Most of the parents said that their children liked to read stories having morals and lessons in the end.

KAEED, implemented the project in Ghizer district where learning about modern pedagogical techniques, including differential treatment of differently abled students, had changed the environment in the schools and its effect could be seen even among parents.

According to the DEO representative who took part in this study, the health-related and psychological aspects of education had never been touched upon by any interventions previously.

Interaction between parents and the schools had increased considerably since the implementation of KAEED’s project in Ghizer.

USAID-funded Pakistan Reading Project 21

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

KAEED managed to provide visual aids to 100% of the children identified with visual impairment during the screening test. The introduction of vocabulary building was a highly effective tool to enhance reading among children according to teachers. It enabled the children to greatly increase their vocabulary in a challenging and competitive manner, allowing them to move from various learning grades in highly systematic manner which could be monitored and followed up by the teachers and parents.

Increasing parent’s awareness of their role in children’s learning was a crucial aspect of the project. This was achieved through reactivating the dormant SMCs and forming new ones in regions where they were non- existent. The role of the SMC was emphasized in awareness sessions with parents and trainings for the SMC members, who were imparted an understanding of the SMC as a legally sanctioned body with specific roles, responsibilities and jurisdictions.

Teachers were found to be taking a more proactive role in the children’s environment, school attendance and interest in the classrooms.

The teachers told the research team during the discussion: “If certain children continue to be absent for more than two days, we take it upon ourselves to visit the child’s home to ask if the child is well or if the family is facing any difficulties. We also set up funds to provide copies and stationery to children whose parents could not afford it”,

Parent’s awareness was raised considerably through KAEEDs efforts in the project areas. Although most parents were uneducated and living in conditions of poverty, the project managed to have a positive impact on creating awareness among parents regarding their role in the child’s education, learning and reading enhancement.

Most parents said that, “Although we cannot directly participate in the reading activities of our child, we follow up on his/her progress with the teachers and sit with the children as they read stories at home. Most houses hardly had more than one room, so all the family was together in the evenings around the fireplace, where the story reading activity had become an important and anticipated activity.” , resulting in increasing the children’s enthusiasm for reading and wanting to enhance their skills.

A significant impact was felt in the parent’s attitudes towards primary education - they stated, “While we had earlier relied on punishment and strictness as a means to establish authority and discipline in the household, now we stress on the use of gentleness and love towards the younger generations.”

Many parents had designated educated family members such as elder children, cousins or relatives to assist the children in reading activities. While the decoration of model rooms and reading corners in the household could not be implemented due to the economic situation of the region, it was felt that the provision of visual

USAID-funded Pakistan Reading Project 22

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

aids contributed positively towards the families’ living conditions, as these issues might never have been diagnosed otherwise.

Children’s performance in the classroom and examinations had improved significantly. Teachers said that children had now become more open with them, replacing the earlier norm of fear and intimidation. Children were also reported to be spending more time in reading activities, with the dedicated reading period being the most looked forward time of the school day. Many children continued to stay in school even after the classes had ended, reading the books from the library and coming to borrow more books even during the winter and summer holidays.

RWF implemented the CRP initiative in Sajawal, where 10830 children from 200 schools were screened and tested and 314 children were provided glasses. Thus the project managed to achieve its stated objectives and targets. The project further built the capacity of 100 teachers. Their skills were increased to screen the children and identify their issues at an early stage and they would continue to do so in their schools. In the beginning there were apprehensions that teachers would not be able to learn the skill of screening and testing in such a short time, but the training proved quite effective and useful. The screening of more than 10,000 children was also significant achievement.

Focal Person from Education Department: “We thought that only three days’ training would not sufficiently equip the teachers to carry out the screening as it is a specialized job. However, when we looked the contents of the manual and met the facilitators, we were convinced that it could be done.”

According to the teachers, the capacity building exercises had helped them in understanding the issues of the students in a better way. The training was effective and it had given them good understanding of the structure of the eye, different eye sight problems of the children and of the methodology of screening and testing the children. They were provided the kits for screening the children, which they were using even then. The understanding of the parents about the issues had also improved.

The sessions with the parents were also considered quite useful. The sessions had helped in increasing interaction between schools and the parents and increased awareness about the issues of the children. The media messages for raising awareness had a mixed effect as some of the participants had not heard the messages on the media.

A Student from Main Primary Sindhi School Chandio: “My main problem was reading form the board. When the teacher wrote something on the board, I was not able to read it, but now I can easily read what the teacher is writing on the board with the help of these glasses.”

SRPO, working in Tando Allahyar coordinated with the education department to achieve its goals, the education department representative was part of the material review committee, which reviewed and

USAID-funded Pakistan Reading Project 23

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

finalized the manuals for the training of the teachers and SMC members. The education department also facilitated in nomination and selection of the teachers; and SMC for the trainings. As a result 100 teachers and 200 SMC members were trained. The material review committee also reviewed and finalized the information charts which were printed and distributed in the schools. Sensitization sessions were also carried out with the officials of the education department to ensure their support and coordination. The SMC members, some of whom were also the parents of the children, took keen interest in the affairs of the school and helped in strengthening of the infrastructure by completing the requirements of the drawing funds and the school principals were appreciative of the role played by the SMC members. The training helped SMC members understand their role better.

The teachers also reported improvement in their capacity to help the students in enhancing their reading skills. The teachers had limited capacity to support children in improving their reading skills. Through the training they learnt new strategies of improving reading skills. Most of the teachers reported that the strategies introduced during the training were useful and innovative. There was only one disagreement about the quality and effectiveness of the training methodology. All the schools and students received reading kits which were designed keeping in view the social and cultural sensitivities. The school rooms were decorated with the different charts provided by the project. The provision of reading kits to the children was very relevant, as parents, being poor could not afford to buy story books and other material for their children.

12 orientation sessions were organized for parents to sensitize them about the importance of girls’ education and to encourage them to send their daughter to school and to keep already school going girls in school. The sessions were liked by parents. These sessions also helped in increasing engagement of the parents with the school.

The effectiveness of the project was above board as no such interventions were introduced in schools for the children. The interventions remained effective in many ways. Training of teachers enhanced their skills of teaching reducing the gap between the students and teachers as students started discussing issues without hesitation and fear with their teachers. Parents Sessions provoked awareness among parents to deal with their children in handling their education issues. They started visiting teachers to discuss the issues of their children confidently and the teachers being trained welcomed the parents. This on one hand enhanced communication between teachers and parents and on other hand, created sense in parents to play their role in the education of their children. Earlier they thought that being illiterate they were not able to play any role in this respect. Study corners at home increased the habit of reading story books at home involving the whole household members in one way or other as an innovative idea of story-telling. Strategies adopted as of training of teachers and parents sessions proved their worth as stated. Interventions changed the perception of parents. As mentioned earlier they thought that being illiterate, they could not play any role in the education of their children. Now they were having meetings with teachers, providing time to their children to listen their stories, awakening themselves along with their children while they were reading and also providing them fruit and nuts as refreshments during study etc.

4.4 Efficiency All the grantees came to an agreement that most of the initiatives introduced in the CRP programme resulted to be beneficial and efficient. The quality of implementation was thoroughly appreciated by all participants. The project managed to achieve the targets set proficiently. The overall programme was received by the community optimistically resulting in being advantageous for their capacity building. The entire programme was cost-effective and was managed completely within the given budget.

USAID-funded Pakistan Reading Project 24

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The training and sensitization sessions were detailed and helped the participants clearly understand the project objectives and activities. Teachers, parents and SMC members who were a part of the training commended the content and quality of trainings. These sessions increased parental engagement with the schools and reduced distances among the schools and the communities. All participants found these trainings to have resulted beneficial in their capacity building, increasing motivational level and; gaining confidence and awareness. Parents in general found the sensitization sessions to be fruitful, as parents resulted in committing to support the education of their children and to ensure their school attendance. The content taught in trainings was simple and according to the level and understanding of the participant. The facilitators were experts in their field and were able to discuss quite technical topics in simple manner making it easier to comprehend.

Short duration of implementation was the most predominant observation from almost all participants in the study. They felt that two or three day trainings were not sufficient to absorb the large amounts of new techniques and strategies that were being discussed in the workshops. More frequent visits and trainings could however lead to better outcomes.

The material developed and distributed was according to the social norms and was developed keeping in view the socio-cultural sensitivities after review by material review committees. The keen interest of the representative of the education department helped in ensuring quality of the materials developed and distributed. The materials were liked and appreciated by all stakeholders. The compilation and translation of the indigenous stories also generated interest of the local communities. Special efforts were made to make material easy to comprehend for non-technical persons that resulted in easy gain of knowledge for participants. The material designed for the project was appropriate and suitable as per culture. For example, material for Sindh schools included local stories like the story of the famous folk character from Tando Allahyar called Watayo Faqir. The students were already familiar with these stories and found it easier to relate with them.

The program was financially efficient as it utilized low to no cost interventions. Existing resources were utilized for implementation of this programme without monetarily burdening participants or their families. They built reading corners at home with the help of the kits given by their grantees. However, these activities utilized participants’ time effectively.

The different competitions, decoration of classrooms and reading corners, story-telling sessions, usage of LED to watch animated learning videos and other group activities among most of the schools proved very effective and efficient ways of involving and motivating the school administrations, teachers and students for enhancing their skills and developing interest. In schools focusing on children with special needs, provision of books, visual aids, training manuals and screening tests reported to have been performed with a high degree of efficiency.

For all grantees focusing on parental engagement and children with special needs, the results can be described as being beneficial for all participants bringing positive impact in the community. As per the participants, outcome of this initiative turned out to be exceptional.

The program was efficient enough in utilizing its resources by producing the desired results. This utilized the existing resources of schools without putting additional burden on the district resources. Only by changing the perception of teachers and parents by providing them training, the project changed the environment of schools and homes towards reading. After training, the teachers were able to create interest in children to

USAID-funded Pakistan Reading Project 25

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

attend school, asking questions, discussing issues with them which ultimately led improvement in their performance. Provision of reading kits and reading glasses and screening tests for children with special needs also saved parents from any financial burden as they were mostly poor and could not afford all these facilities. Reading corners at home intervention also left long lasting effects without much cost. No complaint of lack of resources was noticed in implementation of activities.

4.5 Sustainability The overall results of the CRP programme indicate long-term effectiveness and helped in creating awareness among parents and teachers and enhancing reading skills of children. In some schools, the effects of the project could be seen even a year after its formal ending. On the contrary, it was also a challenge to put up with a few initiatives introduced in this programme that could not have lasting results.

Increased parental engagement was an important part of sustainability of the project. Since the parents were less aware of the importance of communication with their children and teachers prior to this programme, an understanding was created among them helping them to fill the communication gap. However, in most of the schools no proper mechanism existed for formal engagement of parents with schools. Most of the parents visited the schools only when their children had issues or when the school called them for a parent-teacher meeting, which was held a few times annually. Parental willingness to attend school meetings nonetheless were positive indicators pointing to the sustainability of the project. It was noted that in the absence of follow up, it was feared that even the current level of engagement would diminish.

Libraries provided by the grantees were still present in schools and children were still issuing books from them, which showed the sustainability of the initiative. However, almost all participants expressed their observations by saying that due to financial constraints libraries had not further been developed and additional books had not been provided by donors. In some schools, the books given to the schools had either been lost or had been worn out and could not be used for long. This meant that students of the next classes would not be able to use the material which had been provided during the project and would not be able to improve their reading skills. Respondents also mentioned that additional books were needed to keep up the children’s interest which could only be provided by donors after their design and careful consideration of words and images, as colorful books with images made children gain more interest in reading. Some schools also praised the addition of daily newspapers and magazines, which kept the parents engaged with the school administration. As parents would daily visit school in that duration to learn about daily news. However, as soon as the project ended, parents’ interest in visiting schools also lessened due to unavailability of newspapers and magazines.

The project built the capacity of teachers and equipped them with skills to help the students in improving their reading skills. Despite the fact that almost one year had passed since the completion of the project, in many schools, these techniques were still being used by the teachers in the classrooms. On the contrary, some teachers were of the opinion that they followed the orientation given to them in an effective manner and facilitated in conducting the activities smoothly. But they said that as soon the project ended, the activities could not be continued due to financial constraints and they were unable to help students in enhancing their skills through the use of innovative approaches.

It was also observed that the initiative could have been sustained for as long as the trained teachers were teaching in schools, however once those teachers resigned; the programme was unlikely to sustain. They were of the view that the 2 – 3 days trainings were not enough for them to further train teachers, so they suggested that refresher courses must be implemented.

USAID-funded Pakistan Reading Project 26

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Other initiatives that required improvement included issues like providing a limited number of books and reading corner decoration kits, due to which all children could not benefit the advantage of this project and that next classes would be without the support material required to improve the reading skills. Additionally, the provision of glasses to the children with special needs was proven effective as the children provided with eyesight glasses were able to read easily from a distance but there were some children who had either lost their glasses or had left them at home. It was suggested that in future the organization should focus mainly on screening and identification of children and raising awareness amongst parents about this issue. In some schools SMCs had become dysfunctional in the last few months due to re-election of committees in schools. However, the participants who had been a part of the project had gained awareness and were overall pleased with the intervention.

As seen from above that in fact the interventions were meant to change the perception of teachers, students, and parents towards establish habit of reading of children and this objective was mainly achieved by training of teachers and parents which created friendly environment among all three segments i.e. teachers, students and parents lessening the gaps among them to resolve the education issues. So in this respect there is no issue of sustainability. However, somehow it was noticed that no addition of books could take place due to financial constraints of schools. Books were misplaced due to unsafe (without locked cupboards) libraries. Lesson learnt is that after the project closure, the project activities involving finances should be made regular activity with regular budget of the entities. In fact this type of projects’ main objective is always to introduce something new to make the entities habitual of those activities even after the project and that is only possible through regular budget after project closure. Otherwise, it is very evident from the stated above that the most of the activities were still alive after the project.

4.6 Parental Engagement The CRP project focused on increased parental engagement which was one of the major themes of the programme. Majority parents who were a part of this project were illiterate, where fathers worked on daily wages for their bread and butter; and mothers were housewives. However, the response of maximum parents to the programme was commendable.

Parents positively responded to the orientation and sensitization sessions that aimed towards increasing awareness in parents. Moreover, it resulted in the capacity building of parents, as well as major increase in their motivational level and developed their self-confidence. Parents in general found the sensitization sessions to be productive, as they committed to support the education of their children and to ensure their school attendance.

Parents created an understanding that helped them fill in the communication gap they had with the children and their teachers. Parent-teacher communication increased consequently, as parents gained confidence to speak with teachers and teachers became more accessible to parents than they were before. One of the most important aspects that parents were pleased with, was the reduced communication gap with their children. Parents became expressive and developed friendly behavior towards their children and vice versa.

Parents also became more concerned about their children’s educational matters and encouraged their children to read and study more. They started taking keen interest in their children’s school matters.

However, the gap that remained was that no proper mechanism existed for formal engagement of parents with schools. Most of the parents visited the schools either in case of an issue with their children or if they were called by school authorities for a parent-teacher meeting. Parental willingness to attend school meetings nonetheless was positive.

USAID-funded Pakistan Reading Project 27

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Overall, parents’ engagement produced positive results. However, it was noted that the current level of engagement could weaken if there was no mechanism of follow up and monitoring.

4.7 Children with Special Needs The screening testing processes carried out by the organizations in their respective areas, resulted in identification of greater number of the children than the expected numbers. This showed the gravity of the issues. In both the projects, it was found that parents lacked awareness about the problems of the children. They were not able to help their children, as they were unable to identify the issues. Therefore, both the projects (RWF in Sajawal and KAEED in Gilgit) were welcomed and appreciated by parents, communities and the education departments.

In both the districts, the capacity of teachers to identify and help the children with special needs resulted to be one of the key factors in negligence of children. The teachers also agreed that prior to the project; they lacked the capability to identify the children with different learning issues. The teachers had never received any training to help in identification of such children.

The projects were successful in raising awareness among parents, teachers and the communities about the issues being faced by children with special needs. Along with raising awareness, the projects also carried out interventions for identification of such children and helping them through different initiatives like provision of vision glasses and building the capacity of the teachers to help the visually impaired children more effectively and efficiently. All these steps helped the children in overcoming their reading problems.

The strengthened capacity of the teachers in both the places is key factor in ensuring the sustainability of the project beyond its life. Not only were the teachers able to identify the needs of the children at earlier stage and help them, but the attitude of the teachers towards the children had also improved. Instead of punishing the children for their slow learning or their reading issues, they were trying to identify and resolve the issues. The socio-economic status of the parents in both the areas was not very high. It would have been difficult for the parents in both the areas to go for the screening tests and subsequently look for the means to resolve the issues. The strengthened capacity of the teachers to screen the children was a step in the right direction as they could continue screening the children in schools and identify their problems at an early stage.

The projects for children with special needs were also welcomed and appreciated by education departments in both the places, as the departments didn’t have any resources at their disposal to do so on their own. The education departments also do not have any training programs for the teachers in this regard. Therefore, the program was welcomed by the local education authorities

4.8 Student Evaluation While the actual ranges for each of the grantees’ student evaluation results varied, all 7 of them achieved aggregate levels of fair or satisfactory. Students from schools intervened by HDF, AF, RWF and SRPO all achieved fair levels of achievement, with aggregate scores of 67.5, 63.8, 80.69 and 73.72 words respectively. Whereas, students of schools intervened by FIELD, KAEED and HF achieved satisfactory levels of achievement with aggregate scores of 41.88, 22.60 and 50.27 words respectively.

Variations across geographical regions such as socio-economic differences, capacity of teachers, and overall standard of education can contribute to differences in the results of all grantees. Therefore, each of them has been assessed on an independently relative scale, although the scores are absolute.

USAID-funded Pakistan Reading Project 28

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

4.9 hanges occurred The project contributed in bringing about change at many levels. The most significant of which was developing a reading culture among the students by providing them a conducive and supportive environment both at home and at school. In different projects, libraries were established in schools, reading corners were decorated at homes of the children, and reading competitions were held among students. All these steps motivated the children to enhance their reading skills. The communication between the students and the teachers also improved significantly, thus improving the learning environment. The overall results of the school also improved. The children, who previously felt shy and reluctant to talk to teachers and parents, were also now more confident in sharing their problems.

The project also contributed in the development of parental engagement. It was generally felt that previously the coordination and communication between the school management and the parents was not up to the mark. With a focus on parent teacher coordination meetings, joint orientation sessions for parents and teachers and increased role of parents in school management through school management committees, the project succeeded in bringing parents to school. This increased their engagement and interest.

Another significant change was the engagement of parents in the learning process of their children. It was a common complaint by the teachers and the education department authorities across different projects that parents did not take much interest in the studies of the children and did not provide them adequate support. Through different activities like parent –children story telling sessions, the parents were encouraged to provide greater support to their children in their studies. The parents were also motivated to improve the learning environment at home and make it more conducive through decorating reading corners in their homes. The communication between the parents and the children also improved.

The project not only contributed in enhancing the capacity of the teachers but also changed their attitude towards their students, especially slow learners. The representatives of the education department and teachers themselves were of the view that before that project, the teachers had limited capacity in identifying the issues of the children and did not know about the latest techniques for enhancing the reading skills of the early grade students. After the training workshops and orientation sessions, teachers developed a greater understanding of the issues of the students with different disabilities and adopted a more positive attitude towards them. Instead of punishing them, the teachers were now trying to help them out. The motivation of the teachers had also increased. The competitions between different schools also motivated both the students and the teachers to perform even better.

Another change caused by the project was the increased awareness and realization of the existence of the problem of disability and the issues of the children with special needs. All stakeholders agreed that before this project, the parents and the teachers were not aware of the problems of the children with different disabilities and did not have the capacity to address them. Despite the fact that several organizations had implemented projects in the target area, this was the first project of its kind.

The project also raised awareness in the communities about the information about the importance of the education of the children and specially the importance of girls’ education. This is expected to bring a very positive change in the lives of the girls.

4.10 Advocacy Index Scores As shown in Figure 1, individual scores for each of the seven CSOs ranged from 23.16 to 24.75, with HF scoring the lowest of the index scale and KAEED scoring the highest on the index scale. The aggregate score of the entire cohort of CSOs was 23.76, which also falls within the overall range of scores. This is evidently a highly

USAID-funded Pakistan Reading Project 29

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

narrow variation between the various CSOs, and demonstrates that all the CSOs delivered similar advocacy performance. This is specifically because the projects for all seven CSOs were similar in their requirements and activities, and the profiles of the CSOs themselves were also very similar. The scores also demonstrate that the size of the CSOs – in terms of human and financial resources and the scale of implementation – was not an influencing variable. Larger size CSOs, such as HF and HDF, score lower than smaller CSOs such as KAEED. This observation should be noted while taking into account that the Advocacy Index scores are only a measure of advocacy capacity and not organizational capacities more broadly.

Figure 2: Advocacy Index Score of All Grantees

Advocacy Index Score

SRPO Azat Foundation

RWF FIELD KEED

Human Development Hashoo Foundation Hashoo Foundation Human Development FIELD Foundation Azat Foundation KEED

22 22.5 23 23.5 24 24.5 25

The following are the measurements of each of the CSOs capacity along the eight advocacy dimensions that contribute to the final the Advocacy Index score.

Table 3: Advocacy Index Score for all Grantees SRP RW HD Cohort Advocacy Dimension KAEED HF FIELD AF O F F Scores Ensuring the issue is timely and important. 3.5 3.5 3 3.5 3 3.5 4 3.43 Devising a strategy or action plan. 2.8 3 3 3 3 3.2 2.8 2.97 Collecting information about the issue like 3 3 2 2.4 3 2 2.6 2.57 through research and talking to stakeholders. Formulating a position or recommendations on 3 3 3 3 2.3 3 3.3 2.94 the issue. Securing sufficient resources for the initiative. 3.2 3.75 4 4.5 4.5 4.5 3 3.93 5 Building partnerships or coalitions with other 2.75 2.5 3.5 3 3 2.75 3.5 3 interested groups. Taking actions or activities to promote your 2.4 1.5 1.7 1.86 3 2 1.86 2.06 position or recommendations on the issue. 2 7 1 Monitoring any changes resulting from your 2.2 2.7 3.2 efforts to ensure that they are implemented 2.75 3 3.25 2.75 2.86 5 5 5 well.

As shown in Figure 2, taken as a cohort the CSOs were weakest in their capacity to “take actions or activities to promote your position or recommendations on the issue” with an aggregate score of 2.06. This is an expected scoring result, given none of the seven CSOs were advocacy organizations nor were their projects advocacy specific. Promoting policy positions or recommendations of their area of focus were therefore not the priority.

USAID-funded Pakistan Reading Project 30

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

With the exception of one CSO RWF, none of the CSOs worked in coordination with the local media to influence the stakeholders in their community and institutional environment. Taken as a cohort, the CSOs were strongest in their capacity to “secure sufficient resource for the initiative” with an aggregate score of 3.93. The CSOs dedicated their human resource, included their programmatic leadership outside the project team, mobilized voluntary action consistently, and were able to secure and implement sufficient financial resources in the form of their grant.

Figure 3: Advocacy Cohort Score of All Grantees

Cohort Scores

Monitoring any changes resulting from your efforts to Ensuring the issue is timely and Taking actions or activities to important. promote your position or Building partnerships or Devising a strategy or action plan. coalitions with other interested Securing sufficient resources for the initiative. Collecting information about the Formulating a position or issue like through research and recommendations on the issue. talking to stakeholders. Collecting information about the Formulating a position or issue like through research and recommendations on the issue. Devising a strategy or action plan. Securing sufficient resources for Ensuring the issue is timely and the initiative. important.

0 1 2 3 4 5

Findings Given the scope of CSO interventions funded under CRP was not advocacy, but instead focused on education, the CSO advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. As mentioned earlier, none of CSOs themselves were advocacy-focused civil society organization, and therefore only conducted advocacy as a crosscutting part of their overall work. Generally, the CSOs had been involved in one to two advocacy initiatives prior to 2016, and only two CSOs (HDF and HF) had been involved in advocacy activities after their grants closure. The overall cohort score of 23.76 however is significant, and correlates that the CSOs did have the capacity to influence political or social change in their area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II was both timely and important across all the geographies and demographics for each of the seven grants. All the CSOs gathered both formal and informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. They were also provided with crucial evidence from CRP, which strongly facilitated development and implementation of evidence-led activities. In fulfillment of their grant requirements, all CSOs demonstrated strategic considerations and action plans for their grants. None of the CSOs however regularly practiced the development of formal strategic documents, nor did they have any alternative plans in case the intervention faced any challenges. Notwithstanding this, the CSOs demonstrated sufficient monitoring of their activities and the changes resulting from these activities. This enabled the CSOs to be more responsive to any challenges: they delivered alternative activities even if there was not prior alternative plan available.

USAID-funded Pakistan Reading Project 31

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

While the CSOs communicated and implemented activities upon a clear position on reading skills and habits, they did not have any dedicated advocacy documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects the CSOs’ agenda- setting capacity around their specific issue, given any turnover of current staff or change in funding priorities may steer a CSO towards a different position or restrict it from effectively or regularly communicating to decision-makers on its specific issue.

As mentioned earlier, the CSOs demonstrated sufficient mobilization of both financial and human resources. They secured and expended the grant fund, and also mobilized voluntary participation required for the project. Both the MRCs as well as the SMCs activated by the CSOs for their grants acted as crucial institutional mechanisms for this mobilization. Across all grants, a key attribute of the CSOs was that they had in-depth and long-term involvement and knowledge of their context and communities. This enabled the CSOs to mobilize large voluntary participation, and to also develop effective partnerships or coalitions with other interested groups, including the local government and community leaders. A key gap in the CSOs’ stakeholder management however was working with the media, which the CSOs did not mobilize or involve in the process of their grants, with the exception of one CSO. Beyond press coverage for opening or closing event of the grant, no media representatives participated in any of the CSOs activities. This leads to the weak promotion of CSOs’ position or recommendations on how to improve the reading skills and habits of learners in Grades 1 and 2. It should be noted however that the CSOs’ media landscapes were generally driven less by standard urban communication channels such as a print or electronic media, and more driven by community and interpersonal media. All the CSOs demonstrated their use of these two forms of media.

The majority of CSOs were able to improve their advocacy capacity as a result of the CRP grants. While no Advocacy Index measurement of the CSOs was implemented prior to the start of the grants in 2016, there is sufficient evidence to conclude that the CRP grants improved the advocacy performance because of technical and financial assistance. The technical requirements of the grants for evidence-led and planned activities ensured that the CSOs devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, SRPO also demonstrated an improved capacity to receive and utilize sufficient resources. Based on the responses of the CSO representatives during the interview, the Advocacy Index exercise itself allowed the CSOs to better recognize advocacy within their own work, and to use that improved recognition to better define past, current and future work around advocacy. To be sure, while all the CSOs and their grants were not advocacy driven, they consisted of the majority of advocacy dimensions that constituted the Advocacy Index. In other words, there was a significant overlap between the CSOs and their grants, and the advocacy work as defined by the Advocacy Index. The overall cohort score of 23.76 reflects this significant overlap.

4.11 Success Stories A number of success stories were shared by teachers and parents throughout the evaluation process. Some of them are listed below:

“Zainab from grade 1 of Government Girls Public School Mahonari, Mardan got her story published in the Indigenous story book containing stories developed by different children from the 20 beneficiary schools after the intervention, whereas Zainab previously had no interest in book reading or story writing.” – Proudly said her teachers during the FGD. GBMS Sheikh Majid, is a stand out success story in Kharan, where, inspired by the room decoration by the students of primary section for the competition, the students of middle section decided to decorate their class rooms on their own.

USAID-funded Pakistan Reading Project 32

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

“Observing her son’s growing enthusiasm for reading and his interest in reading stories at home, his younger sister who was only 3 years old had started to sit with him as he read her the stories from the library books he brought home. Even before she had started school, she was learning to read and speak Urdu with the help of her brother, who was in class 2. She had started insisting for us to get her books and copies so she could practice writing just like her elder brother.” – Said an excited parent from G.P.S. Hajiga, Skardu

4.12 Limitations The evaluation was constrained by few factors listed below: In Gilgit, the regular classes were not taking place due to winter breaks, though the schools were still open. Most of the children along with their parents had gone home to their native villages for the winter break. As a result, it was difficult for the IP’s focal person and school administration to gather required number of relevant people together that would provide a representative sample. The study design included FGDs with parents. Most of the parents were quite poor and worked as daily wage laborers. They found it difficult to forgo their daily wages and come to the schools to participate in the FGDs. Due to unavailability of groups of the parents; in some districts all three FGDs were not possible. In some places, the EDO and the focal persons of the department, who were part of the material committee, had been transferred. Therefore the key informant interview had to be conducted with the officials who had been posted in their place. In a couple of districts, some of the participants who had been a part of the training were unavailable. For instance, the principal of one school had been transferred to another school a few months after the CRP project had been implemented. Though the new principal did have knowledge about the project, yet was not able to answer all the questions. When this evaluation took place, a year had already passed since the intervention had been implemented, and many students during their individual reading evaluation were unable to read fluently as they had not been reading books for months. Reason being, at some schools, books at libraries had either been torn or issued to students and never returned. Other reasons included financial constraints of schools due to which new books were not added to the libraries. Even some parents admitted that they are unable to buy books for their children due to financial issues even though their children love reading.

4.13 Conclusion All of the implementing agencies managed to meet the designed objectives as laid out in the grant agreements through the implementation of project milestones. The project activities were received positively among the communities, as leading to increasing parental engagement with schools and their children in educational matters.

For organizations working solely on the theme of increasing parental engagement to develop interest of children in reading, establishment of libraries and reading corners by HF and HDF contributed to increasing children’s interest in reading as well as improving school attendance. For SRPO, the activation of SMCs to mobilize communities for parental engagement and provision of reading kits to children contributed to its effectiveness. In the case of FIELD and AF, increased parental engagement was perceived through the discussion held with stakeholders, where previously no such interaction between parents and schools existed.

Interventions by RWF and KAEED focused on improving reading in children with special needs, and were designed to help identify and assist children with visual impairment. RWF’s initiative to increase capacity of teachers to identify visual impairment in children through detailed trainings and manuals contributed to the

USAID-funded Pakistan Reading Project 33

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

sustainability of the initiative, as the teachers would continue to use skills acquired during the course of the project even after the duration of the project. KAEED’s intervention displayed a high degree of community ownership, contributing to the intervention’s efficiency and sustainability. Awareness among teachers regarding inclusion, differently abled children and specific problems such as visual impairment had a considerable effect on their attitudes, resulting in the adoption of new techniques learnt during the training and contributing to a more conducive learning environment.

However, the short duration of project activities was a common complaint in all the interventions. The need for refresher courses and more robust follow-up and monitoring mechanisms was highlighted by all stakeholders included in the evaluation. SRPO, RWF and AF’s interventions were noted for the cessation of activities after the completion of the project duration, leading to issues of sustainability. Teachers in areas covered by HF, HDF and FIELD requested for refresher courses in order to better assimilate the lessons learnt during the project and to address issues of high staff turnover of position transfers in government schools.

Although the grants were perceived to be largely effective in changing the target communities’ perceptions and attitudes towards children’s reading skills, prevailing socio-economic issues of illiteracy and poverty among parents may have prevented the full extent of the desired outcomes to be achieved. Furthermore, undesirable condition of government schools in terms of physical infrastructure in some cases and low levels of motivation in the teaching staff prevent the effectiveness of such interventions, considered as being of secondary importance in terms of priority. Though the grants in this project proved quite effective and efficient, yet sustainability was highlighted in some of the places as a major challenge. At some places the student activities had stopped. Interaction between the parents and the teachers was missing. This may be probably due to weak follow up and lack of resources at schools. In some districts the libraries and reading rooms in some schools were not functional.

4.14 Recommendations The main goal of the project was to enhance reading habits of children by further designing main objectives such as creating interest in children for reading, skills in teachers and sensitization of parents to handle the issues of children, For this a variety of innovative interventions were applied which remained successful to a great extent and appreciated by all target stakeholders i.e. children, teachers and parents. Communities have not seen this type of activities earlier. In the light of results drawn some recommendations are outlined as under:

1. As the project has been very successful, this may be replicated in other schools/ districts as well.

2. It was observed that books could not be added in schools, some lost and in some cases children did not return the books back to schools. Due to financial constraints this activity was hampered and could not go long. So it is recommended that the activities requiring finance should be made a regular activity by the authorities with regular budget and the project authorities may ensure this arrangement by dialogue with authority’s prior to initiation of such activities. They may now even request Education Authorities for the same for continuation of the activities. This strengthens the sustainability of the project activities.

3. It was learnt that after project interaction between parents and teachers faded. This may continue by formulating a mechanism to call parents in schools as a regular phenomenon with proper intervals or on need basis. To resolve urgent issues of children, mobile calls may be made to parents. It has also been observed that majority of parents are poor and work in cities and are unable to spare time to attend teacher –parents meetings. Mobile calls may also help in this regard.

USAID-funded Pakistan Reading Project 34

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

4. To ensure continuation of activities (especially which do not call for any cost such as interaction between teachers and parents, occasional visits to home to observe functionality of study corners at home etc.) a monitoring committee may also be formed at school level.

5. In case a student does not attend school for more than three days without intimation, a representative should visit home to know the reason and try to resolve the issue .This will create harmony among teachers, children and parents which will help improve performance of children ultimately.

6. From the results of the project, this may be inferred that the project was cost effective. Therefore, Education Authorities need to make efforts to introduce such activities at their own using their own resources in other schools as well.

7. One rupee idea for library may also be a good option of fund raising for strengthening library.

8. Short duration of implementation of activities/ and for training of teachers was observed which may be taken care off in future.

9. District Education Authorities may introduce training /refresher training for teachers on the lines of the project to enhance interaction among teachers, children and parents.

USAID-funded Pakistan Reading Project 35

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

ANNEXURE

1. Grantee Wise Evaluation Findings

1.1 Hashoo Foundation (HF)

Background and Desk Review HF is an organization established in 1988 with the aim to enable and empower communities to be independent by facilitating equitable access to opportunities. HF as a grantee implemented the project of “Improving Reading Habits of Grade-I and Grade-II through Parental Engagement” in 40 schools of Islamabad.

By adopting pre-defined methodology, three schools were selected randomly. The selected schools were Islamabad Model School for Girls Alipur, Islamabad Model School for Girls Mallot and Islamabad Model School for Girls Kuri.

Objectives Following were the programme objectives specified by HF: To improve classroom learning environment for reading in 40 schools of Islamabad in Bara Kahu and Nilore, by reaching over 800 Grade I and II students; To upgrade community based opportunities to practice reading out of school.

1.1.1 Targets set vs. Achieved Out of all set targets, HF achieved excellent level by completing targets within the given timeframe. During the grant period HF was able to complete 25 cluster based sensitization sessions for parents and teachers in the selected schools, followed by 25 two-day trainings on promotion of reading habits and skills for parents and teachers of grade I&II.. The 12,000 Rotary books were presented to the Material Review Committee (MRC) having nominated participants from education department, for review and approval, followed by placement in 40 small reading libraries in the selected schools. Flip Cards, diaries, and story reading materials were distributed among the students of Grade-I and Grade-II in 40 Schools. Conducted activities such as “Read a Book, Donate a Book” campaign in 25 clusters, parent-teacher meetings and labeling of household items. 40 Urdu stories shared by parents during organized story-sharing days were selected, reviewed and approved by MRC and compiled in a book form.

1.1.2 Relevance The Assistant Director of Training of Education Department was of the view that most of the initiatives implemented in Islamabad in the past were related to reading and writing, however none of them sustained for too long. She said that the CRP programme targeted communities including children, parents and teachers in order to bring awareness on household level. This included distribution of story books, library enrichment, participation in classrooms and reading corner development. Children had lack of confidence prior to this intervention and had a tough time in reading and sharing their issues to teachers & parents. The Assistant Director of Training also said that the teachers were not skilled enough to motivate children and parents. She also said that the books were appropriate in a way that they were made colorful and attractive for all children to take more interest. They all were appreciated to participate through involvement of their parents that boosted the confidence among the children.

USAID-funded Pakistan Reading Project 36

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Praising efforts by HF, the Assistant Director of Training from the Education Department said: “Such exercises involving training and practical participation with illustrating some real life examples could encourage the teachers to implement such kind of exercises in schools as well.”

The focal person from HF had been involved as a trainer. Before this project, the situation was very bad and parents did not know anything about reading as per her observation. Motivation level of teachers was average before the initiation of project, but after the involvement of parents and teachers, they all were happy and welcoming, as per observation and discussions between HF and school management.

As per the school principals prior to this initiative, children’s attendance was unsatisfactory, teachers were less motivated and parents had no involvement in their children’s school matters. However, motivation of teachers and parents had increased. Talking about other initiatives, all principals were of the view that other projects had been initiated in schools, but none was related to reading and community engagement.

The principal of Islamabad Model School for Girls, Alipur added that: “CRP was a step in the right direction. Generally, this intervention was unique and results were satisfactory.”

Now 50% children read and had books at home as per the principal. They have reading related activities at school as well but communities themselves were not actively involved in reading activities due to their busy routine.

All teachers opined that before the CRP intervention, children had difficulty in Urdu reading and speaking, “Torr Jorr” i.e. constructing/deconstructing words, they did not know pronunciation; and could not identify Urdu letters. Teachers also said that due to high strength of children in schools, it was not easy to give individual attention to all children. Children were also very shy, so they hesitated to share their issues. They also said that they did not have libraries in school before this project and all the focus in schools was on completing syllabus, so no book reading activities were implemented.

A teacher added: “Children were not capable of reading and recognizing Urdu alphabets before, but now they have improved.”

Overall parents were of the view that before this programme, children’s performance in school and interest in reading was below average. Earlier there was no reading culture at home, but after this intervention children took interest in reading at home as well. Moreover, parents appreciated teacher’s skills in cooperating with regard to any issue their children were facing. The parents were of the opinion that with the provisioning of school libraries it had led the children to take keen interest in reading in general, as well as studying and attending school regularly. They also said that there was a huge communication gap between them and their children; also with teachers, prior to this project.

One of the parents said: “The entire system of this school has changed after this programme, my kid studies with interest now.”

1.1.3 Effectiveness The Assistant Director of Training of the Education department was of the view that the CRP initiative had positively contributed to the capacity building of participants. She said that any programme that had been implemented in the past was not long term as children would have to return the books after a specific period of time. She also was of the view that colorful books gained more children’s attention than curricular books did. The programme had lasting impression especially on parents and teachers, as they showed keen interest

USAID-funded Pakistan Reading Project 37

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

in learning new techniques. She also spoke about lack of confidence in children and how first they used to feel shy but through these activities they started sharing their views and ideas openly. About the trainers of HF, she said that they were very well skilled and all stories related material was not just read verbally but was also presented for better understanding. Due to this reason parents highly appreciated such activities because they were illiterate so the activities were really helpful for them. Results of the training and sensitization were fruitful because anything that was given to children and their parents, sensitization was required to get better results. She also was of the view that, giving books in hand of children and also telling them stories practically was really helpful to catch the attention and improve the listening skill of the students. HF team visited parents’ houses to verify and check that reading corners were developed and writing items were labeled at their learning corner.

The Assistant Director of Training pleased with intervention of story books said that: “The books given by HF were not taken back from children and were not only placed on a library shelf as decoration pieces.”

According to the focal person from HF, awareness was developed in parents and teachers after the intervention. A lasting impression on parents, children and teachers had been made by this project. Parents motivated their children to read more enthusiastically. Communities had a positive response to parent-teacher meetings and stories development activities. After this project, children had libraries, separate reading hours at home, parent-teacher communication increased, and donate a book campaigns were introduced on community level. Teachers adopted innovative techniques in promoting children reading habits to some extent.

As per the principals, CRP was an excellent initiative. Before the intervention, the parents undermined and underestimated their importance & role in the education of their children. However, after the project’s implementation, this attitude was improved. There was a satisfactory change in the participant’s attitude. More frequent visits and trainings by HF could improve this initiative.

One principal added: “Parents are welcome to meet school authorities anytime they want regarding any issue.”

Motivation level of teachers had become better. Moreover, children’s reading became satisfactory after this project. It developed students’ reading skills and creativity. It developed keen interest in children with regard to their Urdu speaking skills, word construction and improved their attendance at school. Overall, a lot of improvement in children’s attitude and interest level was observed. Parent-teacher communication was enhanced and regular meetings had started taking place.

Teachers agreed that they learnt from the training sessions, like skills of engaging with children in class in an interesting way, promoting group activities, increasing interest of student in reading parental engagement and most of all enhanced their communication skills. After training, communication gap between students and teachers was reduced thus leading to students sharing their issues.

Some parents attended the training once and some twice, which helped them attaining awareness. They were of the view that after the programme training, they learnt the importance of reading and communicating with their children. They also said that their children had learnt to be capable of communicating with their parents more confidently. They all had made reading corners at home and had labeled them too, which made the children gain interest towards sitting in that corner and studying/reading. Most of the parents said that their

USAID-funded Pakistan Reading Project 38

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

children liked to read stories having morals and lessons in the end. Teacher-parent coordination had also increased as parents had gained confidence and awareness, whereas teachers attained new skill sets after training.

A parent from IMSG Alipur said: “Earlier when I used to ask my daughter about her homework, she used to misbehave and say that the teacher was absent today. But now she attends school and issues books from school on regular basis.”

1.1.4 Efficiency The Assistant Director of Training of the Education Department was of the opinion that the CRP initiative had played a positive role in the communities. The books given by HF were not taken back from children after a specific time like other programs in the past, which is why this programme was more long lasting. She also said that she personally was a part of the training sessions. She appreciated the reading material provided to children. She also commended the establishment of small libraries not only in schools but also the establishment of reading corners and labeling of household items at home. She observed that HF team also visited the houses and not only restricted themselves within schools boundaries but also check the library corner at their home.

This programme was received by the community positively according to the project focal person’s observation. This project was motivating for parents, children and teachers. Existing resources were utilized for implementation of this programme up to some extent. The programme did not put financial burden on district resources. However, it did utilize parents’ and teachers’ time effectively.

The HF’s focal person further added: “Training time duration should increase and refresher workshops should be executed in order to maintain the efficacy of the project in the longer run.”

Principals were generally satisfied with the CRP initiative and found it to be exceptional. There was a satisfactory change in the participant’s attitude. They had reading related activities at school as well. One principal said that they have reading activities in school on daily basis. More frequent visits and trainings could however, could lead to better outcomes. Another principal was of the view that raising awareness of mothers and setting up reading corners in homes have been the most effective features of the CRP program. Overall, they found this programme to have contributed to the capacity building of participants.

Teachers said that training sessions were very beneficial in the building their capacities. They also observed that the environment of schools had improved generally, children were happy and teachers were taking more interest.

Parents said that after the sensitization and training sessions, they have gained confidence and awareness on their role in developing reading habits of their children. They built reading corners at home with the help of the kits given by HF. All parents agreed that they were satisfied with their children’s performance which improved after the initiative and they also said that they all regularly visiting school to discuss any issues.

One parent, added, “I visit my daughter’s school every week. Last week I came to discuss her date sheet. Teachers are very cooperative now.”

USAID-funded Pakistan Reading Project 39

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.1.5 Sustainability The Assistant Director of Training believed that the activities initiated in the programme were still being performed. According to her, these initiatives were really helpful not only for children but also for parents because these initiatives also involved sensitization of parents which really helped them in understanding their role in developing reading habits of their children. She also observed that HF itself monitored all activities by visiting schools and houses.

The focal person for this project was of the view that, teachers adopted innovative techniques in promoting children reading habits to some extent. However, while enquiring about recommendations that according to her were most important after the implementation of the programme, she said that a lasting impression on parents, children and teachers had been made by this project already.

According to the principals the initiative could be sustained for long as the trained teachers were teaching in schools, the project will be sustained. Parents were welcome anytime of the year to discuss issues of any sort. Home-based activities were followed up on daily basis. However, libraries that had been developed under this project needed improvement by adding cupboards for safekeeping of books.

However one principal added that: “Safe keeping is required for the library like cupboard with lock and key, otherwise books are misplaced. Their children had been very keen to read books and borrowed them on daily basis.

Teachers said that different strategies learnt during these trainings were still used in their schools. Class or school level reading competitions took place. Libraries were maintained by children and teachers personally which sometimes becomes a financial burden. Overall, teachers were of the view that the schools were now performing better than before.

One of the parents said: “The school authorities are increasing books by themselves as no more books were added by HF”

All parents were satisfied with overall results of the children after the CRP project and also agreed that their involvement in their children’s educational matters had increased in the last six months. Every parent as a part of the FGD was satisfied with the overall results of the children after this project. The school libraries were still functional, but more books were not added by HF. However, all parents also agreed on the fact that this school was performing better than other schools in the vicinity, especially after the programme.

1.1.6 Parental Engagement Parental Engagement in Islamabad’s case showed positive results as a whole. The entire community was very welcoming towards such initiatives. Books provided were still in use and parents and children were keen to read these books. Parental engagement was developed and teachers’ motivation level was increased. Having developed the reading habits the children Urdu reading skills has also improved. Parent-teacher communication had also developed.

1.1.7 Students Evaluation Students from schools targeted by HF displayed an overall satisfactory reading score of 50.27 words per minute. Word reading exercise results displayed a higher accuracy than the paragraph reading exercise, reading an average of 41 out of 50 words correctly at an average rate of 52.92 words per minute. However, paragraph reading results were slightly weaker, with students reading an average of 33 out 60 words correctly at an average rate of 47 words per minute. The highest aggregate score achieved among the sample students

USAID-funded Pakistan Reading Project 40

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

was 119.61 words per minute while the lowest was 13.5 words per minute. 8 out of 30 students that participated in the evaluation displayed an unsatisfactory reading score of less than 34.7 words per minute, while 11 fell in the satisfactory range, which was 34.73-55.95 words per minute for HF and 11 fell under the fair and good range.

Table: Students Evaluation Results of HF

S.N School Level of Student’s Name read o Name Class Achievement minute Reading readper Aggregate Aggregate Time (sec) Time (sec) Time Paragraph Paragraph readper min Correct words words Correct Correct Words Words Correct Words Correct

24.0 30.4 Unsatisfactory 1 Malik Imran 24 60 2 43 70 36.86 0 3 * Muhammad 12.0 13.5 Unsatisfactory 2 12 60 Ayan 3 30 120 15.00 0 0 * 100.0 90.0 95.0 3 Fiza Fareed 4 50 30 0 60 40 0 0 Good 90.0 82.5 4 Sumaia Fatima 3 50 40 75.00 60 40 0 0 Good 136.3 102. 119. 5 Hamna Ubaid 3 50 22 6 60 35 8 6 Excellent** Kuri 18.0 23.3 Unsatisfactory 6 Asma Asif 18 60 3 43 90 28.67 0 3 * 54.0 64.5 7 Adeeba Naz 3 50 40 75.00 27 30 0 0 Fair 12.0 19.5 8 Laiba 3 27 60 27.00 8 40 0 0 Unsatisfactory 94.7 84.8 9 Sirat Fatima 3 50 40 75.00 60 38 4 7 Good 54.0 43.8 10 Faryal Fareed 3 45 80 33.75 27 30 0 8 Satisfactory 72.0 82.5 11 Sultan Umair 4 45 29 93.10 60 50 0 5 Good 44.0 55.9 12 Hadia 3 43 38 67.89 22 30 0 5 Fair 44.0 44.2 13 Ziniya 3 40 54 44.44 22 30 0 2 Satisfactory 60.0 70.5 14 Shameen Bibi 3 50 37 81.08 30 30 0 4 Fair 44.0 52.7 15 Nahil Rukhsar 3 39 38 61.58 22 30 0 9 Satisfactory Malot 34.5 40.6 16 Mar Yar 3 39 50 46.80 23 40 0 5 Satisfactory 54.0 58.9 17 Zar Khan 3 50 47 63.83 27 30 0 1 Fair 54.0 51.6 18 Sultan Umer 4 32 39 49.23 27 30 0 2 Satisfactory 46.0 41.1 19 Abdul Rehman 3 32 53 36.23 23 30 0 1 Satisfactory 42.0 41.4 20 Raja Shoaib 4 34 50 40.80 21 30 0 0 Satisfactory 42.0 39.8 21 Mahnoor Alipur 3 44 70 37.71 21 30 0 6 Satisfactory

USAID-funded Pakistan Reading Project 41

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

10.2 18.6 22 Aisha Bano 3 27 60 27.00 6 35 9 4 Unsatisfactory 44.0 55.0 23 Iqra Faheem 3 44 40 66.00 22 30 0 0 Satisfactory 22.3 24.2 24 Fatima Bibi 3 37 85 26.12 42 113 0 1 Unsatisfactory 53.4 42.9 25 Saiqa Bibi 3 46 85 32.47 57 64 4 5 Satisfactory 80.0 62.6 26 Hajra Bibi 3 49 65 45.23 60 45 0 2 Fair 17.0 15.5 27 Iqra Bano 3 28 120 14.00 34 120 0 0 Unsatisfactory 23.0 22.7 28 Tehreem Ayaz 2 45 120 22.50 46 120 0 5 Unsatisfactory 44.0 52.7 29 Zeenat Bibi 3 41 40 61.50 22 30 0 5 Satisfactory 46.4 56.9 30 Khatija 4 27 24 67.50 55 71 8 9 Fair Average 50.2 Satisfactory 7 Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent = 98.39-119.61; Good = 77.17-98.39; Fair = 55.95-77.17; Satisfactory = 34.73-55.95; Unsatisfactory = 13.50-34.73

1.1.8 Changes Occurred As per the Assistant Director of Training, the programme had lasting impression especially on parents and teachers, as they showed keen interest in learning new techniques. She also spoke about lack of confidence in children and how first they used to feel shy but through these activities they started sharing their views and ideas openly. She said that the trainers by HF were very well skilled and all stories related material was not just read verbally but was also presented for better understanding. Books provided were in use and books and parents and children keen to read these books.

As per focal person, “Lasting impression on parents, children and teachers had been made by this project. Parents started motivating their children to read. However, teachers’ motivation to improve their skills was satisfactory.”

Principals, teachers and parents: “All were of the view that the CRP project had developed awareness among the community and they look forward to more such activities. They believed that children improved their Urdu reading and developed story reading habits even at home. Parent-teacher communication was also enhanced and regular meetings started taking place, however a lot of more work was required within this project for improvement.

1.1.9 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of HF during the grant period. This measurement captured advocacy as the active process through which HF sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of HF improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, HF’s advocacy performance was measured.

USAID-funded Pakistan Reading Project 42

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Advocacy Index Score HF achieved an overall score of 23.12 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of HF’s overall score on the Advocacy Index across its eight dimensions:

Table: Advocacy Index Score of HF Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3 Devising a strategy or action plan. 3 Collecting information about the issue like through research and talking to 3 stakeholders. Formulating a position or recommendations on the issue. 2.3 Securing sufficient resources for the initiative. 4.5 Building partnerships or coalitions with other interested groups. 3 Taking actions or activities to promote your position or recommendations on 1.57 the issue. Monitoring any changes resulting from your efforts to ensure that they are 2.75 implemented well. Overall Advocacy Index Score(Maximum Score = 40) 23.12

As shown in the table above, HF was weakest in its capacity to “take actions or activities to promote your position or recommendations on the issue” with a score of 1.57. HF was strongest in its capacity to “secure sufficient resource for the initiative” with a score of 4.5.

Findings Given the scope of HF’s intervention was not advocacy, but instead focused on education, HF’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. HF itself is not an advocacy-focused civil society organization, having reported no advocacy initiative prior to 2016. The overall score of 23.12 however is significant, and correlates that HF successfully influenced political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II was both timely and important within the specific geography of Islamabad Capital Territory. HF did gather both formal and informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, HF also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, HF demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for HF to be more responsive to any challenges without the presence of a formal plan.

Moreover, while HF communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects HF’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer HF towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue.

USAID-funded Pakistan Reading Project 43

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

HF demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. Crucially, HF mobilized effective partnerships or coalitions with other interested groups, including the local government and community leaders. The MRC and the SMC were key means for both voluntary participation and partnership development. A key gap in its stakeholder management was with the media, which HF did not mobilize or involve in the process of their project. Beyond press coverage for one formal event of the project, no media representatives participated in any of HF’s activities. The gap between HF’s activities and the media limits actions or activities to promote their position or recommendations to improve the reading skills and habits in the local community. While no Advocacy Index measurement of HF was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the CRP grant initiatives improved the advocacy performance of HF. Because of the technical requirements of the grant for evidence-led and planned activities, HF devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, HF also demonstrated an improved capacity to mobilize sufficient resources. Beyond the life of the grant, HF has also started specialization of expertise and strategic priorities around education, with new human resource being hired and a flagship event on education planned for January 2018 that would be an opportunity for position formulation and a call-to-action.

1.1.10 Success Stories Mahreen, a student of grade 2 had issues reading Urdu, improved her reading skills after the CRP intervention. Hira, of grade 1 improved her reading to a greater extent, told her teachers.

1.1.11 Limitations The major limitation of this study was the unavailability of some of the participants who had participated in the trainings. For instance, the principal of one school have been transferred to another school a few months after the CRP project had been implemented. Also at one school in Mallot when children were required for evaluation, the focal person arranged students of grade 4, who were irrelevant as per our study as Grades 2 & 3 students were to be assessed. This issue was identified by APEX team and intimated to the focal person from HF and they arranged requisite grade students.

1.1.12 Conclusions Overall effectiveness of CRP project was evident in all schools and communities. The project was successful in increasing parental engagement with their children in their educational matters. Communication between parents and teachers also improved. Establishment of libraries played an important role in children’s interest in reading and attending school regularly. Establishing reading corners in their houses also helped children gain interest in reading and studying at home. However, increase in time duration of training and sensitization sessions was preferred by teachers and parents. Refresher courses and reimplementation of the project can prove to bring sustainability to the results of the project.

Human Development Foundation (HDF)

Background and Desk Review HDF is an organization established in 1999 with the mission to facilitate a non-political movement for positive social change and community empowerment through mass literacy, enhanced quality of education, universal primary health and grassroots economic development. HDF as a grantee implemented the project of “Reading Communities” in 20 schools of Mardan, targeting 1200 students of Grade I & II to improve the reading skills.

USAID-funded Pakistan Reading Project 44

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

By adopting pre-defined evaluation methodology, three schools were selected randomly. The selected schools were Government Girls Public School Mahonari, Government Public School Surkh Dheri and Government Public School Mirwas.

Objectives Following were the programme objectives specified by HDF: Ensure the reading environment and improved reading skills in the students of grade 1 and 2 of 20 schools through adequately trained teachers in District Mardan. Improve the engagement of parents in promoting a reading environment at home by reading mechanism for children of grade 1&2 in 20 schools. Develop reading communities and improve their engagement in promoting a reading environment around 20 CRP Schools of district Mardan.

1.2.1 Targets set vs. Achieved Project activities like Family Reading Time, Parent-Teacher Meetings, provision of specific reading material, story and word competition, establishment of reading corners, school libraries, one Rupee libraries etc. were designed to ensure development of reading environment at school, household and at community level to improve the reading skills in students of 20 targeted schools (10 male and 10 female).

Out of all milestones set by HDF, all targets met excellent levels of achievement. 20 digital libraries were established in 20 targeted schools of District Mardan, along with provisioning of books, magazines, LED Screens, Sound System, Newspapers, furniture and librarians. Two days teachers training and one day refresher course, with 40 teachers (20 Male and 20 Females) was also conducted. 1200 reading corners were established by providing reading materials i.e. Story books, magazines etc. to children and their families, including 80 “One Rupee Libraries.” 160 meetings (Parents Teachers Students and storytelling meetings) were aimed to be conducted, 80 meetings for male parents and 80 meetings for mothers. To promote the reading environment, word competitions were held among schools.

1.2.2 Relevance The District Education Officer (DEO), Mardan was of the view that most of the communities in Mardan were illiterate and had no awareness about libraries, story books or reading corners. Children had a tough time in reading and communicating their issues to teachers & parents due to lack of interest. Children also had lack of confidence. The DEO also said that the teachers were not skilled enough to motivate children and parents, as most of the teachers preferred the same old traditional way of teaching.

Observation by the focal person from HDF was that initially the community was not very welcoming towards this intervention; however, after training sessions they developed interest, as the initiative was unique in its own way. The focal person also observed that before this intervention, parents, teachers and students had a huge communication gap due to lack of awareness.

The focal person from HDF, said: “In the start participants were not very welcoming towards this programme. However, when we provided them with kits and products, they became happy and developed interest.”

Principals thought that prior to this initiative, presence of children was not regular, teachers were not motivated enough and parents had no involvement in their children’s school matters. But now their confidence was raised and they took keen interest in schools and reading related activities. Principals also said that libraries had significantly impacted the community, as before CRP grant there was no concept of reading

USAID-funded Pakistan Reading Project 45

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

story books and watching videos consisting of Poster stories to enhance reading & pronunciation skills in schools.

All teachers agreed that before this intervention, children had difficulty in Urdu reading and speaking, they did not know pronunciation; and most of all did not attend school regularly. They also said that libraries were not functional before this project. As for parents, all teachers said that they never used to visit school as they had lack of awareness. Teachers agreed that they used to implement old ways of teaching before the training sessions.

A teacher added: “We used to implement corporal punishment system in our school before this programme, which is why children were afraid of us. But after the training, we started teaching children with politeness, which filled the communication gap with them. As a result, they became bold and started asking questions openly.”

Parents opined that their children did not take interest in studies prior to this intervention. They also said that the school atmosphere was not friendly and their children hesitated to go to school as they were taught by force. They were of the view that they themselves (parents) were mostly illiterate and could not read Urdu, as Pashto was their first language, which is why they did not bother about their children’s performance or reading/writing skills and neither bother visiting school before their training, due to lack of awareness.

1.2.3 Effectiveness The DEO was of the view that the CRP initiative had definitely contributed to the capacity building of participants. Young staff and teachers had become more competent and showed keen interest in learning new techniques. This programme developed students’ reading skills and creativity. Community requirements were put into consideration in a more appropriate and effective manner.

Principals observed that the CRP initiative, developed keen interest in children with regards to their reading habits and improved their attendance at school. They also thought that all steps undertaken in the programme have been very effective in the community. Principals agreed that this intervention was very beneficial for children and it also upgraded the education system of their schools. The programme provided good learning to teachers and students.

Principal also added that: “While the CRP programme was running, parents used to visit school on daily basis for the newspapers and magazines. They would ask the teachers to tell them what is written in the newspaper so they could know about current global affairs too.”

Teachers agreed that they gained knowledge from the training sessions, like flash cards, silent reading, methods of teaching word structuring; promoting group activities and most of all enhanced their communication skills. Due to these communication skills, children started sharing their problems more comfortably. Moreover, teachers were of the opinion that parents also developed interest in their children’s matters and started inquiring about their children’s performance.

USAID-funded Pakistan Reading Project 46

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

When teachers were asked about quality of training, a teacher said: “Flash cards and Jorr Torr were techniques that helped a lot. Now children are good at sentence structuring.”

Parents said that with the sessions, they also learnt the importance of Urdu reading, as before the project most emphases were laid on English. They realized that before this intervention, they had a huge communication gap with their children. But after the sessions parents learnt how to deal with this issue, moreover friendly relation was developed with teachers as well

1.2.4 Efficiency The DEO was largely of the opinion that the CRP initiative had played a positive role in the communities and contributed to the capacity building of participants. Existing resources were utilized for the implementation of this programme and grantee provided kits to students, hence was not a burden on district level. Also, teachers who were young and enthusiastic have been motivating children by putting efforts on personal level.

The DEO said: “I have been a part of the competitions and ceremonies held in Mardan by HDF, and I was very pleased to see the reading corner models and decoration rooms.”

This programme was received by the community positively according to the project focal person’s observation. Parental engagement was developed and teachers’ motivation level was increased. Children improved their Urdu reading skills and developed story book reading habits. The entire programme was cost-effective and was managed completely within the given budget.

Principals shared their experiences with other NGOs and believed that in the past, none of the projects that had been initiated were effective enough. However, they were generally satisfied with the CRP initiative. They said that this intervention developed keen interest in children with regards to reading, Urdu speaking skills and word construction. LEDs, according to principals mainly helped improving the children’s attendance. They observed that the library played a significant role and had a positive impact on the community overall. However, the principal of one school said that due to time constraint it was not possible for them to have specific days/periods for story sharing or class decoration.

Principal said: “Children in my school are very keen to read books and issue them on daily basis from the school. They often fight over books too, as there are a limited number of books to issue.”

Teachers said that training sessions were for two days and were beneficial to boost their motivation level. However, increase of time duration of training sessions was preferred. They also observed that the environment of schools had been improved overall, but regular monitoring could be vital. School administration’s relation with parents and their children had developed after this programme according to the teacher.

After training and sensitization sessions, parents believed that they had gained confidence and awareness. They also said that teachers played an important role in this regard too. The parents had established reading corners at home with the help of the reading kits given by HDF. Parents said that especially after the CRP programme, their children’s schools were performing better than other schools in the locality. Parents also said that the installation of LEDs at school had played a positive role in improvement with regards to children’s attendance and better results.

USAID-funded Pakistan Reading Project 47

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

A parent said: “After this programme my daughter has started performing well in studies overall. Even though this is a girl’s school, I have admitted my younger son to this school as well. I am very happy with the environment of this school.”

1.2.5 Sustainability Although the DEO had not been able to monitor all schools herself, but believed that the activities initiated in the programme were still being performed. She believed that as HDF having an office in Mardan, itself was convenient for maintaining the sustainability of this intervention. However, follow up was required and orders were given to ADEO’s to visit all beneficiary schools and check progress.

The focal person for this project was of the view that, teachers had been motivated to assist children in improving their reading abilities. Parent-teacher communication had enhanced and regular meetings started taking place more frequently as compared to the situation prior to this intervention.

According to the principals the initiative had been sustained after the programme finished. Children did their homework responsibly with interest and parents regularly followed up. Parents and teachers communication had also been developed, where some parents met teachers almost every month and some parents usually communicate over the phone on daily or monthly basis. The library had been maintained fairly, however had not been further developed due to financial constraints.

When asked about the evaluation of children in her school, principal said: “Now children’s can even read course books on their own. This programme should further be implemented so new admissions could also take full advantage of the new techniques and learning.”

Teachers said that different strategies learnt during these trainings were still used in their schools, like Alif se Alfaaz competition, silent reading, group activities etc. Overall, teachers were of the view that the schools were now performing better than before, attendance had improved and Urdu reading was being promoted.

All parents were satisfied with overall results of the children after the CRP project and also agreed that their involvement in their children’s educational matters has increased in the last six months. The school library was still functional, but more books were not added, said the parents. They were of the view that the current teachers were skilled enough, but more teachers were not being trained. All parents also agreed on the fact that after the program, this school was performing better than other schools in the vicinity.

1.2.6 Parental Engagement Parental Engagement in district Mardan case showed positive results as a whole. Despite being illiterate majority parents were very welcoming towards such initiatives. They showed willingness to meet teachers more regularly to discuss their children’s educational matters. They also believed that the programme helped them build better communication with their children and they discussed issues in a more friendly way. Despite having difficulty in speaking and reading Urdu, as Pashto was their first language, parents showed interest in motivating their children to read books in Urdu. The overall increase in parental engagement in their children’s reading was satisfactory.

1.2.7 Students Evaluation Students from schools targeted by HDF displayed an overall fair reading score of 67.49 words. Relatively high degree of both accuracy and fluency was displayed among the 30 students assessed in sampled schools. Word reading exercise results show students reading an average of 45 out of 50 words correctly at an average speed

USAID-funded Pakistan Reading Project 48

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

of 61.9 words. Paragraph reading results were similar, with students reading an average of 51 out 60 words correctly at an average rate of 73.05 words. The highest aggregate score achieved among the sample students was 117.69 words while the lowest was 27.83 words. Only 3 out of 30 students that participated in the evaluation displayed an unsatisfactory reading score of less than 45 words, while the majority (15) fell in the fair range, between 63.77-81.75 words for HDF.

Table: Students Evaluation Results of HDF

read

Student’s School Level of S.No

Name Name Class Achievement per min per Aggregate Time (sec) Time (sec) Time per minute per Paragraph Reading Paragraph Correct words read words Correct Correct Words read read Words Correct Words Correct

1 Marwa 102.3 3 45 34 79.41 58 34 90.88 Good 5 2 Iqra 2 42 72 35.00 45 62 43.55 39.27 Unsatisfactory 3 Saweera 2 41 49 50.20 48 32 90.00 70.10 Fair 4 Iqra 3 46 45 61.33 60 38 94.74 78.04 Fair 5 Tamana 2 45 40 67.50 56 40 84.00 75.75 Fair Maho Nari 6 Aimen 2 38 57 40.00 49 56 52.50 46.25 Satisfactory 7 Laiba 105.4 3 44 37 71.35 58 33 88.40 Good 5 8 Khais 3 42 49 51.43 58 48 72.50 61.96 Satisfactory 9 Sehar 3 45 51 52.94 59 46 76.96 64.95 Fair 10 Bisma 3 44 48 55.00 52 37 84.32 69.66 Fair 11 Rohail Khan 12 3 42 60 42.00 56 28.00 35.00 Unsatisfactory 0 12 M. Gaiz 2 48 43 66.98 57 40 85.50 76.24 Good 13 Ruman 3 44 40 66.00 25 30 50.00 58.00 Satisfactory 14 M. Tuheeb 2 45 30 90.00 27 40 40.50 65.25 Fair 15 M.Ali 2 50 50 60.00 44 50 52.80 56.40 Satisfactory 16 Rizwan Mirwas 2 50 50 60.00 60 50 72.00 66.00 Fair Ullah 17 Abu Bakr 3 45 35 77.14 53 55 57.82 67.48 Fair 18 Irshad 3 48 38 75.79 58 39 89.23 82.51 Good 19 Waheed 115.3 120.0 117.6 3 50 26 60 30 Excellent 8 0 9 20 Shayan 3 45 47 57.45 58 45 77.33 67.39 Fair 21 M.Ali Khan 2 50 50 60.00 27 40 40.50 50.25 Satisfactory 22 Wasif 3 50 45 66.67 60 40 90.00 78.33 Fair 23 M. Faizan 4 50 50 60.00 60 40 90.00 75.00 Fair 24 Kainat 4 45 38 71.05 60 42 85.71 78.38 Fair 25 Yar Muhamma Surkh 4 50 40 75.00 27 35 46.29 60.64 Satisfactory d Dheri 26 Ahsan 3 46 51 54.12 60 45 80.00 67.06 Fair 27 Sudais 3 45 50 54.00 60 45 80.00 67.00 Fair 28 Muskan 2 44 42 62.86 58 38 91.58 77.22 Good 29 Zakia 3 41 69 35.65 20 60 20.00 27.83 Unsatisfactory* 30 Hajra 4 40 55 43.64 60 41 87.80 65.72 Fair Average 67.49 Fair

USAID-funded Pakistan Reading Project 49

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent =99.72-117.69; Good = 81.75-99.72; Fair = 63.77-81.75; Satisfactory = 45.80-63.77; Unsatisfactory = 27.83-45.80

1.2.8 Changes Occurred As per the focal person’s observation, noticeable change was seen in parents’ and teachers attitude after this programme. Children improved their Urdu reading and developed story reading habits even at home. Parent- teacher communication was enhanced and regular meetings started taking place.

Principals, teachers and parents all were of the view that the CRP project had developed awareness among the community and they look forward to more such activities. They believed that children’s reading skills had improved to a satisfactory level after this intervention and teachers had become more skilled than before.

1.2.9 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of HDF during the grant period. This measurement captured advocacy as the active process through which HDF sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of HDF improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, HDF’s advocacy performance was measured.

Advocacy Index Score HDF achieved an overall score of 23.65 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of HDF’s overall score on the Advocacy Index across its eight dimensions.

Table: Advocacy Index Score of HDF Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3.5 Devising a strategy or action plan. 3 Collecting information about the issue like through research and talking to 2.4 stakeholders. Formulating a position or recommendations on the issue. 3 Securing sufficient resources for the initiative. 4.5 Building partnerships or coalitions with other interested groups. 3 Taking actions or activities to promote your position or recommendations on the 2 issue. Monitoring any changes resulting from your efforts to ensure that they are 2.25 implemented well. Overall Advocacy Index Score (Maximum Score = 40) 23.65

As shown in the table above, HDF was weakest in its capacity to “take actions or activities to promote your position or recommendations on the issue” with a score of 2. HDF was strongest in its capacity to “secure sufficient resource for the initiative” with a score of 4.5.

USAID-funded Pakistan Reading Project 50

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Findings Given the scope of HDF’s intervention was not advocacy, but instead focused on education, HDF’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. HDF itself is not an advocacy-focused civil society organization, having initiated only two interventions on advocacy prior to 2016. The overall score of 23.65 however is significant, and correlates that HDF successfully influenced political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades 1 and 2 was both timely and important within the specific geography of Khyber Pakhtunkhwa. HDF did gather both formal through a baseline and informal evidence through observation and experience that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, HDF also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, HDF demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for HDF to be more responsive to any challenges without the presence of a formal plan, such as dedicated more leadership tier staff for resolving strategic challenges in the area of interventions.

Moreover, while HDF communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects HDF’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer HDF towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue.

HDF demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. The MRC and the SMC were key means for both voluntary participation and partnership development. A key gap in its stakeholder management was with the media, which HDF did not mobilize or involve in the process of their project. Beyond press coverage for two formal events of the project, no media representatives participated in any of HDF’s activities. The gap between HDF’s activities and the media limits actions or activities to promote their position or recommendations to improve the reading skills and habits in the local community.

USAID-funded CRP grant initiatives improved the advocacy performance of HDF. Because of the technical requirements of the grant for evidence-led and planned activities, HDF devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, HDF also demonstrated an improved capacity to mobilize sufficient resources.

Success Stories Zainab from grade 1 of Government Girls Public School Mahonari got her story published in HDF-USAID’s book containing stories developed by different children from the 20 beneficiary schools.

Limitations The major limitation of this study was that some of the participants who had been a part of the training were unavailable. For example, the principal of Government Girls Public School Mahonari had been transferred to another school a few months after the CRP project had been implemented, although the new principal did have knowledge about the project.

USAID-funded Pakistan Reading Project 51

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Another limitation faced by APEX team was that at Government Public School Surkh Dheri, a few parents who were a part of the FGD were irrelevant, as they were not part of the sensitization sessions that were conducted during CRP intervention.

Conclusion The overall effectiveness of CRP project was perceived positively by the community. The project theme focused on parental engagement and improvement of Urdu reading skills in children, which was evident as per results of students’ evaluations. Establishment of school libraries with the facilitation of LEDs played a vital role in children’s regular attendance in schools. Parental engagement increased and communication between parents and teachers developed on a greater scale. However, participants requested for refresher courses and reimplementation of the project so other children and teachers could take advantage of the new techniques and strategies that were being discussed in the workshops as well.

1.3 AZAT Foundation (AF)

Background and Desk Review AF is a not for profit, non – governmental civil society organization. It was established on 17th February 2003. AF was registered under The Voluntary Social Welfare Agencies (Registration and Control) Ordinance, 1961 (Registration No. EDO-CD/30) on 2nd June, 2005. In 2008 AF also got registered under Societies Act 1860. AF is engaged in advocacy, campaigning, community mobilization, research, networking, challenging anti poor policies, plans and actions at local, national and international level with different coalitions. AF also works for development of unprivileged and less developed communities and disaster response in disaster prone areas to make sure all human exercise equal rights and get equal opportunities of development.

Objectives The USAID funded CRP project titled “Community Based Reading Improvement Initiative” was implemented in 20 schools of Tehsil and District Kharan of Balochistan. The project duration was of 12 months. The project mainly focused on family based, School based and Community based reading related activities. Project objective was to improve the reading skills of 1,000 children studying in 20 boys & girls primary schools through parental engagement.

1.3.1 Targets set vs. Achieved To achieve the above stated objective a number of activities were planned which included identification and selection of 20 schools and organizing community based orientation sessions in those schools, community based orientation sessions, parent-children story telling activities, provisioning of story books to households, decoration of reading corners in households, letter writing, decoration of class rooms and celebration of classroom decoration days and parent-teacher gatherings. All the activities were completed in time and involved parents and community.

1.3.2 Relevance Kharan is one of the most backward districts of Balochistan. The educational indicators in Kharan, including the class specific reading skills are quite poor. Primary schools don’t have any facilities like libraries and reading rooms. Keeping in view the existing conditions, most of the stakeholders agreed that the project was quite relevant to the needs of the people and addressed the felt needs.

USAID-funded Pakistan Reading Project 52

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

According the Deputy District Education Officer: “It was the first project of its kind in Kharan district. Since Kharan is a sensitive area, not many NGOs and INGOs had implemented any projects in the area. AF was the only organization working in the education sector in collaboration with education department.”

However, even their earlier projects had focused mostly on building infrastructure and providing facilities like computer labs to the school. He felt that the parents, being mostly uneducated, were not able to support children in addressing their education related project. Moreover, the interaction between the school and the community was also limited. The motivation and commitment of the teachers was always an issue.

The teachers were of the view that: “Project had contributed in creating an enabling environment in the schools for the children, which encouraged them to engage in reading skills enchantment activities. Earlier the environment in the schools and in the homes was not conducive for encouraging children to improve their reading skills.”

The schools had no libraries and reading material. The parents were mostly uneducated and unable to assist and support their children in their studies. The interaction between the schools and the community was limited and not very proactive. The project helped in improving coordination and engagement with the parents.

The principals of the schools were also of the view that: “Intervention was quite relevant and addressed the needs of the children and schools. The principals complained about the lack of facilities and the lack of support and cooperation from the department in providing those facilities.”

Many schools lacked basic facilities like washrooms and drinking water. So it was impossible for them to provide or establish reading facilities in the schools. They also informed that parents had limited interaction with the schools.

A Principal in Kharan said: “Have a look at the infrastructure. The main gate of the boundary wall is falling down. Even the newly constructed middle section has no washrooms and no water tank. In these circumstances, it is impossible for us to think of establishing reading rooms and school libraries”

According to the parents: “The most relevant aspect of the project was the establishment of the reading corners in the homes. They informed that usually in their homes they didn’t have reading culture and did not have specific reading corners for children.”

1.3.3 Effectiveness The activities carried out under the project proved quite effective and useful. The orientation sessions were received quite well by the parents and the teachers. These sessions helped the parents and teachers understand the objectives of the program and the activities to be carried out throughout the program. All the sessions were completed in time and attended by the desired number of participants.

USAID-funded Pakistan Reading Project 53

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Focal person AF said; “The focal person from the education department took keen interest in the review and development of the material that we felt that it was his project and we were just supporting him”

The material was appreciated and liked by the children and their parents. 20 schools were provided steel bookshelves and reading material for organizing different reading activities in the school. All the schools included in the survey had books shelves in them. However in some places the reading material was not available. It was said that it was almost a year since the end of the project, therefore most of the books had either been taken away by the children, who had not returned them and some of the books were worn out due to excessive use.

All the competitions and community based activities proved quite effective in increasing the interest of the parents and community in the project and encouraged the children and schools to take active part in them. The schools took keen interest in the reading room decoration competitions and even after a year, the rooms were still decorated. The competitions also helped in developing a story culture among the students. The initiative of decorating the reading corners in homes was well received in the community. The children were provided reading material and support for decorating the rooms. The activity was designed in such a manner that it did not put any financial burden on the parents. It was reported that in most of the houses the corners were still decorated and could be visited.

The representative of the education department also agreed that: “The project was successful in increasing interest of the children in enhancing their reading skills and increased parental engagement.”

However it was also pointed out that project could have proven even more successful if the teachers had also been given detailed training instead of just orientation about the project objectives and activities. This would result in increased capacity to help children in improving their skills.

Though the AF had decent advocacy capacity, the scope of the project did not have many advocacy activities. Therefore, AF did not regularly practice the development of formal strategic documents and did not develop an advocacy plan to address any unforeseen challenges. However, the linkages developed with the stakeholders were quite effective. Stakeholders like education department and the local government provided strong support to the grantee organizations.

1.3.4 Efficiency All the stakeholders agreed that most of the initiatives were quite efficient and quality of implementation was good. The orientation sessions were detailed and helped the participants clearly understand the project objectives and project activities. Sessions with the community helped in increasing ownership of the community and parents. These sessions increased parental engagement with the schools and reduced communication gap between school and the community.

The material developed and distributed was according to the social norms and was developed keeping in view the socio-cultural sensitivities. The materials were liked and appreciated by all stakeholders. The compilation and translation of the indigenous stories also generated interest of the local communities.

The different competitions and especially the reading room decoration competition among schools proved an effective and efficient way of involving and motivating the schools administration, teachers and students for enhancing their skills. In one school, inspired by the room decoration competition in primary section, the

USAID-funded Pakistan Reading Project 54

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

students of middle section had also decorated their class rooms with colorful charts. The competitions and story reading sessions encouraged the students to improve their skills.

Principal of Girl’s Primary School said: “It has been a year since the closure of the project, but you can still see the decorated room. We are a school in far flung area, even than we managed to win the third prize. It was very encouraging for my students and my teachers”

One of the most appreciated initiatives was the reading corner decoration in 1000 households. The material for decoration like charts was provided by the project and did not put any pressure on the family resources. The social mobilizers visited households to help them in decorating the corners. The decorated corners helped in developing a conducive environment at homes for reading. It was shared that the reading corners were still decorated and in use after one year.

The story-telling sessions and the sessions in the community were also very effective and efficient in increasing coordination among parents and children and reducing communication gap between them.

1.3.5 Sustainability Though most of the activities were very effective and helped in increasing parental engagement and enhancing the reading skills of the children, yet it was felt by all the stakeholders that sustainability was a challenge for some initiatives.

Though the project had helped in increasing the interaction between the communities and schools and increased parental engagement, yet the practice could not continue for long after the project. Some of the parents still visited the schools to discuss the issues of their children if required; there existed no mechanism for formal engagement of parents with schools. In the absence of follow up, it was feared that even the present level of engagement would diminish.

Principal of Boys Middle School Kharan said: “Parents often ask us that when we would invite them to the school again? But we are not able to hold the meetings as we do not have resources for organizing the meeting”

The project helped in enhancing reading skills of the students of grade I and II by engaging them in different activities with positive results and also provided reading material to the schools to continue the practice, yet, in the absence of follow up, these activities had stopped in majority of the schools visited during evaluation. The books given to the schools had either been lost or had been worn out and could not be used for long. It meant that students of the next classes would not be able to use the material which had been provided by the project and would not be able to improve their reading skills.

The project did not include capacity building of teachers to introduce them with strategies to help children improve their reading skills. The teachers attended orientation sessions in which they were introduced with the objectives of the project and with the activities which were supposed to be carried out under the project. They followed the orientations given to them in an effective manner and facilitated in conducting the activities smoothly, but when the project ended and the activities could not be continued due to financial constraints, they were not able to help the students in enhancing their skills through use of innovative strategies.

Focal person from Education Department said:

USAID-funded Pakistan Reading Project 55

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

“I think what we need most is the capacity building of the teachers. Most of the teachers are not very highly qualified and they have not received any formal training in teaching and specially in helping students in enhancing their reading skills. We need to build the capacity of our teachers”

1.3.6 Parental Engagement The project focused on increasing parental engagement, through orientation sessions, sensitization sessions and different community based events involving both the community and schools. Both the parents and the school administration reported increased parental engagement during and after the project. Parents also displayed greater interest in supporting their children in their studies and in improving their skills. However, the interaction between the schools and the parents had once again decreased after the project, though parents still made individual visits sometime to discuss the issues of their children, but there was no formal or systemized engagement between the two.

The sensitization sessions and community based events had also helped in making parents more aware on the importance of education and development of reading habit in early grades as foundation for future good or bad performance.

1.3.7 Students Evaluation Students from schools targeted by AF displayed an overall fair reading score of 63.82. Word reading and paragraph reading exercise results displayed similar accuracy and fluency, with students reading an average of 41 words correctly at an average rate of 62 words for both. The highest aggregate score achieved among the sample students was 131.88 words while the lowest was 3 words. 7 out of 30 students that participated in the evaluation displayed an unsatisfactory reading score of less than 28.76 words per minute, while 4 achieved excellent scores, higher than 106 words for AF. 6 students each fell into the good and satisfactory ranges, while 3 were in the fair range.

Table: Students Evaluation Results of AF

ords ords

Level S.N Student’s School of read o Name Name Class

Reading Achievement Time (sec) Time (sec) Time Aggregate Paragraph Paragraph readper min Correct w Correct Correct Words Words Correct Words Correct readper minute

1 Javed Ahmed 130.4 133.3 131.8 3 50 23 3 60 27 3 8 Excellent 2 Shakoor 4 10 60 10.00 6 60 6.00 8.00 Unsatisfactory 3 Abu Bakar 107.1 144.0 125.5 3 50 28 4 60 25 0 7 Excellent 4 Tayab Zaro Zai 103.4 116.1 109.7 3 50 29 5 60 31 3 9 Excellent 5 Idrees 3 50 42 71.43 60 47 76.60 74.01 Fair 6 Nasrulah 3 50 47 63.83 60 51 70.59 67.21 Fair 7 Mudassar Ahmed 4 50 50 60.00 44 60 44.00 52.00 Satisfactory 11 Shabina 3 2 60 2.00 1 15 4.00 3.00 Unsatisfactory 12 Sehar Gul 3 27 60 27.00 5 25 12.00 19.50 Unsatisfactory 13 Sabina Unsatisfactory Noroz Kallat 3 15 60 15.00 8 30 16.00 15.50 ** 14 Muskan 3 50 57 52.63 24 60 24.00 38.32 Satisfactory 15 Marwa 3 50 31 96.77 60 53 67.92 82.35 Good

USAID-funded Pakistan Reading Project 56

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

16 Nazla 107.1 120.0 113.5 3 50 28 4 60 30 0 7 Excellent 17 Maikan 3 50 58 51.72 24 60 24.00 37.86 Satisfactory 18 Hira Gul 3 43 60 43.00 19 60 19.00 31.00 Satisfactory 19 Kausar 3 50 55 54.55 60 57 63.16 58.85 Fair 20 Mareen Gul 2 26 60 26.00 10 30 20.00 23.00 Unsatisfactory 21 Yasir 103.4 112.5 107.9 3 50 29 5 60 32 0 7 Excellent 22 Gulzaman 109.0 101.4 3 50 32 93.75 60 33 9 2 Good 23 Zaheer 4 43 60 43.00 30 60 30.00 36.50 Satisfactory 24 Uzair 3 50 33 90.91 60 37 97.30 94.10 Good 25 Sudais 102.8 Shaikh 2 50 31 96.77 60 35 6 99.82 Good 26 Azmat Ullah Masjid 3 36 60 36.00 60 50 72.00 54.00 Satisfactory 27 Afaq Ahmed 3 36 60 36.00 11 40 16.50 26.25 Unsatisfactory 28 Sameer Unsatisfactory 2 24 60 24.00 15 40 22.50 23.25 ** 29 Khawaja Ahmed 100.0 2 50 30 0 60 42 85.71 92.86 Good 30 Abdul Wajid 103.4 3 50 29 5 60 41 87.80 95.63 Good Average 63.82 Fair Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent = 106.10-131.88; Good = 80.32-106.10; Fair = 54.54-80.32; Satisfactory = 28.76-54.54; Unsatisfactory = 3.00-28.76

1.3.8 Changes Occurred The project had contributed in motivating the children in improving their reading skills. The project had also helped in developing a reading culture among the students by providing them with different types of storybooks and encouraging them to exchange storybooks. The change was also observed at the household level, where parents, despite their own lack of education, had maintained the reading corners which were established and decorated during the project. This helped in creating an enabling environment at home to give constant encouragement to the children.

1.3.9 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of AF during the grant period. This measurement captured advocacy as the active process through which AF sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of AF improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, AF’s advocacy performance was measured.

Advocacy Index Score AF achieved an overall score of 24.16 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of AF’s overall score on the Advocacy Index across its eight dimensions:

USAID-funded Pakistan Reading Project 57

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Table: Advocacy Index Score of AF Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 4 Devising a strategy or action plan. 2.8 Collecting information about the issue like through research and talking to 2.6 stakeholders. Formulating a position or recommendations on the issue. 3.3 Securing sufficient resources for the initiative. 3 Building partnerships or coalitions with other interested groups. 3.5 Taking actions or activities to promote your position or recommendations 1.71 on the issue. Monitoring any changes resulting from your efforts to ensure that they are 3.25 implemented well. Overall Advocacy Index Score (Maximum Score = 40) 24.16 As shown in the table above, AF was weakest in its capacity to “take actions or activities to promote your position or recommendations on the issue” with a score of 1.71. AF was strongest in its capacity to “secure sufficient resource for the initiative” with a score of 4.

Findings Given the scope of AF’s intervention was not advocacy, but instead focused on education, AF’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. AF itself is not an advocacy-focused civil society organization, but advocacy is a crosscutting focus in many of their projects. They conducted six projects prior to 2016 of which advocacy was part of the activities. The overall score of 24.16 however is significant, and correlates that AF has the capacity to successfully influence political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II through parental engagement was both timely and important within the specific geography of Kharan. AF gathered largely informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, AF also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges.

Moreover, while AF communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects AF’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer AF towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue. Even though AF is a small organization of fewer than 10 people, it demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. Crucially, AF mobilized effective partnerships or coalitions with other interested groups, including the local government and community leaders. The MRC and the SMC were key means for both voluntary participation and partnership development. A key gap in its stakeholder management was with the media, which AF did not mobilize or involve in the process of their project. Beyond press coverage for one formal event of the project, no media representatives participated in any of AF’s

USAID-funded Pakistan Reading Project 58

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

activities. The gap between AF’s activities and the media limits actions or activities to promote their position or recommendations to improve the reading skills and habits in the local community.

While no Advocacy Index measurement of AF was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the CRP Grant Initiatives at least maintained the advocacy performance of AF. AF is already active in advocacy, and this grant was an opportunity to continue this work. Because of the technical requirements of the grant for evidence-led and planned activities, AF devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, AF also demonstrated an improved capacity to mobilize sufficient resources.

1.3.10 Success Stories The project had resulted in many success stories like a far flung school like GGPS Noroz Kallat winning a prize in the class room decoration competitions due to the interest of the children and dedication and support of the teachers. GBMS Sheikh Majid, is a stand out success story, where, inspired by the room decoration by the students of primary section for the competition, the students of middle section decided to decorate their class rooms on their own. Students developed beautiful color charts with illustrations, quotations, poems and stories and pasted them in their class rooms.

1.3.11 Limitations The study design included FGDs with parents. Most of the parents were quite poor and worked as daily wage laborers. They found it difficult to forgo their daily wages and come to the schools to participate in the FGDs. Due to unavailability of groups of the parents, only one FGD was possible. Even in that FGD, some of the parents were those who had not attended the sessions or events. But they knew about the project, as their children were studying in the school in which the activities were carried out. The DEO and the focal person of the department, who were a part of the material committee, had been transferred. Therefore the key informant interview was conducted with the current district education officer of Kharan. Many of the teachers who participated in the orientation sessions had also been transferred to other schools, which made the FGD with teachers a challenge and it was not possible to get the required number of teachers in one place.

1.3.12 Conclusion Overall the project has shown positive results and has been received well by the stakeholders. Analysis of the feedback from the stakeholders including parents, teachers, education department and community shows that the activities, carried out under the project, were effective and efficient. The positive indicators of the project were enhanced parental engagement and motivation of the students to improve their reading skills. The sustainability of the project was a question mark. Many of the initiatives started through the project had stopped. The reading material given to the schools could not be retained and reading skills enhancement activities had been discontinued. It would have been more effective if along with orienting teachers about the objectives and activities of the project, their capacity had also been built by introducing them with modern strategies for helping the students in improving their reading skills.

USAID-funded Pakistan Reading Project 59

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.4 Foresight Institute for Education and Leadership Development (FIELD)

Background and Desk Review The FIELD is a non-government organization registered under Section 42 of the Companies Ordinance in 2008. The mission of this organization is to work “for a society endeavoring to access to prosperous lifestyle and sustainable development”. For the last seven years, FIELD has been working specifically on “non-formal education for out-of-school children in marginalized communities, early childhood development, parental education for better parental role in educating their children, right to education of every child, enterprise development, youth capacity building and rehabilitation and disaster risk management activities”.

Over the course of its existence, FIELD has worked in Islamabad, Rawalpindi and Gilgit-Baltistan (GB) to implement various development activities. Its total outreach to date includes 728 teachers, 4512 students, 3605 community members, 214 CSO staff, and 341 School Management Committees.

Objectives Under the Complementary Reading Project (CRP) grant initiative, FIELD’s project goal was to achieve improved quality of reading skills of 1000 grade 1 and 2 students from 20 schools of District Skardu in GB. The project focused on improving the reading skills of children, engaging parents in enhancing the reading skills and involving parents and teachers in developing a reading culture at home.

1.4.1 Targets set vs. Achieved The FIELD team was able to accomplish the targets set in the grant application, which consisted of 12 milestones designed to meet the objectives. All the project milestones were achieved, including parent teacher gathering sessions, development of indigenous story books and school newsletters. Items such as carpets, cushions, bookshelves, suggestion boxes and chart displays were also implemented successfully.

1.4.2 Relevance The efforts made by FIELD were found to be largely relevant to the project objectives. Even though agencies such as AKDN, UNICEF and MAARIF had worked in the region, their focus had largely been restricted to teachers’ training and infrastructure development. The Deputy Director of Education, Skardu was of the opinion that most education related intervention focus on improving English language skills in students, leaving a gap in Urdu language proficiency. The CRP project was solely designed for Urdu reading, which was felt to be an area that greatly needed improvement and the activities carried out by the implementing organizations did contribute in improving the reading skills of the children. The current curriculum being implemented in Govt. schools does not fulfill all the needs of the community. The CRP intervention was thought to address the standards of quality and regional requirements of the area by the DEO, suggesting a high relevance of the project intervention in the district.

The project focal person in Skardu felt that: “Previously, no intervention had gone directly to the households for increasing parental engagement. With a high prevalence of poverty and illiteracy among parents of students studying in the covered schools, this aspect was thought to be addressing some very important needs that had a direct impact on increasing parental engagement with their children in educational matters.

USAID-funded Pakistan Reading Project 60

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

could not offer much help in the learning matters of their children, but the will to impart a quality education to their children existed, which was greatly boosted by the CRP initiative. Prior to the program, only some students discussed their educational matters with the parents, and to get them to cooperate with the school administration used to be a difficult task.

As for the school infrastructure, only one high school in Skardu had a library. The rest of the schools did not possess any reading or learning materials which could aid in developing children’s interest in Urdu reading. The provision of small libraries and vocabulary books to the schools as part of the CRP implementation was very relevant in addressing this need. Most of the schools in the area have been getting by on the most basic facilities, with some even lacking electricity and chairs for all the students.

The principals of the selected schools were of the view that the parents mostly belong to backward areas and felt that since they are uneducated, they have no responsibility in their children’s education. In the Boys Primary School at Sukamaidan it was noted that 80% of the children’s parents belonged to areas outside of the city, with the parents working in other people’s homes as domestic servants.

Principal of the sampled school: This aspect created problems in gaining access to the children’s parents and has a negative impact on the relevance and subsequent effectiveness of the intervention”

The head teacher of Sukamaidan told the research team. Otherwise, it was felt that the CRP initiative was a step in the right direction to increase the reading skills of primary school children and to increase parental engagement in improving their child’s reading skills.

Parents were largely of the opinion that there was some difficulty in reading for children before the program and not much awareness among parents regarding their children’s problems in improving reading. Home activities were mostly restricted to children doing homework without any parental supervision. The role of supervising the children’s education was relegated mostly to mothers, the majority of whom could not read to them. There was a general perception in society that Government schools have very low engagement between the teachers and the children, learning is restricted to the school boundaries and parents do not feel the need to be involved in the learning activities of school going children.

An enlightening aspect was revealed in the discussion with parents, which stated that they were facing problems which became apparent only after FIELD’s intervention through the awareness sessions. Most children are engaged in various educational activities throughout the day like going to the ‘madrasah’ after school, then some children goes attend tuitions and by the time they come home they are too exhausted to spend any time studying or reading.

A parent described his feelings as follows: “Before the CRP programme came to our area, we had no idea of involvement in school activities. Being uneducated, we felt we could not contribute in any way to our child’s education. But then the FIELD staff came to our homes and talked to us, which showed us how important it is for us to help our child in any way possible. Even if can’t help him/her to read, we can still provide a nice environment for him/her to study in.”

1.4.3 Effectiveness

USAID-funded Pakistan Reading Project 61

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The effectiveness of the CRP intervention in Skardu was determined to a great degree by its relevance- i.e. the need for such a program to raise awareness among parents in engaging with their children to improve reading skills.

The Deputy Director of Education in Skardu said: “Initiatives undertaken by NGO’s and other development organization generally generate more interest in the public, being considered more attractive compared to government sponsored projects, which are thought to have low ownership and lacks monitoring protocols”. He added, “NGOs also use alternative channels, such as community notables and ‘ulema’ to increase their traction in the communities, which contributes to the effectiveness of these initiatives”.

The most positive indicators contributing to the effectiveness of the CRP initiative from The DEO’s view point was that: “Increased enrollment in CRP schools, greater fluency in reading and an increase in parent’s trust in schools abilities to increase reading skills in children, reflected in the adoption of learning techniques advocated by the Complementary Reading Project.”

The focal person responsible for implementing the project in Skardu was of the view that positive effects were visible in school children immediately after the introduction of new techniques. A notable difference was felt in parent’s attitudes towards their children reading skills.

In terms of home-based activities, the introduction of story reading hours with the parents, Item labeling and the display of charts at home were thought to be the most effective in developing an environment conducive to increasing parental engagement with children. At school level, the establishment of small libraries was thought to be the most effective method, followed by indigenous story compilation, which generated an interest in the local culture and developed interest among the children in learning through reading. Furthermore, flash cards were seen to be an effective method in getting the children to physically engage in the classroom environment. It was evident in the discussion with teachers, that materials distribution in the form of stationary packets was helpful in generating parent’s interest towards the project.

The focal person for FIELD told the research team: “When we gave them the stationary packets to take home, it convinced the children’s parents that this was not a meaningless project and it was earnestly trying to help the children in improving their abilities.”

After the implementation of the CRP by FIELD in Skardu, teachers feel that more parents are willing to come to school to attend meeting which are held twice or thrice every month. Furthermore, parents also come on their own if they feel that their child is facing some difficulties to discuss it with the teachers. The provision of magazines, cultural story books and the vocabulary books provided under the CRP initiative helped create an interest among both students and teachers, by making them aware of their own culture and the outside world. Identification of students facing difficulty in reading has been made easier after the project, and such students are now said to be given more attention.

USAID-funded Pakistan Reading Project 62

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

the children’s reading skills. Despite being illiterate, the parents sought the help of teachers to discuss the child’s problems and accepted their responsibility in their child’s education. Profound effects in the home environment were expressed by parents, who had become more engaged in their child’s reading after they sat with them as he/she read story books, and this effect was being passed on to other children in the household who displayed more enthusiasm about reading and going to school.

1.4.4 Efficiency In terms of efficiency of implementation and effectiveness, the poverty and illiteracy of parents is believed to have been an obstacle which was further compounded by their villages being outside Skardu City. The program was financial efficient as it utilized no/low cost interventions and most of the resources used in implementation were locally present i.e. teachers, school buildings, etc.

The residence of many parents being outside the city meant that children who came to study at schools in Skardu could not engage effectively with their parents during school term. Many of these parents worked as domestic servants in households which were not too interested in the reading skills of these children; teachers were of the view that some schools in the city center are affected more by this phenomenon, while schools in the periphery of M.C. Skardu cater to largely indigenous communities.

From the teachers’ perspective, the school based activities had contributed more to improvement in children’s reading skills compared to home based activities as the teachers had received training and could actively engage in improving the student’s reading habits and skill. The school staff which had been part of the trainings felt that too much information had been offloaded in a short span of three days, in which most of the techniques imparted to the teachers, was completely new.

The teachers complained that: “It was very hard for us to keep attentive with extremely long duration of daily training, in which a lot of new information was being told to us, and then we had to go back and look at the manuals and training materials later, and used the DVDs provided to refresh our memories.”

Regarding level of books the opinion of teachers was divided, out of 11 teachers six were satisfied however, rest of the teachers felt that most of the books provided in the library were too advanced for the intended grades, and children had difficulty reading them. The books that were accessible to the children in grades I and II had been read many times since the end of the project and were no longer being effective in increasing the children’s reading skills. These teachers expressed a functional gap in the reading material that would enable the children in transitioning from basic to more advanced story books.

1.4.5 Sustainability The libraries provided by the grantee were being maintained by the schools and children were still issuing books from these libraries, which show the sustainability of the initiative. Furthermore, teachers were still implementing the techniques learnt during the course of the training and imparting these techniques to teachers who had not attended the trainings. The effects of the project could be seen even a year after its formal ending. Children’s motivation to increase reading skills, displays of charts in the classroom and parental willingness to attend school meetings were positive indicators pointing to the sustainability of the project.

USAID-funded Pakistan Reading Project 63

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

However, respondents mentioned that: “Additional books were needed to keep up the children’s interest which could only be provided by donors due to their design and careful consideration of words and images. The school libraries had not seen an increase in the number of books present.”

1.4.6 Parental Engagement The overall increase in parental engagement in their children’s reading was nominal, being restricted by their own socio-economic status and levels of literacy. Fathers still played a very minimal role in the educational matters of their children, leaving it to the mothers among whom very few were educated. Most of the parents had moved to Skardu from surrounding areas in order for their children to study in schools, which were usually the first generation that was being formally educated.

While the parents themselves could not participate in improving the children’s reading skills, they were still highly motivated for the children to improve and showed willingness to come to school meetings and discuss the progress of their children. They also spoke of spending more time with the children at home, which encouraged the children to read and tell the stories to parents.

1.4.7 Student Evaluation Test Students from schools targeted by FIELD displayed an overall satisfactory reading score of 41.43 words. Word reading exercise results displayed a higher accuracy than fluency, reading an average of 45 out of 50 words correctly at an average speed of 39 words. However, paragraph reading results were weaker, with students reading an average of 33 out 60 words correctly. The highest aggregate score achieved among the sample students was 92.05 words while the lowest was 11.5 words. 5 out of 12 students that participated in the evaluation displayed an unsatisfactory reading score of less than 27.6 words, while 3 managed to achieve excellent score, which was higher than 75.94 words for FIELD.

Table: Students Evaluation Results of FIELD

Level Student’s S.No Of

Name Class District per min per Achievement Time (sec) Time (sec) Time Aggregate per minute per School Name School Paragraph Reading Paragraph Correct words read words Correct Correct Words read read Words Correct read Words Correct

Unsatisfactory 1 Asfreen 2 45 120 22.50 15 60 15.00 18.75 * 2 Hajra 2 50 160 18.75 15 100 9.00 13.88 Unsatisfactory Unsatisfactory 3 Fatima 2 30 120 15.00 8 60 8.00 11.50 * 4 Rida Fatima 2 50 120 25.00 10 30 20.00 22.50 Unsatisfactory Konain 102.8 5 2 50 48 62.50 60 35 82.68 Excellent Fatima Hajigam Skardu 6 6 Hijab Zehra 2 50 45 66.67 60 50 72.00 69.33 Good 7 Aimen Zara 2 48 135 21.33 8 30 16.00 18.67 Unsatisfactory 8 Farial 2 45 75 36.00 55 90 36.67 36.33 Satisfactory Muqadis 109.0 9 2 50 40 75.00 60 33 92.05 Excellent Zehra 9 10 Abeeha 1 50 40 75.00 60 40 90.00 82.50 Excellent 11 Adeel Sukamida Skardu 1 30 120 15.00 5 30 10.00 12.50 Unsatisfactory

USAID-funded Pakistan Reading Project 64

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

12 Daniyal n 2 50 70 42.86 45 90 30.00 36.43 Satisfactory Average 41.43 Satisfactory Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Range: Excellent =75.94-92.05; Good = 59.83-75.94; Fair = 43.72-59.83; Satisfactory = 27.61-43.72; Unsatisfactory = 11.50-27.61

1.4.8 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of FIELD during the grant period. This measurement captured advocacy as the active process through which FIELD sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of FIELD improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, FIELD’s advocacy performance was measured.

Advocacy Score FIELD achieved an overall score of 23.56 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of FIELD’s overall score on the Advocacy Index across its eight dimensions:

Table: Advocacy Index Score of FIELD Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3.5 Devising a strategy or action plan. 3.2 Collecting information about the issue like through research and talking 2 to stakeholders. Formulating a position or recommendations on the issue. 3 Securing sufficient resources for the initiative. 4.5 Building partnerships or coalitions with other interested groups. 2.75 Taking actions or activities to promote your position or recommendations 1.86 on the issue. Monitoring any changes resulting from your efforts to ensure that they 2.75 are implemented well. Overall Advocacy Index Score (Maximum Score = 40) 23.56

Findings Given the scope of FIELD’s intervention was not advocacy, but instead focused on education, FIELD’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. FIELD itself is not an advocacy-focused civil society organization, having initiated only one intervention on advocacy prior to 2016. The overall score of 23.56 however is significant, and correlates that FIELD successfully influenced political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II was both timely and important within the specific geography of Gilgit Baltistan. FIELD did gather both formal and informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, FIELD also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor

USAID-funded Pakistan Reading Project 65

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, FIELD demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for FIELD to be more responsive to any challenges: it therefore delivered alternative activities (such as mediating community disagreements) without the presence of a formal plan.

Moreover, while FIELD communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects FIELD’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer FIELD towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue.

FIELD demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. Crucially, FIELD mobilized effective partnerships or coalitions with other interested groups, including the local government and community leaders. A key gap in its stakeholder management was with the media, which FIELD did not mobilize or involve in the process of their project. Beyond press coverage for one formal event of the project, no media representatives participated in any of FIELD’s activities.

While no Advocacy Index measurement of FIELD was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the Complementary Reading Project (CRP) Grant Initiatives improved the advocacy performance of FIELD. Because of the technical requirements of the grant for evidence-led and planned activities, FIELD devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, FIELD also demonstrated an improved capacity to mobilize sufficient resources.

1.4.9 Change Occurred Overall, respondents noted an improvement in children’s enthusiasm towards reading Urdu. The environment in class had also improved, with teacher’s noting more enthusiasm among the children, greater communication with the children and their problems regarding reading. An increased willingness among parents to cooperate with the school staff was also seen as a result of the CRP initiative, now those parents who come to school almost twice a month comparatively.

As a basic indicator of improvement, the results of the schools that were part of the programme have seen a notable improvement, especially in reducing the number of failures and higher enrollment rates. This was credited to the introduction of new techniques and classrooms activities, such as flash cards, “word typing”, charts etc. The improvement in results has also generated interest in other schools, from which teachers come to learn about the techniques.

1.4.10 Success Stories Parents generally noted a positive change in their children’s attitude towards reading. A parent stated that CRP had a positive impact on her home’s environment with regard to promoting a conducive environment for reading. “Observing her son’s growing enthusiasm for reading and his interest in reading stories at home, his younger sister who was only 3 years old had started to sit with him as he read her the stories from the library books he brought home. Even before she had started school, she was learning to read and speak Urdu with the help of

USAID-funded Pakistan Reading Project 66

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

her brother, who was in class 2. She had starting insisted us to get her books and copies so she could start practicing writing just like her elder brother. This really convinced us how joyful it can be for children to read.”

1.4.11 Limitations The major limitation of this study was the cessation of regular classes due to winter break. Although schools were operational, the examinations had ended and most of the children along with their parents had gone home to their native villages for the winter break. As a result, it was difficult for the grantee’s focal person and school administration to gather required number of people together that would provide a representative sample.

No teachers from B.P.S Astana participated in the teachers FGD. Similarly 12 students were assessed out of the 30 proposed samples. All parents that participated in the FGDs gathered at the G.P.S. Hajigam, which defeated the purpose of sampling parents from 3 different randomly sampled schools.

1.4.12 Conclusion Overall effectiveness of the CRP project filters to be somewhat positive as perceived by all the stakeholders’ vis-à-vis parents, teachers, children, principal and staff of the directorate of the education, though results of the test of the children show otherwise. In case of far flung and hard areas like GB project schools of this grantee, the project was able to increase parental engagement with their children on educational matters, involving the poor and illiterate parents to take interest in their children’s education, increased enrolment in CRP schools, greater fluency of children in reading and increase in parents trust in schools.

Sindh Rural Partners Organization (SRPO)

Background and Desk Review SRPO is women led non-profit, non-governmental organization. SRPO was registered in 2001 under Voluntary Organization’s Ordinance XLVI 1961. SRPO has been working for Poverty Alleviation, Rural Development and Women Empowerment through ensuring access to healthcare services, raising educational awareness, and lobbying & advocacy at national and provincial level. SRPO envisions a just, gender equitable, agriculturally rich, healthy, educated and prosperous rural society. The mission of SRPO is to promote, expand and strengthen social organizational base in Rural Sindh to alleviate poverty, develop human resource, empower rural women, better irrigation and livestock management, educational awareness, ensuring improved healthcare services, gender equity and equality, and promote environment friendly agricultural practices.

Objectives SRPO implemented CRP project titled, “Ensure parental engagement to increase enrolment of children in education system at primary level through sensitized stakeholders, better school governance, enhanced reading levels and better, safer and gender sensitive schools” in District Tando Allahyar. The duration of the project was nine months and it was implemented in four union councils of tehsil Jhando Mari. The objectives of the program were to advocate with stakeholders in education to ensure their engagement to enhance enrollment of girls by 25 % and retention of girls by 10 % in primary grades (1-2) and increase the interest of parents and students in education at 100 girls and boys schools through increasing the reading levels of 6,000 girls and boys students in English, and Urdu/Sindhi and making the school child friendly, gender sensitive, inclusive and safe by building the capacity of teachers. The project also focused on improving the management and governance in 100 schools by strengthening the SMC’s and education school supervisors.

USAID-funded Pakistan Reading Project 67

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.5.1 Targets set Vs. Achieved SRPO approved work plan included 25 different activities to achieve eleven milestones. These included training of 100 teachers and provision of 100 kits to these trained teachers, provision of 600 reading kits to the students, strengthening & activation of 100 SMCs through capacity building of 200 SMC members, formation of material review committees, designing, printing and distribution of 100 information charts, sensitization of education department & parents and publication of quarterly newsletters along with baseline and end line surveys.

According to the analysis of the results obtained through checklist, SRPO team was able to achieve the targets.

1.5.2 Relevance The project was implemented in four union councils of Tehsil Jhando Mari. Jhando Mari is a backward area and education of children especially girls is not a priority for most of the parents. Most of the stakeholders agreed that project was very relevant to the needs of the target areas. Poverty and lack of education among parents were cited as two main reasons behind parents’ lack of interest in the education of the girls and for limited interaction with schools.

The interaction between the parents and the school administration was quite limited before the project. Most of the teachers complained the parents did not take much interest in the education of the children. Once the child was enrolled in the school, the parents felt that it was no more their responsibility to support and help the child. The situation had improved quite significantly after the project as SMCs had started playing an effective role and some of the parents had started interacting with schools more regularly.

Reading skills of the children in the selected schools before the project were quite poor as the environment in the schools was also not very conducive for learning and increasing their reading skills. Schools had no libraries and no specific reading periods. Teachers also agreed that previously they were not using any special strategies to help the children improve their reading skills. The project had helped in increasing the interest of the students in enhancing their reading skills and marked improvement was observed.

The principals felt that project was quite relevant to the needs of the beneficiaries in the target area. In their opinion, the engagement of the parents with the schools was not up to the mark. SMCs were not playing an effective role. Many NGOs including SRPO itself had implemented projects in the target areas earlier, but none of them had focused on increasing parental engagement and enhancing the reading skills of the children. The earlier projects had focused on improving disaster preparedness among the children and the teachers and building infrastructure. Lack of capacity among teachers was also cited as one of the reasons behind weak reading skills of the children.

Principal of Dasori said: “We do not have trained teachers. Their own capacity is limited. How can they help the children? It is a blessing that PRP is organizing trainings for the teachers, so that they can enhance their skills.”

1.5.3 Effectiveness The analysis of the discussions held with the stakeholders through KIIs and FGDs and results of the checklist shows that intervention was quite effective and managed to achieve the targets and the activities were completed on time.

The coordination of SRPO with education department was satisfactory. The education department representative was part of the material review committee, which reviewed and finalized the manuals for the

USAID-funded Pakistan Reading Project 68

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

training of the teachers and SMC members. The education department also facilitated in nomination and selection of the teachers and SMC for the trainings. As a result 100 teachers and 200 SMC members were trained. The material review committee also reviewed and finalized the information charts which were printed and distributed in the schools. Sensitization sessions were also carried out with the officials of the education department to ensure their support and coordination.

The SMC members, some of whom were also the parents of the children, took keen interest in the affairs of the school and helped in strengthening of the infrastructure by completing the requirements of the drawing funds. The school principals were also appreciative of the role played by the SMC members. The training helped SMC members understand their role better.

SRPO has advocacy capacities that influence political or social change in its area of intervention. However, the scope of the project in terms of advocacy was limited. Therefore, apart from its interaction with education department to secure support and buy in for the project not many advocacy activities took place.

The interaction of parents with the schools had improved significantly and some of the parents were interacting more regularly with the school regarding the education of their children. The perception regarding the responsibility of the education of the children had also changed and parents now considered it a shared responsibility.

1.5.4 Efficiency All the initiatives of the project proved quite efficient. The teachers and the SMC members liked the content and quality of the trainings. Though some of the methodologies for enhancing the reading skills were new and strange for the teachers, who had not previously utilized such techniques, yet most of the teachers were utilizing the methodologies in their classes.

The training of the SMCs was also quite efficient as most of the SMCs became quite active and took interest in resolving the issues of the schools such as building of the boundary walls and other infrastructure projects. The content of the training and the quality was also appreciated by the SMC members.

The material designed for the project was appropriate and suitable. It included local stories like the story of the famous folk character from Tando Allahyar called Watayo Faqir. The students were already familiar with these stories and found it easier to relate with them.

The reading kits provided to the schools were also liked by the teachers and the students. It included story books and other reading material designed to improve the reading skills of the students. The material was colorful and according to the level and liking of the students. However at some places it was complained that material was not developed keeping in view the infrastructure of the schools. The kits included magnetic

USAID-funded Pakistan Reading Project 69

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

alphabets to help children in recognizing the alphabets, but the schools had no metal boards or even metal doors to stick those alphabets.

A Teacher from GPS Said: “There are no metal doors and metal boards in my school. Where am I going to stick these alphabets?”

The sensitization sessions with parents also proved successful as parents committed to support the education of their daughter and ensured their attendance.

The coordination with the education department was strong and productive. Education department took keen interest in the initiative and extended full cooperation to the project.

All the activities carried out were in line with the budget and work plans.

1.5.5 Sustainability The project had built the capacity of the teachers and equipped them with skills to help the students in improving their reading skills. Despite the fact that almost one year had passed since the completion of the project, these techniques were still being used by the teachers in the classrooms. The sensitization of the parents regarding the importance of girls’ education was also a step in the right direction.

Increased parental engagement is also important for the sustainability of the project. Since the parents were now more aware and more engaged, it was expected that it would lead to increased/sustained enrolment of the female student and increased interaction between the school and parents.

Training of SMC members led to increased involvement and participation of the SMC members in the affairs of the school. It was reported by head teachers that they had got support from the SMC members for the building of the infrastructure of the school. However, a few months back, the head teachers were asked to re- constitute the committees after conducting re-elections of the chairpersons. The new chairpersons had been elected, but had not yet been notified by the department, because of that that they had not got the signing authority yet. This meant that SMC committees and the head teachers were not able to withdraw funds from the accounts. Thus no work was being done for the last few months and SMCs had become somewhat dysfunctional.

100 schools were provided with reading kits, which included story books and other reading materials. It was assumed that these kits would remain in the schools and would be used for the later classes as well. However, unfortunately, in majority of the schools visited, the complete kits were not available. In those schools only few items from the kit were still available. It was reported that most of the items had been taken away by the students. This meant that next classes were without the support material required to improve the reading skills.

1.5.6 Parental Engagement The project focused on increasing parental engagement, through sensitization sessions and capacity building of the SMC members. Both the parents and the school administration reported increased parental engagement after the project. Most of the SMC members were parents of the children studying in the school. Now they took keen interest in the matters of school and provided support to head teachers in addressing their issues. The sensitization sessions had also helped in making parents more aware of the importance of girls’ education. However, all the stakeholders felt that parental engagement needed further improvement, as majority of the

USAID-funded Pakistan Reading Project 70

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

parents still didn’t interact with the school very regularly nor were they able to provide support to their children in improving their reading skills.

1.5.7 Student Evaluation Test Students from schools targeted by SRPO in Tando Allahyar displayed an overall fair reading score of 73.72 words per minute. Reading accuracy was similar in the paragraph and word reading exercise, with students reading an average of 44 correct words at a rate of 71 words per minute for the word exercise and 76 words per minute for the paragraph exercise, suggesting a higher fluency in paragraph reading. The highest aggregate score achieved was 147 words per minute, while the lowest was 9 words per minute. 13 out of 30 students assessed fell in the good range, which was between 91.80 - 119.40 words per minute for SRPO, while 9 students displayed unsatisfactory results of less than 36.60 words per minute.

Table: Students Evaluation Results of SRPO

Level S.N School Student’s Name of read o Name Class

Reading Achievement Time (sec) Time (sec) Time Aggregate Paragraph Paragraph readper min Correct words words Correct Correct Words Words Correct Words Correct readper minute

1 Unsatisfactory Farhan Ali 3 40 60 40.00 15 60 15.00 27.50 * 2 Ali Ahmed 3 30 60 30.00 20 60 20.00 25.00 Unsatisfactory 3 150.0 144.0 147.0 Ali Hassan 4 50 20 60 25 0 0 0 Excellent 4 Unsatisfactory Irfan Ali 4 10 60 10.00 8 60 8.00 9.00 * 5 Ehsan Ali Dasori 3 50 37 81.08 60 45 80.00 80.54 Fair 6 100.0 102.8 101.4 Mehtab 5 50 30 60 35 0 6 3 Good 7 107.1 120.0 113.5 Vishal 3 50 28 60 30 4 0 7 Good 8 Noman Ali 3 50 39 76.92 60 37 97.30 87.11 Fair 9 Dheero 3 50 50 60.00 60 48 75.00 67.50 Fair 10 Zahid Hussain 3 50 55 54.55 55 60 55.00 54.77 Satisfactory 11 Unsatisfactory Hamid 2 27 60 27.00 15 60 15.00 21.00 * 12 Khan 3 20 60 20.00 17 60 17.00 18.50 Muhammad Unsatisfactory 13 Ume Roman 2 10 60 10.00 8 60 8.00 9.00 Unsatisfactory 14 112.5 Zumra 3 50 37 81.08 60 32 96.79 0 Good 15 Saif ullah Qazi Noor 3 50 32 93.75 60 40 90.00 91.88 Good 16 Muhammad 100.0 116.1 108.0 Tayaba 3 50 30 60 31 0 3 6 Good 17 Unsatisfactory Kashif 3 15 60 15.00 10 60 10.00 12.50 * 18 Hamesh 3 32 60 32.00 25 60 25.00 28.50 Unsatisfactory 19 107.1 120.0 113.5 Irum 3 50 28 60 30 4 0 7 Good 20 Khameeso Khan 2 25 60 25.00 12 60 12.00 18.50 Unsatisfactory 21 Mashaekh 112.5 104.6 Zaman Ali 2 50 31 96.77 60 32 Hoti 0 4 Good

USAID-funded Pakistan Reading Project 71

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

22 107.1 116.1 111.6 Shahid Ali 2 50 28 60 31 4 3 4 Good 23 112.5 101.7 Rashid 3 50 33 90.91 60 32 0 0 Good 24 111.1 116.1 113.6 Amna Hassan 2 50 27 60 31 1 3 2 Good 25 102.8 Dua 2 50 33 90.91 60 35 96.88 6 Good 26 107.1 124.1 115.6 M. Umar 2 50 28 60 29 4 4 4 Good 27 Riaz 2 50 40 75.00 60 42 85.71 80.36 Fair 28 Maryam 2 50 39 76.92 60 40 90.00 83.46 Fair 29 Abu Bakar 2 50 47 63.83 60 52 69.23 66.53 Fair 30 120.0 105.4 Abera 2 50 33 90.91 60 30 0 5 Good Average 73.72 Fair Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent =119.40-147.00; Good = 91.80-119.40; Fair = 64.20-91.80; Satisfactory = 36.60-64.20; Unsatisfactory = 9.00-36.60

1.5.8 Changes Occurred The change had occurred at multiple levels. The trainings of the teachers had not only built their capacity but also motivated them to help their students in improving their skills. For a short time, the reading kits provided to schools also contributed in improving the environment and providing opportunities for the students to improve their skills, but unfortunately, could not be sustained for a longer period of time. The greatest change was increased interest and support of SMC members in the affairs of the school, which was a very positive sign and indicative of the ownership of the community.

A Teacher from Girls Primary School: “I managed to get many things done in the school with the help of SMC. This was a great help for me as SMC members were very supportive. But now I am handicapped because of the delay in notification of new SMC members and chairs. Once that is done, I shall immediately start some infrastructure improvement work.”

1.5.9 Advocacy USAID’s Advocacy Index was implemented to measure the advocacy performance of SRPO during the grant period. This measurement captured advocacy as the active process through which SRPO sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of SRPO improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, SRPO’s advocacy performance was measured.

Advocacy Index Score SRPO achieved an overall score of 23.41 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of SRPO’s overall score on the Advocacy Index across its eight dimensions:

USAID-funded Pakistan Reading Project 72

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Table: Advocacy Index Score of SRPO Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3.5 Devising a strategy or action plan. 2.8 Collecting information about the issue like through research and talking to 3 stakeholders. Formulating a position or recommendations on the issue. 3 Securing sufficient resources for the initiative. 3.75 Building partnerships or coalitions with other interested groups. 2.75 Taking actions or activities to promote your position or recommendations 1.86 on the issue. Monitoring any changes resulting from your efforts to ensure that they are 2.75 implemented well. Overall Advocacy Index Score(Maximum Score = 40) 23.41

As shown in the table above, SRPO was weakest in its capacity to “take actions or activities to promote your position or recommendations on the issue” with a score of 1.86. SRPO was strongest in its capacity to “secure sufficient resource for the initiative” with a score of 3.75.

Findings Given the scope of SRPO’s intervention was not advocacy, but instead focused on education, SRPO’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. SRPO itself is not an advocacy-focused civil society organization, having initiated no intervention on advocacy prior to or after 2016. Advocacy was however a crosscutting features of their ongoing work. A key example was their Norway-funded research and advocacy work on minority rights specifically focusing on the Promulgation of Hindu Marriage Act. The overall score of 23.41 is significant, and reflects that SRPO has overlapping advocacy capacities that influence political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II was both timely and important within the specific geography of Tando Allahyar. SRPO did gather both formal and informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, SRPO also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, SRPO demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for SRPO to be more responsive to any challenges, even if not accounted for in earlier planning.

Moreover, while SRPO communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects SRPO’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer SRPO towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue.

SRPO demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. Crucially, SRPO mobilized

USAID-funded Pakistan Reading Project 73

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

effective partnerships or coalitions with other interested groups, including the local government and community leaders. The MRC and the SMC were key means for both voluntary participation and partnership development. A key gap in its stakeholder management was with the media, which SRPO did not mobilize or involve in the process of their project. No press coverage or participation of media representatives took place during any of SRPO’s activities. The gap between SRPO’s activities and the media limits actions or activities to promote their position or recommendations to improve the reading skills and habits in the local community. While no Advocacy Index measurement of SRPO was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the CRP grant initiative improved the advocacy performance of SRPO. Because of the technical requirements of the grant for evidence-led and planned activities, SRPO devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, SRPO also demonstrated an improved capacity to mobilize sufficient resources.

1.5.10 Success Stories Mashaekh Hoti is one of the success stories of the project. The motivation and passion of the teachers for improving the skills of the students, the use of a combination of old and new strategies for improving the reading skills and the enthusiastic response of parents and guardians of the children were all very encouraging. It was also among the schools, where the complete reading kits provided by the project were still available and being used.

Principal said that: “We have excellent coordination with parents. This school is like home for most of them as they have also studied in this school. They can walk in any time and discuss the issues of the children with us. After the project, the interaction has increased even more.”

1.5.11 Limitations The study design included FGDs with parents. The consultant was informed by the local coordinators that most of the parents were quite poor and worked as daily wage laborers and found it very difficult to forgo their daily wages and come to the schools to participate in the FGDs. Due to unavailability of groups of the parents, only one FGD was possible. Even in that FGD, some of the participants were not parents of the children. They were uncles or elder brothers, who had not participated in the orientation sessions and meetings. Therefore they did not know about the programs. On the days of the visit, the key officials of the education department, who had been involved in the implementation of the project was not available as they were busy in an event organized by the education department in Hyderabad. Thus the KII with EDO was not possible. In Sindh, Biometric system of attendance has been introduced. Therefore, the teachers were reluctant to leave their duty stations and come to participate in the FGD. The FGD had less than the required number of teachers available.

1.5.12 Conclusions Overall the project had been effective and has managed to largely achieve its objectives. The stakeholders’ feedback analysis shows positive impact. The important achievements of the project include increased capacity of the teachers and strengthening of SMCs. The strengthening of SMCs has resulted in their increased ownership. Provision of reading kits to 6000 children also yielded positive results. However, the efforts to make the school environment more friendly and supportive by provision for kits and charts, though useful and important, could not be sustained for a very long time.

USAID-funded Pakistan Reading Project 74

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Root Work Foundation (RWF)

Background and Desk Review Root Work Foundation (RWF) is not for-Profit, non-governmental civil society organization registered under the Societies Registration Act 1860. The mission of the organization is to strive for the development and empowerment of less developed and marginalized groups of society. RWF is a multiple focus organization and has implemented different programs to address the poverty linked issues of disadvantaged groups. The core programs of RWF include Education, Food Security and Sustainable Livelihoods, Human Rights and Gender Justice, Community Infrastructure, Disaster risk reduction and Climate Change.

The organization was established in 2002 by a group of volunteers with an aim of creating opportunities and spaces for marginalized and less developed groups of the society. RWF has successfully implemented a number of projects in ten districts of Sindh province & eleven districts of Balochistan province.

Objectives Under CRP grants RWF implemented the project titled “Supporting Visually Impaired Children for Enhancing Reading Skills” in Sajawal District of Sindh Province. The main objectives of the program were to Identify and support visually impaired children of grades 1 to 5, to train teachers on visual screening of impaired children and supporting visually impaired children through classroom support and inclusive education and to provide vision screening kits to selected 100 schools.

To achieve the above mentioned objectives RWF carried out a number of activities. These included coordination meetings with the education department & signing of MOU, project launching ceremony, identification of 200 schools, selection and training of 100 teachers, orientation sessions with the parents, screening of children, provision of glasses to children to support them in reading and project closing ceremony.

1.6.1 Targets set vs. Achieved To achieve the above mentioned objectives RWF carried out a number of activities. These included coordination meetings with the education department & signing of MOU, project launching ceremony, identification of 200 schools, selection and training of 100 teachers, orientation sessions with the parents, screening of children, provision of glasses to children to support them in reading and project closing ceremony. RWP team was expected to organize 17 different types of activities to achieve the targets. According to the analysis of the results, most of the targets were achieved and activities were completed in time. Shortcomings were noticed only in a couple of areas. Though the public information messages were developed and aired on local FM channels, yet some of beneficiaries were not aware of the messages or had not heard them on local radio, which showed that the radio campaign had not reached all the beneficiaries.

1.6.2 Relevance According to all the stakeholders, there are a number of factors, which make the project quite relevant to the needs of the children and parents. These factors include the socio-economic status of the parents, low levels of information, lack of awareness among parents on visual disability issues in children, lack of capacity among teachers and low level of nutrition among the children.

10830 children from 200 schools were screened and tested and 314 children were provided glasses. Thus the project managed to achieve its stated objectives and targets.

USAID-funded Pakistan Reading Project 75

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

It was felt that the project was relevant to the needs of the children. Previously, teachers were not aware of such issues among their students. Since most of the teachers were working in single room schools with multi- grade teaching system, they had to deal with a large number of students from different classes simultaneously. This made it difficult for them to identify such issues among children. However, after the training, they were able to identify and help the children easily. When they received the training, they were able to identify many children with visual disabilities in their classes and refer them for further testing and glasses. The coordination between the schools and parents was also quite weak earlier and there was little or no communication between the two. The parents being uneducated themselves, were not able to identify the issues of the children and discuss their issues with the teachers.

A teacher from Government Boys Primary School said: “The parents are mostly uneducated themselves. They cannot read. How can they identify the reading issues of their children? They think that their responsibility had ended once they have sent their children to schools.”

The head teachers also felt that such projects were needed keeping in the view the prevalent poverty in the area and the lack of education among the parents. The parents, being poor and uneducated felt that their responsibility ended after sending the children to school. They were not able to identify the issues of their children and this project helped them. Thus the project addressed a need, which was previously overlooked. However, it was also stated that such project should continue and such camps should be organized every year. Thought the teachers felt that sessions with the parents were useful, yet the interaction with the parents had not improved much after the intervention and the teachers and principals still complained that parents were not taking enough interest in the studies of their children and there was lack of coordination among the parents and the schools regarding the problems being faced by the children in studies.

1.6.3 Effectiveness According to the Tehsil Education Officer, the project was effective as it built the capacity of 100 teachers to screen the children and identify their issues at an early stage. The teachers would continue to do so in their schools. In the beginning there were apprehensions that teachers would not be able to learn the skill of screening and testing in such a short time, but the training proved quite effective and useful. The screening of more than 10,000 children was also significant achievement.

Focal Person from Education Department said: “We thought that only three days’ training would not sufficiently equip the teachers to carry out the screening as it is a specialized job. However, when we looked the contents of the manual and met the facilitators, we were convinced that it could be done.”

Teachers felt that, the capacity building had helped them in understanding the issues of the students in a better way. The training was effective and it had given them good understanding of the structure of the eye, different eye sight problems of the children and of the methodology of screening and testing the children. They were provided the kits for screening the children, which they were using even then. The understanding of the parents about the issues had also improved.

The sessions with the parents were also considered quite useful. The sessions had helped in increasing interaction between schools and the parents and increased awareness about the issues of the children. The media messages for raising awareness had mixed effects as some of the participants had not heard the messages on the media.

USAID-funded Pakistan Reading Project 76

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The most positive effect was observed on the children, who had received glasses through the project. Most of the children, who received glasses, were able to read easily with the help of the glasses. When they were asked to read without glasses, many of them were not able to do so easily. Many of the children were finding it difficult to read things written on the board by the teachers, but with the help of the glasses, they were able to overcome that difficulty. A Student from Main Primary Sindhi School Chandio said: “My main problem was reading form the board. When the teacher wrote something on the board, I was not able to read it, but now I can easily read what the teacher is writing on the board with the help of these glasses.”

1.6.4 Efficiency The project managed to overachieve the target set in the beginning and the resources were used efficiently. According to Tehsil Education officer, the department was involved from the beginning and department representative was part of the material review committee. Efforts were made to make material easy to comprehend for non-technical persons. The organization also worked closely with education department to identify the schools and to get enrolment lists. The department also facilitated in selection of the teachers for training and in their participation in the training. The relationship remained smooth throughout the project. The teachers agreed that the quality of the training was very good. The content was simple and according to the level and understanding of the participants. The facilitators were experts in their field and were able to discuss quite technical topics in simple manner making it easier to comprehend. The kits given to the teachers for screening the kits were easy to use and of good quality.

The orientation sessions were also appreciated by teachers and parents. They were organized at school level and helped in increasing awareness about the problem of visual impairment. The parents acknowledged that previously they did not understand the problems of their children and these sessions were useful and informative of them.

1.6.5 Sustainability According to stakeholders, the capacity building of the teachers was the most important factor contributing in the sustainability of the project. Though the project had a very limited focus and focused mainly on screening of the children and provision of the glasses, which were one time activities, yet the training of the teachers and provision of the screening kits would mean that teachers would be able to identify the children and screen them for any problems. The teachers also reported increased capacity to identify the issues of the children and to address them more effectively.

The awareness raising sessions with parents also increased understanding of the issues of the children with visual impairment. The increased awareness among the entire stakeholder about the importance of the issues and the increased interaction between parents and schools are going to be important for the sustainability of the project.

1.6.6 Children with Special Needs The primary focus of the project was the needs of the visually impaired children. Most of the children, who were provided with glasses, were from poor families. Their parents were uneducated. It would have been very difficult for the parents even to identify the reading issues of their children and buy them glasses. The teachers also did not have the capacity to identify & help the children with special needs. Therefore, the project, served a very specific purpose. It is expected that after the training and provision of screening kits to the schools, the practice would continue and the cases of children, who needs support would be identified earlier.

USAID-funded Pakistan Reading Project 77

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

The project engaged parents only for increasing their awareness. RWF team conducted awareness sessions with the parents of visually impaired children. The sessions were focused on the need of supporting visually impaired children, providing proper care of them and specific role of parents in supporting these children. Overall 60 sessions were conducted in three rounds with parents. No meetings between the parents and school administration were organized as these were not part of the design and proposal of the project. However, these sessions were conducted in schools and did contribute in enhancing engagement of parents with the school. The parents visited the schools to discuss if their children are facing any problems, but there was no mechanism for regular interaction between the school administration and parents.

1.6.7 Student Evaluation Test Student evaluation tests were carried out with children who had been provided glasses after the screening. The evaluation focused on two aspects: the ability to read the provided material and the ability to read with or without glasses. Overall 14 children from three schools were tested. Some the children faced problems while reading without glasses, while other reported that their main issue was reading from the class board which is usually at a distance of 5 to 10 feet from the students and they faced problems in reading there. These problems were addressed by the vision glasses. This shows that provision of vision glasses had indeed contributed in addressing their issues. Reading skills of the students from two schools was categorized as fair and from one school as good. All the selected schools in the target areas were Sindhi medium. They were tested for their Sindhi reading skills.

Table: Students Evaluation Results of RWF

Level S.N Student’s School Name Of read o. Name Class Achievement Time (sec) Time (sec) Time Aggregate readper min Correct words words Correct Correct Words Words Correct Words Correct readper minute Paragraph Reading Paragraph

1 Muhammad Sahib Dino Irfan Pussio 3 50 40 75.00 60 45 80.00 77.50 Fair 2 Ajmeer Ali GBPS Tahirani 5 50 35 85.71 60 45 80.00 82.86 Good 3 Dilbar Ali 2 50 40 75.00 60 50 72.00 73.50 Fair 4 Sabeeha 5 50 54 55.56 60 55 65.45 60.51 Fair 5 Main Primary 120.0 105.0 Amna Sindhi School 4 50 25 0 60 40 90.00 0 Excellent 6 Shahzad Chnadio 100.0 Memon 5 50 30 0 60 40 90.00 95.00 Good 7 Arshad Ali 4 50 45 66.67 60 50 72.00 69.33 Fair 8 Unsatisfactor Shahbaz 4 15 60 15.00 10 60 10.00 12.50 y 9 102.8 Laksh 4 50 40 75.00 60 35 6 88.93 Good 10 Allah Dino 5 50 43 69.77 60 45 80.00 74.88 Fair 11 Umama 5 50 48 62.50 60 51 70.59 66.54 Fair 12 Farooq Ahmed 5 50 41 73.17 60 52 69.23 71.20 Fair 13 130.4 120.0 125.2 Ashfaq 4 50 23 3 60 30 0 2 Excellent 14 103.4 150.0 126.7 Zulqarnain 5 50 29 5 60 28 0 2 Excellent

USAID-funded Pakistan Reading Project 78

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Average 80.69 Fair Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent =103.88-126.72; Good = 81.03-103.88; Fair = 58.19-81.03; Satisfactory = 35.34-58.19; Unsatisfactory = 12.50-35.34

1.6.8 Change Occurred The main change caused by the project was the increased awareness and realization of the existence of the problem of disability and the issues of the children with special needs. The entire stakeholder agreed that before this project, the parents and the teachers were not aware of the problems of the children with visual impairment and did not have the capacity to address them. Despite the fact that several organizations had implemented projects in the target area, this was the first project of its kind.

The increased capacity of the teachers had also changed their attitudes towards the children who faced difficulties in reading. Teachers reported that instead of punishing them, they tried to find out the reasons behind the inability to read and to address them.

1.6.9 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of RWF during the grant period. This measurement captured advocacy as the active process through which RWF sought to influence political or social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of RWF improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, RWF’s advocacy performance was measured.

Advocacy Index Score RWF achieved an overall score of 23.67 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of RWF’s overall score on the Advocacy Index across its eight dimensions:

Table: Advocacy Index Score of RWF Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3.5 Devising a strategy or action plan. 3 Collecting information about the issue like through research and talking to 3 stakeholders. Formulating a position or recommendations on the issue. 3 Securing sufficient resources for the initiative. 3.25 Building partnerships or coalitions with other interested groups. 2.5 Taking actions or activities to promote your position or recommendations on 2.42 the issue. Monitoring any changes resulting from your efforts to ensure that they are 3 implemented well. Overall Advocacy Index Score(Maximum Score = 40) 23.67

As shown in the table above, RWF was weakest in its capacity to “take actions or activities to promote your position or recommendations on the issue” with a score of 2.42. RWF was strongest in its capacity to “ensure the issue is timely and important” with a score of 3.5.

USAID-funded Pakistan Reading Project 79

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Findings Given the scope of RWF’s intervention was not advocacy, but instead focused on education, RWF’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. RWF itself is not an advocacy-focused civil society organization, having initiated no “pure advocacy” interventions prior to or after 2016. RWF however cited projects that featured advocacy, including work with UNICEF that involved lobbying the government and engagement with PDMA and district government in a consortium led project on livelihoods in disaster areas. The overall score of 23.67 is significant, and correlates that RWF has the advocacy capacity to influence political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades I and II in the context of visual impairment was both timely and important within the specific geography of Sajawal. RWF relied on formal from CRP and informal evidence from its own ongoing work that the issue is a vital concern to the local constituents, including community members, local influencers and local decision-makers. In fulfillment of their grant requirements, RWF also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, RWF demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for RWF to be more responsive to any challenges: it therefore delivered alternative activities (such as mediating community disagreements) without the presence of a formal plan.

Moreover, while RWF communicated and implemented activities upon a clear position on reading skills and habits, it did not have any dedicated documents – such as a position paper or a policy brief – that presented its positions or recommendations in a formal manner. This negatively affects RWF’s agenda-setting capacity around the issue, given any turnover of current staff or change in funding priorities may steer RWF towards a different position or restrict them from effectively or regularly communicating to decision-makers on this specific issue.

RWF demonstrated sufficient mobilization of both financial and human resources. It secured and expended the grant fund, and also mobilized voluntary participation required for the project. Crucially, RWF mobilized effective partnerships or coalitions with other interested groups, including the local government and community leaders. The MRC and the SMC were key means for both voluntary participation and partnership development. RWF also worked closely with the media, not only undertaking print media coverage formal events of the project but also involving media representatives into its activities. RWF engaged local radio: messages were broadcasted on radio to raise awareness about the issues of disability and visual impairment in the children. This accounts for RWF’s higher score of 2.42 on “taking actions or activities to promote your position or recommendations on the issue” than other CSOs receiving the CRP grants.

While no Advocacy Index measurement of RWF was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the USAID-funded CRP grant initiatives reinforced the advocacy performance of RWF. Because of the technical requirements of the grant for evidence-led and planned activities, RWF devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, RWF also demonstrated an improved capacity to mobilize sufficient resources.

1.6.10 Limitations The study design included FGDs with parents. The consultant was informed that most of the parents were quite poor and worked as daily wage laborers and found it very difficult to forgo their daily wages and come to

USAID-funded Pakistan Reading Project 80

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

the schools to participate in the FGDs. Due to unavailability of groups of the parents, FGDs could not be conducted and only Key Informant Interviews were possible with them. One of the schools selected in the sample had closed down due to retirement of both teachers. Another school was chosen in its place. The focal person of the grantee failed to help the visiting team for data collection in making the teachers and parents available. Though, he was informed about the visit schedule through all means of possible communication well in time.

1.6.11 Conclusion The project had a focused intervention. The objective was to identify and screen the children with visual impairment and provide the identified children with vision glasses in order to help them in their reading. The project was successful in achieving all its targets. The increased capacity of the teachers to identify and screen the children also meant that this practice would continue in future as well and they would be able to help the students in coming years as well. However, it was also noticed that some of the children who were provided glasses were not using them. There was need of introducing a stronger follow up mechanism to ensure the children realize the importance of using glasses consistently to avoid further deterioration and to ensure proper utilization of resources provided through the project.

Karakoram Associates for Educational and Economic Development (KAEED)

Background and Desk Review Karakoram Associates for Educational and Economic Development (KAEED) is a not for profit non-government organization established in 2010. It has been working in the mountain communities of Pakistan, which are vulnerable to natural disasters, political conflicts, extreme poverty and perpetual human suffering. Lack of awareness, knowledge, skills and resources inhibit these communities to improve their condition.

KAEED has been working to improve the living conditions of these mountain communities by working in the thematic areas of social capital development, inclusivity and sustainability.

As part of the CRP Grants Initiatives, KAEED implemented the Reading Enhancement through Inclusiveness (RETI) project in District Ghizer with a special focus on increasing reading skills in children with special needs.

Objectives The overall project goal was to enhance reading abilities in school children from classes 1-5, with a special focus on dyslexic and visually impaired children. The project was implemented in three Union Councils of District Ghizer, Gilgit-Baltistan and covered 40 Government and Basic Education Community Schools (BECS). The specific project focused on creating awareness among parents and school management committee members about dyslexia, building capacity of head teachers and teachers for creating inclusive environment for children for improving their reading skills by providing teaching and learning aids to schools and providing adaptive aids.

Targets set Vs. Achieved A total of 12 milestones were set by KAEED to meet the objectives of the RETI project in the 3 UCs that came under the project coverage area in District Ghizer. All these milestones were implemented successfully, while some activities achieved greater success than was set in the targets.

USAID-funded Pakistan Reading Project 81

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.7.1 Relevance All the activities were in accordance with the overall objectives of the project. The need for such activities was greatly felt among all groups of people included in the study. Education was not considered a priority in the region before the implementation of the KAEED project, working in tandem with the CRP. Previously, development projects in the area had been carried out with AUSAID funding through PPAF in the MIED programme, which focused on the hard aspects of education- school facilities, training material etc.

The parents were of the opinion that project had increased the parental engagement with schools whereas prior to the implementation of this project, parental engagement with the school was non-existent. They were of the view that “our parents never took an interest in our studies, and neither did we.” Rather, children talking to the parents about learning issues were considered a nuisance, who scolded them by saying “if you want to study, do it on your own and do not disturb us”.

Some parents reported: “Prior to the sensitization and awareness workshops, the situation in the area was just like ancient times- fathers went to do labor work during the day and came home tired, smoking cigarettes and drinking tea. Women were busy in the household duties while children were left to play around in the mud in the school playground.”

School principals were of the view that even though most parents are illiterate, they still play an important role in the education of their children by providing encouragement and a conducive environment to study at home. This project raised awareness among the parents regarding this aspect, which was a critical need in the region. There were some villages in Kuchdeh inhabited by ‘gujar’ nomads, who had never sent the children to school before.

For the teachers, the most relevant aspect of the project was learning about differential treatment towards children having fewer abilities than others, or suffering from specific problems. They felt that earlier their skill was not sufficient to be able to understand and address the various problems faced by students. A teacher said, “Before the project, all the children sat together and punishment was the basic method relied upon to address the students who seemed to be inattentive.” Teachers mostly work in the BECS schools on a voluntary basis, and had previously received no formal training in educational techniques other than what had been taught to them during university education.

According to the principal of BECS Kuchdeh: “This project was the first time they were taught how to use phonetics to teach Urdu, as well as the identification of dyslexic children, which was something they never knew existed before.”

Unavailability of appropriate books for children, with pictures and themes to increase their interest was also cited as a problematic area. It was generally perceived among teachers that parent did not see primary school education as a priority area, only getting interested in the child’s education as he approached the middle or high school.

Prior to the program, parental involvement in schools was almost non-existent. No interaction with parents was visible, who sent the children to school because of l fees. He added that BEC Schools, which did not have any fees, showed even lesser involvement, and only 2-3% of such schools had SMCs in which parents were active, while Government schools displayed less than 20% active SMCs. The project helped not only in enhancing parental engagement but also helped in improving the effectiveness of SMCs.

USAID-funded Pakistan Reading Project 82

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.7.2 Effectiveness In terms of effectiveness of the project, school principals were of the view that most of the project goals were met with a positive impact in the schools and the community. Learning about modern pedagogical techniques, including differential treatment of differently abled students, had changed the environment in the schools and its effect could be seen even among parents.

According to the DEO representative who took part in this study, the health-related and psychological aspects of education had never been touched upon by any interventions previously. Interaction between parents and the schools had increased considerably since the implementation of KAEED’s project in Ghizer.

Principals and teachers said: “Parents did not bother coming to school even during the annual result ceremony. However, after the project majority of the parents have started taken an interest, attending the bi-monthly meetings held in the school to discuss the learning progress of children. Parents also stop to talk to school teachers if they cross paths outside school and ask about the educational progress of their children.” Initially, KAEED had planned to implement the project with a 20-30% predicted prevalence of visual impairment among the children, however, during the course of the project this was changed to provide visual aids to 100% of the children identified with visual impairment during the screening test.

The introduction of vocabulary building was a highly effective tool to enhance reading among children according to teachers. It enabled the children to greatly increase their vocabulary in a challenging and competitive manner, allowing them to move from various learning grades in highly systematic manner which could be monitored and followed up by the teachers and parents.

In the baseline survey only 10 dyslexic students were identified. After the course of the project, where the teachers were also trained to identify dyslexia among children, a total of 173 children suffering from dyslexia had been identified and were taught according to the pedagogical techniques learnt in the trainings. A teacher of the school in Chatorkhand said, “Such (dyslexic) students were asked to come sit in front of the class and paired with a better performing student who would engage him in the lessons being taught in the classroom.” Furthermore, a total of 143 children were given visual aids after being screened for visual impairment. Some very serious ailments were also identified among certain students, who were then sent to Gilgit for further treatment. As a result of the project, almost 70% of the targeted schools have prepared detailed lists of differently abled children, the nature of their problems and the special attention and treatment they require during lessons, which are now displayed in the classrooms for the aid of current and new teachers.

Increasing parent’s awareness of their role in children’s learning was a crucial aspect of the project. The role of SMC was emphasized in awareness sessions with parents and trainings for the SMC members, who were imparted an understanding of the SMC as a legally sanctioned body with specific roles, responsibilities and jurisdictions. Teachers were found to be taking a more proactive role in the children’s environment, school attendance and interest in the classrooms.

USAID-funded Pakistan Reading Project 83

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Most parents said that: “Although we cannot directly participate in the reading activities of our child, we follow up on his/her progress with the teachers and sit with the children as they read stories at home. Most houses hardly had more than one room, so all the family was together in the evenings around the fireplace, where the story reading activity had become an important and anticipated activity.”

This resulted in increasing the children’s enthusiasm for reading and wanting to enhance their skills. Parents said that when the children read late into the night’s they would keep them awake also, asking them to get tea or nuts and fruit for them.

A significant impact was felt in the parent’s attitudes towards primary education - they stated: “While we had earlier relied on punishment and strictness as a means to establish authority and discipline in the household, now we stress on the use of gentleness and love towards the younger generations.”

Parent’s felt that their perception of education as an extravagance had changed, and now understood that education and learning dominates all aspects of modern life, being necessary even for gainful employment as a laborer. Many parents had designated educated family members such as older children, cousins or relatives to assist the children in reading activities.

While the decoration of model rooms and reading corners in the household could not be implemented due to the economic situation of the region, it was felt that the provision of visual aids contributed positively towards the families’ living conditions, as these issues might never have been diagnosed otherwise.

Children’s performance in the classroom and examinations had improved significantly. Teachers said that children had now become more open with them, replacing the earlier norm of fear and intimidation. Children were also reported to be spending more time in reading activities, with the dedicated reading period being the most looked forward to time of the school day. Many children continued to stay in school even after the classes had ended, reading the books from the library and coming to borrow more books even during the winter and summer holidays.

1.7.3 Efficiency The hard aspect of project outputs such as provision of books, visual aids, training manuals and administration of screening tests was reported to have been performed with a high degree of efficiency. All activities were managed within the allocated budget, and no burden was reported to have been placed on the inhabitants. A short duration of implementation was the most prevalent complaint from almost all participants in the study.

Teachers said, “Single 5-day training was not sufficient to absorb the large amounts of new techniques and strategies that were being discussed in the workshops and that at least 10-15 days of training over an extended period of time would have been needed to fully understand the techniques that were discussed in the training.”

USAID-funded Pakistan Reading Project 84

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Furthermore, they felt that the trainings were missing practical demonstration of the techniques and strategies that were talked about, which would have been very useful for the teachers who were completely unfamiliar with the concepts being introduced through the project.

KAEED staff opined, “A lot of time was wasted on administrative matters such as procuring NOCs from the state department, and negotiating bureaucratic red tape with the government as well as donor agencies cost a lot of time which could have been used towards the implementation of the project outputs.”

The fact that KAEED personnel were known in the area and already familiar with the communities contributed to the efficiency of the project by decreasing the distrust and alienation usually associated with interventions by various agencies in the region. All project staff were locals of the district and could communicate easily with the target populations.

1.7.4 Sustainability According to the Directorate of Education representative interviewed in the study, a strong exit strategy was the greatest advantage KAEEDs initiative had over other development initiatives that had taken place in the district. As a result of the exit strategy, which relied more on the capacity building of school staff, parents and communities, the activities started by the IP could be maintained even after the duration of the project had ended.

All the schools visited still had functioning libraries in the form of steel cases in which the big books were kept and could be issued by the students. Although the number of books present had not increased, they were still being read by students, even those who had not participated in the project. SMCs were said to be engaging with the communities, but it was the teachers who were seen to be taken a proactive role in engaging with the parents to participate in the learning process of the children.

The sustainability of the RETI project was said to be integrated into the project design by decreasing the reliance on donor inputs, focusing on the role of local communities and teachers to implement the strategies formulated as a result of the project. However, it was stated by study participants that increase in number of books could only be possible by donor support. Teachers were cooperating with each other to disseminate the learning’s regarding inclusivity and treatment of differently abled children. Ownership of the project in the community and schools also contributed positively to its sustainability.

1.7.5 Children with Special Needs The health and psychological aspects of children’s learning processes and its associated issues were a completely novel idea in the region. The introduction of the concept of Dyslexia had a profound effect on teacher’s attitudes towards the learning abilities of students. As a result, they implemented techniques learnt in the training workshops to address differently abled children.

The strategies learnt as a result of KAEED’s project were seen as a significant development towards more inclusive pedagogies from traditional teaching methods which relied heavily upon an authoritarian approach towards education both in the school and at home.

Screening of schoolchildren and the provision of eye and sunglasses to children with visual impairment was also an area that had not been addressed before, and highly appropriate for the region which is considerably remote from advanced medical facilities and afflicted with poverty.

USAID-funded Pakistan Reading Project 85

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

1.7.6 Student Evaluation Test Students from schools targeted by KAEED displayed an overall satisfactory reading score of 22.60 words per minute. Word reading exercise results showed students reading an average of 31 out of 50 words correctly at an average speed of 23.63 words per minute. However, only 9 out of 25 students were able to read more than 50 words correctly in the paragraph reading exercise, while the remaining students could read only 15 words or less in the 60 word paragraph administered. The highest aggregate score achieved among the sample students was 82.5 words per minute while the lowest was 2.25 words per minute. 15 out of 25 students that participated in the evaluation displayed an unsatisfactory reading score of less than 18.3 words per minute, while only 1 managed to achieve excellent score, which was higher than 66.45 words per minute for KAEED.

Table: Students Evaluation Results of KAEED

S.N Student’s School Aggregat Level read o Name Name Class e Achievement Reading Time (sec) Time (sec) Time Paragraph Paragraph readper min Correct words words Correct Correct Words Words Correct Words Correct readper minute

1 Zulfia 2 6 80 4.50 0 60 0.00 2.25 Unsatisfactory* 2 Atiqa 12 22.0 10.0 2 44 10 60 16.00 Unsatisfactory* 0 0 0 3 Sambree 12 23.5 12 28.5 2 47 57 26.00 Satisfactory n 0 0 0 0 4 Molia 12 15.0 2 30 6 60 6.00 10.50 Unsatisfactory* Madad 0 0 Chatorkhan 5 Habiba 47.0 57.0 d 2 47 60 57 60 52.00 Good 0 0 6 Sufian 13.3 11.0 2 20 90 11 60 12.17 Unsatisfactory* 3 0 7 Fehmida 56.6 53.5 2 50 53 58 65 55.07 Good 0 4 8 Reshma 75.0 90.0 2 50 40 60 40 82.50 Excellent Abbas 0 0 9 Usman 33.3 20.0 1 50 90 10 30 26.67 Satisfactory 3 0 10 Neelum 13.3 2 6 27 4 32 7.50 10.42 Unsatisfactory 3 11 Sher 10.0 2 10 60 0 60 0.00 5.00 Unsatisfactory* Zaman 0 12 Shafi 12 10.0 10.0 2 20 5 30 10.00 Unsatisfactory Abdullah 0 0 0 13 Haider 1 5 45 6.67 0 60 0.00 3.33 Unsatisfactory* Kuchdeh 14 Aakash 14.0 2 7 30 5 33 9.09 11.55 Unsatisfactory 0 15 Faiz 2 4 45 5.33 0 60 0.00 2.67 Unsatisfactory* 16 Suat Ali 12 2 14 7.00 0 60 0.00 3.50 Unsatisfactory* 0 17 Fahim 12 14.0 2 28 6 60 6.00 10.00 Unsatisfactory* 0 0 18 Bibi Nafia 12 15.0 2 30 5 60 5.00 10.00 Unsatisfactory* 0 0 19 M. Zakria 51.7 64.3 Gaut Gupis 3 50 58 59 55 58.04 Good 2 6

USAID-funded Pakistan Reading Project 86

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

20 Tasneem 45.2 39.3 2 49 65 59 90 42.28 Fair Bano 3 3 21 Basit 12 20.5 10.0 2 41 10 60 15.25 Unsatisfactory* 0 0 0 22 Fatiya 12 23.0 12 28.5 2 46 57 25.75 Satisfactory 0 0 0 0 23 Yausira 11 24.3 10 33.7 3 47 59 29.01 Satisfactory 6 1 5 1 24 A ali 18.2 10 34.8 3 28 92 58 26.53 Satisfactory Shah 6 0 0 25 Nahida 12 22.0 15.0 3 46 15 60 18.54 Unsatisfactory* 5 8 0 Average 22.60 Satisfactory Computational Parameters *: Results of Word and Paragraph reading exercises have been constrained to within 20% tolerance margins **: Time durations have been allowed a tolerance margin of +/- 5 seconds Key: Range: Excellent =66.45-82.50; Good = 50.40-66.45; Fair = 34.35-50.40; Satisfactory = 18.30- 34.35; Unsatisfactory = 2.25-18.30

1.7.7 Change Occurred The most apparent change was observed in the attitude of teachers with regard to an increased understanding of inclusiveness and treatment of differently abled children. Teacher’s engagement with parents was also seen to have increased as a result of KAEEDs efforts. Parent’s attitudes towards their children’s education were also changed to some extent, although its impact on enhancement of children’s reading skills is debatable.

Teachers and parents both reported, “We have seen an increase in children’s motivation towards reading specifically and education in general, with the children showing more enthusiasm in going to school.” Furthermore, the environment of homes and classrooms was reported to be significantly more conducive for children to read and study in, with both teachers and a parent playing more supportive roles in the children’s learning processes. Effects of the project intervention were visible in improvement in the overall results of targeted schools.

1.7.8 Advocacy Index USAID’s Advocacy Index was implemented to measure the advocacy performance of KAEED during the grant period. This measurement captured advocacy as the active process through which KAEED sought to influence political and social change. Beyond the measurement, a key area of interest analytically was whether the advocacy performance of KAEED improved as a result of the grant, both during the course of implementing its grant and after. Based on USAID’s Advocacy Index, KAEED’s advocacy performance was measured .

Advocacy Index Score KAEED achieved an overall score of 24.75 out of a maximum score of 40 along the eight advocacy dimensions. The following is the breakdown of KAEED’s overall score on the Advocacy Index across its eight dimensions:

USAID-funded Pakistan Reading Project 87

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Table: Advocacy Index Score of KAEED Advocacy Index Score Advocacy Dimension 1 (Lowest) to 5 (Highest) Ensuring the issue is timely and important. 3 Devising a strategy or action plan. 3 Collecting information about the issue like through research and talking to 2 stakeholders. Formulating a position or recommendations on the issue. 3 Securing sufficient resources for the initiative. 4 Building partnerships or coalitions with other interested groups. 3.5 Taking actions or activities to promote your position or recommendations on 3 the issue. Monitoring any changes resulting from your efforts to ensure that they are 3.25 implemented well. Overall Advocacy Index Score (Maximum Score = 40) 24.75

As shown in the table above, KAEED was weakest in its capacity to “collect information about the issue like through research and talking to stakeholders” with a score of 2. KAEED was strongest in its capacity to “secure sufficient resource for the initiative” with a score of 4.

Findings Given the scope of KAEED’s intervention was not advocacy, but instead focused on education, KAEED’s advocacy performance did not significantly score across all eight dimensions of the Advocacy Index. KAEED itself is not an advocacy-focused civil society organization, having initiated only one intervention on advocacy led by the British Council prior to 2016. The overall score of 24.75 however is significant, and correlates that KAEED demonstrated adequate capacity to influence political or social change in its area of intervention.

The issue to improve the reading skills and habits of learners in Grades 1 and 2 was both timely and important within the specific geography of Ghizer. KAEED did gather both formal and informal evidence that the issue is a vital concern to the local constituents, including community members, local influencers and local decision- makers. In fulfillment of their grant requirements, KAEED also had strategic considerations and action plans for its project. It however did not regularly practice the development of formal strategic documents, nor did it have any alternative plans in case the intervention faced any challenges. Notwithstanding this, KAEED demonstrated sufficient monitoring of its activities and the changes resulting from these activities. This allowed for KAEED to be more responsive to any challenges without the presence of a formal plan.

KAEED communicated and implemented activities upon a clear position on reading skills and habits. Beyond the grant’s immediate scope of work, KAEED also had dedicated position documents – such as a charter of demands as part of the British Council work – that presented its positions or recommendations in a formal manner. This positively affected KAEED’s agenda-setting capacity around the issue in the local area: KAEED had already successfully reformed a series of local laws as a result of their activities, and both their staff and their stakeholders were familiar and receptive to advocacy efforts.

KAEED demonstrated sufficient mobilization of both financial and human resources, despite their grant being reduced in duration due to the processing time involved for official permission to proceed with the project. KAEED secured and expended the grant fund, and also mobilized voluntary participation required for the project. A distinctly attribute of this voluntary participation was the inclusion of local minorities – the nomadic

USAID-funded Pakistan Reading Project 88

Evaluation of Complementary Reading Project (CRP) Grants Initiatives

Gujar population – into the activities of the grant. KAEED also mobilized effective partnerships or coalitions with other interested groups, including the local government and community leaders. The MRC and the SMC were key means for both voluntary participation and partnership development. A key gap in its stakeholder management was with the media, which KAEED did not mobilize or involve in the process of their project. Beyond press coverage for one formal event of the project, no media representatives participated in any of KAEED’s activities. The gap between KAEED’s activities and the media limits actions or activities to promote their position or recommendations to improve the reading skills and habits in the local community. It needs to be observed however that the media landscape in Ghizer is not one the envisaged by the Advocacy Index, which therefore introduces a bias against CSOs that work with community and interpersonal media rather than mass media (such as print and electronic media).

While no Advocacy Index measurement of KAEED was implemented prior to the start of the project in 2016, there is sufficient evidence to conclude that the USAID-funded CRP grant initiatives improved the advocacy performance of KAEED. Because of the technical requirements of the grant for evidence-led and planned activities, KAEED devised action plans, collected information from research and stakeholders, regularly monitored activities, and developed and implemented an approach on the reading skills and habits of learners. By virtue of the grant, KAEED also demonstrated an improved capacity to mobilize sufficient resources.

1.7.9 Success Stories The project had considerable effects on changing certain communities’ perceptions regarding education of children.

One such community was the ‘gujars’, who practice a nomadic way of life, seasonally moving with their herds in search of pastures. They had never sent their children to school, and were known in the region as being completely illiterate. KAEED project staff approached these communities and convinced them to attend the awareness and sensitization sessions. Participating in these sessions had a profound effect on the community members, who not only immediately enrolled their children in school but also acted as voluntary ambassadors for the duration of the project, going from house to house to convince people to attend the planned sessions. A similar situation was seen in the village of Bathret, where there was 0% girls’ enrollment prior to the start of the project. Through the efforts of KAEED project staff in community mobilization, 35 girls had enrolled in the school at Bathret. This was done through the advocacy of community notables, who were also a part of the SMC, to raise awareness among community members regarding the importance of girl’s education and changing perceived notions of girls’ roles in the community.

1.7.10 Limitations No hurdles were faced by the research team in the area during the course of the study.

1.7.11 Conclusions Ownership of the project by the targeted communities, as part of the KAEEDs exit strategy, contributed to increasing the project’s effectiveness in terms of efficiency and sustainability

Awareness among teachers regarding inclusion, differently abled children and specific problems such as Dyslexia and visual impairment had a considerable effect on their attitudes, resulting in the adoption of new techniques learnt during the training and contributing to a more conducive learning environment.

USAID-funded Pakistan Reading Project 89