GEORGIA PRIMARY EDUCATION PROJECT BASELINE/IMPACT ASSESSMENT REPORT Contract No. AID-114-C-09-00003 USAID COR: Medea Kakachia Chief of Party: Nancy Parks June 2013 This publication was produced for review by the United States Agency for International Development. It was prepared by Chemonics International Inc. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 1 Table of Contents The Context ......................................................................................................................................................................... 3 Purpose of Impact Baseline Study ............................................................................................................................. 3 Study Design ....................................................................................................................................................................... 3 Pilot and Control Group School Sampling and Selection Process ................................................................. 7 Selection of Students for Baseline Study ............................................................................................................. 10 Test Administration Manuals ................................................................................................................................... 10 Development of Reading and Math Diagnostic Assessment Tools ........................................................... 10 Data Collection ................................................................................................................................................................ 11 Baseline Study Quality Control ................................................................................................................................ 12 Data Entry ......................................................................................................................................................................... 20 Annex 1: Sampling Strategy ..................................................................................................................................... 21 Annex 2. Test Administration Guidelines ............................................................................................................34 2 The Context The USAID/Georgia Primary Education Project (G-PriEd) is a five-year project designed to provide comprehensive assistance to the primary education system to improve reading and math competencies of Georgian and ethnic minority students. This will be achieved through supporting instructional improvements, using diagnostic testing, improving standards, using technology, training teachers, and building the capacity of subject experts in reading and math in schools. G-PriEd aims to support the introduction of innovative teaching methodologies through intensive primary reading and math training sessions followed by the organization of school-based Teacher Learning Circles in each school that offer continuous peer support. The program strives to promote the development of skills and active-learning strategies for teaching reading and math, by aligning existing pedagogical practice with research-based best practices, for a student population with a wide range of abilities and backgrounds. Purpose of Impact Baseline Study The purpose of the impact study is to assess the progress of student learning outcomes in Grades 1-6 reading and math. The impact evaluation study uses the data collected from the baseline assessment to set the baseline values for student learning outcomes in reading and math. Setting baseline target values is an important first step of the impact assessment as the values are the basis for assessing project performance and impact. Baseline targets are vital for monitoring progress toward established goals over time and for evaluating the extent to which the project achieves results at the end of the implementation period. The key evaluation question of the impact study is: To what extent have the students’ learning outcomes in reading and math in Grades 1-6 changed/improved as compared to the baseline values? Baseline values will be set after analysis of the baseline information on student’s learning outcomes in reading and math in grades 1-6 has been completed. The data from the follow-up assessment will be compared to the baseline and the progress towards planned results will be measured as compared baseline values. G-PriEd worked on the design of impact assessment methodology which included several interrelated activities, such as: • Sampling strategy development: design of the sampling strategy including the selection of the pilot schools in accordance with the sampling strategy and selection of control schools; • Reading and Math Tests-evaluation tools development: creation of two variations of reading and math test forms for baseline data collection; creation of test forms for GSL in reading and math for ethnic minority schools; • Test Administration Manual development: developing three sub-modules to the methodology, including general administration procedures and subject (reading and math) procedures; finalization of the methodology in parallel with the actual baseline impact assessment field work. • Baseline Impact Assessment: procurement of the subcontractor, field administration work, baseline test observation, creation of the online software for data entry. Study Design The impact evaluation study of the G-PriEd project is a quasi-experimental, longitudinal study. The causal impact of G-PriEd’s intervention on its target population involves repeated observations of the same variables over several years. The G-PriEd impact evaluation examines changes in outcome that are directly attributable to the program through the use of a counter-factual. A counter- factual is “what would have happened if the intervention had not occurred” and this is provided by a “control” group which does not receive the “treatment” or “intervention.” The impact evaluation will assess the degree to which changes in students learning outcomes can be attributed to the project interventions. 3 A causal relation between the impact achieved by G-PriEd interventions (teacher professional development, provision of supplemental materials, and increased parental engagement) and the increase in students’ learning outcomes of math and reading can be tested by comparing the experience of a group of students affected by the project vs. the (counter-factual) experience of the group of students who were not affected by the project. This design considers two aspects: 1) Study of 1-6 grade students in the experimental schools (where the project activities will be implemented, i.e., G-PriEd pilot schools) and other groups of schools in the same districts and with similar conditions where the project interventions will not be implemented; 2) Study of the experimental and non-experimental (control) groups by the end of the pilot period after two years of project implementation phase; The first aspect considers that approximately 19,000 students in Georgian language and ethnic minority schools will be included in the experimental group (project implementation activities) in the pilot year, February 2013- September 2014. The second aspect considers the differences of results in all groups over the time of the project implementation. In this regard, the study conducted in February-April 2013 with the treatment and control groups will be considered a baseline study; the following study in Feb-March-2015 will be considered as a follow-up study. Important principles of the impact evaluation to be considered are: 1. Since there is a period of time between baseline and end-line measure, both the “treatment” and “control” groups must have on-going monitoring to detect any possible outside influences other than the intervention (contamination effect, such as another donor starting a reading or math program) that could influence student reading/math scores. This will possibly affect the scores in both the treatment and control groups. Thus monitoring both “treatment” and “control” groups might be considered for possible contamination effects. 2. Monitoring for spill-over effects should be considered as well. Spill-over effects are, for example in the project, when a teacher receives training and materials in one of the “treatment schools” and shares their training experiences and materials with one or more teachers in a “control” school. This will possibly affect the outcomes in the control schools. Sampling Strategy for Selecting Pilot Schools In order to select an appropriate and meaningful distribution of schools to make up the initial cohort, a decision was made to focus on students as the unit of emphasis. Therefore, the sampling population was defined at the first stage as the grade 1-6 students in Georgian and ethnic minority public schools. This student-focused rationale reflects the overall goal of the quality improvement initiative as measured in terms of impact on student learning. The student-focused rationale also organizes strategic resources in ways that reflect the demographic distribution of children in Georgia and targets interventions in proportion to school-aged populations. The sampling process started in parallel with identifying the pilot schools in which G-PriEd would implement its pilot. An initial estimation of