Georgia Primary Education Project

GEORGIA PRIMARY EDUCATION PROJECT

BASELINE/IMPACT ASSESSMENT REPORT

Contract No. AID-114-C-09-00003 USAID COR: Medea Kakachia Chief of Party: Nancy Parks

June 2013

This publication was produced for review by the United States Agency for International Development. It was prepared by Chemonics International Inc. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

Table of Contents The Context ...... 3 Purpose of Impact Baseline Study ...... 3 Study Design ...... 3 Pilot and Control Group School Sampling and Selection Process ...... 7 Selection of Students for Baseline Study ...... 10 Test Administration Manuals ...... 10 Development of Reading and Math Diagnostic Assessment Tools ...... 10 Data Collection ...... 11 Baseline Study Quality Control ...... 12 Data Entry ...... 20 Annex 1: Sampling Strategy ...... 21 Annex 2. Test Administration Guidelines ...... 34

The Context The USAID/Georgia Primary Education Project (G-PriEd) is a five-year project designed to provide comprehensive assistance to the primary education system to improve reading and math competencies of Georgian and ethnic minority students. This will be achieved through supporting instructional improvements, using diagnostic testing, improving standards, using technology, training teachers, and building the capacity of subject experts in reading and math in schools. G-PriEd aims to support the introduction of innovative teaching methodologies through intensive primary reading and math training sessions followed by the organization of school-based Teacher Learning Circles in each school that offer continuous peer support. The program strives to promote the development of skills and active-learning strategies for teaching reading and math, by aligning existing pedagogical practice with research-based best practices, for a student population with a wide range of abilities and backgrounds.

Purpose of Impact Baseline Study The purpose of the impact study is to assess the progress of student learning outcomes in Grades 1-6 reading and math. The impact evaluation study uses the data collected from the baseline assessment to set the baseline values for student learning outcomes in reading and math. Setting baseline target values is an important first step of the impact assessment as the values are the basis for assessing project performance and impact. Baseline targets are vital for monitoring progress toward established goals over time and for evaluating the extent to which the project achieves results at the end of the implementation period. The key evaluation question of the impact study is: To what extent have the students’ learning outcomes in reading and math in Grades 1-6 changed/improved as compared to the baseline values? Baseline values will be set after analysis of the baseline information on student’s learning outcomes in reading and math in grades 1-6 has been completed. The data from the follow-up assessment will be compared to the baseline and the progress towards planned results will be measured as compared baseline values. G-PriEd worked on the design of impact assessment methodology which included several interrelated activities, such as: • Sampling strategy development: design of the sampling strategy including the selection of the pilot schools in accordance with the sampling strategy and selection of control schools; • Reading and Math Tests-evaluation tools development: creation of two variations of reading and math test forms for baseline data collection; creation of test forms for GSL in reading and math for ethnic minority schools; • Test Administration Manual development: developing three sub-modules to the methodology, including general administration procedures and subject (reading and math) procedures; finalization of the methodology in parallel with the actual baseline impact assessment field work. • Baseline Impact Assessment: procurement of the subcontractor, field administration work, baseline test observation, creation of the online software for data entry.

Study Design The impact evaluation study of the G-PriEd project is a quasi-experimental, longitudinal study. The causal impact of G-PriEd’s intervention on its target population involves repeated observations of the same variables over several years. The G-PriEd impact evaluation examines changes in outcome that are directly attributable to the program through the use of a counter-factual. A counter- factual is “what would have happened if the intervention had not occurred” and this is provided by a “control” group which does not receive the “treatment” or “intervention.” The impact evaluation will assess the degree to which changes in students learning outcomes can be attributed to the project interventions.

A causal relation between the impact achieved by G-PriEd interventions (teacher professional development, provision of supplemental materials, and increased parental engagement) and the increase in students’ learning outcomes of math and reading can be tested by comparing the experience of a group of students affected by the project vs. the (counter-factual) experience of the group of students who were not affected by the project. This design considers two aspects: 1) Study of 1-6 grade students in the experimental schools (where the project activities will be implemented, i.e., G-PriEd pilot schools) and other groups of schools in the same districts and with similar conditions where the project interventions will not be implemented; 2) Study of the experimental and non-experimental (control) groups by the end of the pilot period after two years of project implementation phase; The first aspect considers that approximately 19,000 students in Georgian language and ethnic minority schools will be included in the experimental group (project implementation activities) in the pilot year, February 2013- September 2014. The second aspect considers the differences of results in all groups over the time of the project implementation. In this regard, the study conducted in February-April 2013 with the treatment and control groups will be considered a baseline study; the following study in Feb-March-2015 will be considered as a follow-up study. Important principles of the impact evaluation to be considered are: 1. Since there is a period of time between baseline and end-line measure, both the “treatment” and “control” groups must have on-going monitoring to detect any possible outside influences other than the intervention (contamination effect, such as another donor starting a reading or math program) that could influence student reading/math scores. This will possibly affect the scores in both the treatment and control groups. Thus monitoring both “treatment” and “control” groups might be considered for possible contamination effects. 2. Monitoring for spill-over effects should be considered as well. Spill-over effects are, for example in the project, when a teacher receives training and materials in one of the “treatment schools” and shares their training experiences and materials with one or more teachers in a “control” school. This will possibly affect the outcomes in the control schools.

Sampling Strategy for Selecting Pilot Schools In order to select an appropriate and meaningful distribution of schools to make up the initial cohort, a decision was made to focus on students as the unit of emphasis. Therefore, the sampling population was defined at the first stage as the grade 1-6 students in Georgian and ethnic minority public schools. This student-focused rationale reflects the overall goal of the quality improvement initiative as measured in terms of impact on student learning. The student-focused rationale also organizes strategic resources in ways that reflect the demographic distribution of children in Georgia and targets interventions in proportion to school-aged populations. The sampling process started in parallel with identifying the pilot schools in which G-PriEd would implement its pilot. An initial estimation of G-PriEd resources allowed working with approximately 13,000 students in approximately 110-120 schools; however, later information supplied by the Ministry of Education and Science (MES) indicated that approximately 19,000 students were enrolled in the target schools. These 19,000 students in grades 1-6 comprise the study population. A multi-stage proportionate random sampling strategy was employed. 1. Geographic clusters 11 clusters (10 regions + Abkhazeti) 2. Types of schools in each cluster by the language of instruction: a. Georgian b. ethnic minority

3. Types of schools in each cluster by the school size: a. Small, 1-99 students b. Medium-size, 300-599 students c. Large schools: over 600 students 4. Grades a. First grade b. Second grade c. Third grade d. Fourth Grade e. Fifth grade f. Sixth grade With the multi-stage sampling, 45 blocks of schools were identified with a homogenous student population. We applied the first proportion for the proportionate random sample to estimate the number of (homogenous) students and (homogenous) schools in each block to be included in the G-PriEd activities as the following: • First, the number of 1-6 students from each block to be included in the pilot was identified proportionally to the proportion of the size of this block to the national data of student population.

For instance, if in the Abkhazeti block, there are 1,044 students of grade 1-6, this is approximately 40% of all students of grades 1-6 in Georgia’s public schools (260,060 students); hence, the 4% of the 13,000 students, or 52 students of grade 1-6, had to be from the Abkhazeti block. • Then, it was determined how many schools of each block would encompass the identified number of students from that block, by dividing the number of students by the average school size in that block; a round-up and hand-adjusted number of students was used to come with the discrete numbers of schools. In the case of Abkhazeti Block, 52 students were divided by 70 (average number of students-6 per school), and the number of schools to be included was rounded up to one school. Because G-PriEd needs to observe the performance of students, needs of children and teachers for each of grades 1 through 6, it was decided to have at least 200 students per grade in each of 11 regions’ or at least 2200 students per grade. Therefore, where the number of students per grade was less than 200, it was disproportionally adjusted to 200. This approach was used with two regions (Abkhazeti and Rach-Lechkhumi) of all 11 regions. Annex 1 details the stages of multi-stage random sampling. The total number of schools to work with was estimated at 121; and the total number of students to be included in G-PriEd activities from these schools was 13,188. This number was later adjusted to 19,000 students and 122 schools when actual data was available. Once the school and students numbers were identified for each block, the second proportionate approach was used to calculate the number of student samples per grade in each school. In each types of schools identified, the proportionate number of students were to be samples from each grade.

o For schools with less than 100 students- 1 student per grade o For schools with 100- 200 students - 2 students per grade o For schools with 200- 300 students - 3 students per grade o For schools with 300- 400 students - 4 students per grade o For schools with 400- 500 students between - 5 students per grade o For schools with 500 and more students – 6 students per grade The result of this theoretical framework is provided in the table 1 Table 1: Expected Number of 1-6 Grade Students and schools by 45 selected blocks of samples

Total # Total # Average Percent Initial Hand- # of Schools of of 1-6 # of 1-6 from the Stipulation adjusted schools -Hand- Region schools students students total of pilot # of adjusted students students

1. 15 1 044 70 100% 52 200 2.9 3 Apkhazeti 2. N/A 3 N/A Sub-Total for 15 1 044 70 0,40% 52 200 2.9 3 Apkhazeti 1. 185 8 988 49 38% 449 9.2 9 Adjara 2. 29 5 529 191 24% 276 1.4 2 3. 15 8 966 598 38% 448 0.7 1 Sub-total for Adjara 229 23 483 103 9,03% 1 174 1 174 11.4 12 1. 85 4 190 49 60% 209 4.2 4 Guria 2. 10 2 177 218 31% 200 1.0 1 3. 2 658 329 9% 33 0.1 1 Sub-total for Guria 97 7 025 72 2,70% 351 351 5.0 6 1. 19 1 767 93 2% 88 0.9 1 Tbilisi 2. 36 8 026 223 10% 401 1.8 2 3. 119 67 224 565 87% 3 360 5.9 6 Sub-total for Tbilisi 174 77 017 443 29,62% 3 850 3 850 9.0 9 1. 299 14 385 48 41% 719 15.0 15 Imereti 2. 45 9 137 203 26% 457 2.3 2 3. 24 11 617 484 33% 581 1.2 1 Sub-total for Imereti 368 35 139 95 13,51% 1 757 1 757 18.0 18 1. 134 9 654 72 44% 483 483 6.7 7 Georgian 121 9 351 77 97% 468 6.0 6 Ethnic- Minority 13 303 23 3% 15 0.6 1 2. 42 8 250 196 37% 412 412 2.1 3 Kakheti Georgian 36 5 453 151 66% 273 1.8 2 Ethnic- Minority 6 2 797 466 34% 140 0.3 1 3 9 4 178 464 19% 209 209 0.4 2 Georgian 7 2 960 423 71% 148 0.3 1 Ethnic- Minority 2 1 218 609 29% 61 0.1 1 Sub-total for Kakheti 185 22 082 119 8,49% 1 104 1 104 9.0 12 1. 76 3 121 41 56% 111 2.7 3 Mtskheta -Mtianeti 2. 7 1 449 207 26% 52 0.2 1 3 3 1 046 349 19% 37 0.1 1 Sub-total for 86 5 616 65 2,16% 281 200 5.0 5 Mtianeti Racha- 1. 66 1 416 21 89% 179 8.3 8 Lechkhu 2. 1 170 170 11% 21 0.1 1 mi & Kvemo 3 0% 0 Svaneti

Sub-total for Racha- Lechkhumi/Kv. 67 1 586 24 0,61% 79 200 12.0 9 Svaneti Samegrelo 1. 207 11 221 54 54% 561 10.3 10 & Zemo 2. 20 3 776 189 18% 189 1.0 1 Svaneti 3. 13 5 625 433 27% 281 0.6 1 Sub-total for Samegrelo and Zemo 240 20 622 86 7,93% 1 031 1 031 12.0 12 Svaneti 1. 188 8 778 47 68% 439 439 9.4 9 Georgian 83 3 961 48 45% 198 4.2 4 Ethnic- Minority 105 4 817 46 55% 241 5.3 5 2. 12 2 389 199 18% 119 300 1.5 2 Samtskhe- Georgian 5 912 182 38% 115 0.6 1 Javakheti Ethnic- Minority 7 1 477 211 62% 185 0.9 1 3. 4 1 789 447 14% 89 300 0.7 2 Georgian 3 1 521 507 85% 255 0.5 1 Ethnic- 1 268 268 15% 45 0.2 1 Minority Sub-total for 204 12 956 64 4,98% 648 648 10.0 13 Samtskhe-Javakheti 1 184 11 971 65 34% 598 599 9.2 9 Georgian 41 4 769 116 40% 238 2.1 2 Ethnic- 143 7 202 50 60% 360 7.2 7 Minority 2. 43 9 747 227 28% 487 487 2.2 2 Kvemo Georgian 20 4 959 248 51% 248 1.0 1 Kartli Ethnic- Minority 23 4 788 208 49% 239 1.2 1 3. 26 13 201 508 38% 660 660 1.3 2 Georgian 19 10 254 540 78% 513 1.0 1 Ethnic- Minority 7 2 947 421 22% 147 0.4 1 Sub-total for Kvemo 253 34 919 138 13,43% 1 746 1 746 13.0 13 Kartli 1. 120 7 672 64 41% 383 6.0 6 Shida 2. 35 7 192 205 39% 359 1.7 2 Kartli 3. 8 3 707 463 20% 185 0.4 1 Sub-total for Shida 163 18 571 7,14% 928 8.0 Kartli 114 928 9 Total 2 081 260 060 125 100% 13 000 13 188 115.5 121

Pilot and Control Group School Sampling and Selection Process The start of the G-PriEd was announced in cooperation with the Ministry of Education and Science. The MES approved the sampling criteria, i.e., school size, regional distribution of schools and language of instruction, to be used to select the pilot schools.

Table 2 below summarizes the sampling parameters by regional representation, language of instruction and school size characteristics across the country.

Table 2. Sampling summary

School Size by students in Total # of grades 1-6 # Region Language of schools per region Instruction 1-299 300-599 >=600

1 Adjara Georgian 9 2 1 12 2 Guria Georgian 4 1 1 6 3 Tbilisi Georgian 1 2 6 9 4 Imereti Georgian 15 2 1 18 5 Kakheti Georgian 6 2 1 9 Ethnic-Minority 1 1 1 3 6 Mtskheta-Mtianeti Georgian 3 1 1 5 7 Racha-Lechkhumi & Kvemo Georgian 8 1 0 9 8 Samegrelo & Zemo Svaneti Georgian 10 1 1 12 9 Samtskhe-Javakheti Georgian 4 1 1 6 Ethnic-Minority 5 1 1 7 10 Kvemo Kartli Georgian 2 1 1 4 Ethnic-Minority 7 1 1 9 11 Shida Kartli Georgian 6 2 1 9 12 Abkhazeti schools Georgian 3 0 0 3 Total 84 19 18 121

G-PriEd with MES support ran an informational campaign in December 2013 to attract candidates for the pilot schools. Participation in the project was voluntary for all the schools. The MES placed the announcement and the application form on its website. Additionally, the Deputy Minister had a press briefing with participation of G-PriEd staff. Concurrent with the informational campaign, G-PriEd called 90% of all public schools, preference given to mountainous, rural and remote to make sure that the information was adequately disseminated. These and other related activities are listed in the text box right:

Campaign to Identify Pilot Schools Activities Deputy Minister and G-PriEd held a media briefing for (1) TV: Public Broadcaster, TV 9, Maestro; (2) Informational Agencies: PIRWELI, Palitra TV, and Interpresnews.

The MES posted the description of the project and the application form in Georgian and Russian languages on the website

The MES disseminated information to all the schools through the Educational Resource Centers. All the school principals received sms notification, and the schools with internet access also received messages via e-mail with attached application forms

G-PriEd called 1,734 schools out of total 1,908 public schools (91%) All interested schools completed an application form during the G-PriEd G-PriEd received a total of 817 applications: 765 from informational campaign period designed Georgian and 52 from ethnic minority public schools. to attract interested schools and sent it to the G-PriEd office. Due to the limited G-PriEd finalized the selection process for 121 schools by resources, schools were selected based on the end of December and the MES approved the list of the first-come/first-served principle, i.e., selected schools in the beginning of January. schools that applied earlier than others within their categories were able to become part of the pilot. It should be noted here that Ushguli School, which is based in the Mestia district of Samegrelo and Zemo Svaneti region, determined that it was unable to participate in the pilot due to a lack of teacher commitment to participate in the project’s interventions. G-PriEd replaced this school with the next school on the waiting list, Nakurtsilao School, based in Martvili district of Samegrelo and Zemo Svaneti region. However, this left the project without a presence in Mestia. As a result, G-PriEd included Lenjeri School, the first on the waiting list for that district. These changes brought the total pilot schools from 121 to 122.

Georgia Primary Education Project selected 122 schools to participate in the pilot stage of the project with 19,070 student population. In addition, to counterbalance each school selected for the pilot, a control school was selected in the same region, from the same school size category, and with the same language of instruction. One condition was absolutely preserved: none of those schools that had applied to be part of the G-PriEd Project was selected for the control group. However, G-PriEd will explore the opportunity for them to be included in the roll-out interventions beyond the pilot stage. Table 3 below describes the number of study population (1-6 students involved in the pilot and control schools) in Georgian and Ethnic Minority instruction schools and sample size identified from each grade level for the baseline study.

Table 3. Number of study population and sample size/grade level Grade # of students in # of students in Sample size in Sample size in Total # of pilot schools control schools pilot schools control schools students in pilot and

control group 1 2,976 2,975 274 265 539 2 3,446 3,295 276 262 538 3 3,441 3,098 278 263 541 4 2,966 2,793 280 262 542 5 3,105 2,781 279 264 543 6 3,136 3,065 278 263 541 19,070 18,007 1,665 1,579 3,244

The size of the study population calculated by each of the six grade levels both in Georgian and minority language instruction schools totals 19,070 students in pilot schools, which is distributed among 122 schools: 103 Georgian and 19 Ethnic-minority instruction schools. The size of the study population calculated by each of the six grade levels in control group schools totals 18,007 students, distributed in 120 schools; 101 Georgian instruction and 19 ethnic-minority instruction schools.

Selection of Students for Baseline Study Once the schools were identified and the total sample size was determined, the number of students in each grade in each school was calculated and students were selected using systematic random sampling within each school and each grade. When a school had more than one class of the same grade, the joint roster of all students was developed, from which the desired number of students was selected randomly. With systematic random sampling strategy, we first calculate the sampling interval by dividing the total number of students of grade 1-6 in particular school (411) by the number of grades we want in the sample (6) and by number of students we want in sample per grade (5). Sampling interval = 411/6/5=14 In this case sampling interval is 14. We then select a number between 1 and the sampling interval from the random number table (in this case 6). The first selected student in the sampling frame (students’ list) is #6. We then count down the students list starting with student #6 and select each 14th student. The second selected student from the students’ list is #20 (6+14), the third selected student is # 34 (20+14); the fourth selected student is # 48 (34+14); the fifth selected student is # 62 (48+14). The detailed instruction about students sampling procedures is described in general administration manual of baseline study. (See annex 2). Test Administration Manuals G-PriEd created training manuals to train assessors of baseline study in test administration. First, it was applied for G-PriEd recruits for validation phase. Later, it was revised and used to train the Baseline Impact Assessment subcontractor’s data collectors and field supervisors. Three manuals were developed by G-PriEd project to provide clear and detailed instructions regarding the baseline study test administration procedures: • General Guidelines on Diagnostic Assessment Test Administration for Impact/baseline Assessment; • Reading Diagnostic Test Administration Manual • Mathematics Diagnostic Test Administration Manual The test administration manuals cover the following procedural elements of the impact/baseline study: general test administration instructions including detailed data collection procedures; timing and scheduling of the test administration in different size/type of schools; detailed description of the sampling procedures at school level, including the number of students’ to be covered in each grade based on the size/type of school; detailed instructions of reading and math test administration procedures in each primary grade level. These manuals were used as a guide to train assessors responsible for the field work/data collection of the impact baseline study. (Refer to the detailed documents in Annex 2). Development of Reading and Math Diagnostic Assessment Tools

Math Topics

G-PriEd supported the development of Numbers and Operations Algebra and Patterns diagnostic/formative assessment tools to be used for Counting (grades 1-2) Patterns (grades 1-4) Impact/baseline study to assess letter and word Number Identification (grades Algebra (grades 2-6) recognition, fluency, comprehension, numeracy, 1-6) mathematical reasoning, and other skills important for Expressing Relations success in reading and math. Using USAID’s Early Number Comparison (grades Between Quantities Grade Reading Assessment (EGRA), Early Grade math 1-6) (grade 6) Assessment (EGMA) as well as similar international diagnostic assessments as best practice models, a group Operations on Numbers Geometry and Space of Georgian educators, linguists, psychometrist and (grades 1-6) Perception child psychologists designed the grade-level tests, tailoring them specifically to the Georgian context and Probability, Statistics and Geometric Figures Data Analysis (grades 1-6) Georgian curriculum. The development of the Georgian Diagnostic Assessment of Reading and Math started in Data Analysis (grades 2-6) Area (grades 5-6) July 2012 and was competed in March 2013. The process was preceded by creation of the Reading and Math Working Groups in cooperation with the Ministry of Education and Sciences (MES) and the National Center for Education Quality Enhancement (NCEQE). G-PriEd, through its Reading and Math Working Groups, developed test creation methodology, for example in reading, the tests were organized around the following competencies: phoneme and syllable segmenting, letter sounds/ word/passage reading frequency, vocabulary, comprehension of narrative/information texts and writing tests. In math, tests were organized around the basic math competencies, as grouped in the text box to the right. Reading and Math Working Groups developed 2 forms and answer keys in each subject. The forms, answer keys and scoring rubrics were finalized for Baseline Impact Assessment based on the results of validation data psychometrical analysis. One of the reading forms was adapted for GSL learners. One of the math forms was translated into Russian, Azeri and Armenian languages for ethnic minority students. Once developed, the tests were validated in ten Georgian primary schools. After revising specific test items, the tests have been finalized and selected to be ready for use for the baseline/impact assessment. Valid and reliable reading and math test items were selected as a result of the validation study and final version of baseline study test forms developed to measure the progress of reading and math competences of Georgian students. Two variations of tests in reading and math called form 1 and form 2 were used to collect student learning outcome data in all 1-6 grades. Data Collection G-PriEd subcontracted GORBI to conduct test administration/data collection of reading and math for impact baseline study. G-PriEd trained GORBI assessors and supervisors in test administration procedures using the test administration manuals for reading, math and general procedures. Field work of baseline study commenced on February 2013 and was completed in April in ethnic minority schools. The field work started with a trial stage of reading test administration in five schools. This stage was followed by a feedback session to revise and adjust the test administration procedures based on the observation visits conducted by G-PriEd staff during the trial stage and the feedback of assessors who were engaged in data collection. During the feedback session decisions were made to make several procedural changes during the test administration, such as: revising typical schedule of test administration process; revision of math tests and making the 2nd grade test verbally administered; additional 3rd person to the assessors’ team to help with the sampling process and continue baseline study with the updated procedures.

In parallel to the baseline study G-PriEd organized observation visits to schools to control the quality of the test administration process. “Quality Control Assessment Team” was composed of G-PriEd technical staff and USAID COR.

Baseline Study Quality Control G-PriEd’s sub-contractor GORBI started preparation for the field work early in February 2013. Prior to startup of field work GORBI assessors received a detailed orientation about reading and math test administration procedures. G-PriEd also trained the GORBI assessors in general test administration procedure including random sampling of students in pilot and control group schools, setting up comfortable environment for test administration, and other test administration related procedures. On February 7, 2013 GORBI invited 48 participants at the training organized by G-PriEd project, 42 interviewers and 6 supervisors from different regions of Georgia. All 48 participants were trained in general administrative procedures and math test administration procedures. At the end of the training all participants passed a test in math and reading test administration abilities. Out of 48 participants 32 failed the reading test. Therefore, for the next day training in reading test administration, only 16 participants were invited to participate. GORBI hired additional assessors for reading and math test administration process. G-PriEd trained the newly hired assessors in test administration procedures in February 14-15, 2013. The baseline study implementation began on February 13, 2013, with the trial of reading tests administration. G-PriEd staff prepared the schedule of observation visits in East and West Georgia and observed implementation of baseline study in randomly selected schools. The purpose of the observation visits was to monitor the accuracy of the test administration process and observe the quality of reading and math test administration in primary grades of pilot and control group schools. The quality observation process could be divided into two stages: The first stage covered the period when G-PriEd staff found series of problems in the administration process which required solutions; G-PriEd proposed possible solutions/recommendations immediately. The second stage started from the period when GORBI assessors continued test administration with the updated procedures of test administration from February 25-March 28, 2012. The observation visits were implemented by G-PriEd technical team and observed by the USAID COR Medea Kakachia and USAID representative Erica Rounsefell.

Table 4. Baseline study observation in East Georgia:

Region Date School name School Type Observer Tbilisi 13.02.2013 Tbilisi school # 133 Pilot Sophie Malashkhia Kakheti 14.02.2013 Kalauri school Control Keti Chachkhiani Tbilisi school # 182 Control Sophie Malashkhia/Gia Tbilisi 19.02.2013 Nozadze Tbilisi 28.02.2013 Tbilisi schools #123 Control Sophie/ Indira

Tbilisi 28.02.2013 Tbilisi schools # 64 Pilot Keti Chachkhiani/Nino Shida Kartli 1.03.13 Gori district Patara Pilot Paata Papava Sophie Malashkia/Nino Shida Kartli 13.03.2013 Gori school # 12 Pilot Udzilauri Shida Kartli 14.03.2013 Dusheti school # 2 Pilot Tbilisi 15.03.2013 Tbilisi school # 102 Pilot Sophie Malashkhia Shida Kartli 25.03.2013 Gori school # 8 Pilot Keti Chachkhiani/Nino

Kvemo Kartli 20.03.2013 Rustavi school # 6 Pilot

Table 5. Baseline study observation in West Georgia:

Region Date School name School G-PriEd Observer Imereti 07.03.2013 Kutaisi school # 22 Control Nato Kobuladze Imereti 11.03.2013 Terjana. Village Gogni Control Nato Kobuladze Imereti 12.03.2013 Kutaisi school # 38 Control Nato Kobuladze Imereti 13.03.2013 Kutaisi school # 35 Pilot Nato Kobuladze Imereti 15.03.2013 Zastaponi/ village Puti Control Nato Kobuladze Imereti 18.03.2013 Kutaisi school # 33 Control Nato Kobuladze Imereti 19.03.2013 Martvili school # 2 Pilot Nato Kobuladze Guria 20.03.2013 Daba Ureki school Control Nato Kobuladze Samegrelo 21.03.2013 Village Bandza school Pilot Nato Kobuladze Samegrelo 22.03.2013 Village Nakhurtsilao Pilot Nato Kobuladze

First Stage of the Observation

At the first stage of observation G-PriEd staff observed serious of procedural violations during the baseline filed administration. All the problems observed during the field visits were discussed at the feedback session, facilitated by G-PriEd and the USAID COR, on February 21st, 2013, when GORBI assessors and supervisors were invited to the feedback session in Tbilisi. The focus of the discussion was about the common problems that assessors encountered during the test administration process, as well as problems during observation visits. The solutions to those problems were discussed together with the assessors and consensus was made to change some areas of the test administration process. The table below summarizes the problems identified during the observation visits and suggests project proposed recommendations to improve test administration process. Table 6. Test Administration Problems Observed and Suggested Recommendations

Problems observed Project Recommendation Solution

Typical Schedule It seemed to be that in most of the Typical Schedule needs revising and The recommendation was cases the individual as well as group adjusted accordingly; accepted: tests went much faster than planned; It was recommended that the schedule sometimes assessors had children in be changed considering these two • Timing of the the room, but were still waiting for findings: individual test was the scheduled time to start. tightened from 15 • The timeline of individual test minutes to 10 could be tightened; minutes;

Confusing messages were given • Typical schedule was about the students taking reading and • The schedule should be revised in a redesigned in a way math tests on different days. If a way that once a child takes a that reading and math student took a reading test one day, reading test, he/she should take the test could be held at but did not show up for the math test math test the same day the same day in the the next day, Gorbi representative given grade level told the assessor that they could

13 substitute a different student for the math test. This contradicts with the direction given at the orientation conference. Selecting students Project Recommendation Solution

There was a case when student from It was recommended to re-consider the Proposed the sampling list was not selected sampling process, make sure that the recommendations were properly. The randomly selected project controls it (not the school accepted: student was a student with special principal). needs (Down’s Syndrome) and assessor moved to another student. However, the principal insisted that second student could not be tested because he had problems at home; then the assessor moved to another Sampling should be as per the original student. The case leaves open the procedure (from the list, with the pre- possibility that the principal tried to defined sample number and interval). influence the process when he did However, instead of the not feel comfortable testing the principal/teacher selecting and escorting student by explaining that the student the student to the testing room, the had some problems. assessor should go into the class and The assessor should have insisted on call out the pre-selected child from the testing the same student who was list. Only in the case of the child’s selected in the sample (assuming the absence the assessor will move to the child does not refuse and assuming next child on the list. he/she is taught under the Ministry’s Gorbi added the 3rd regular curriculum and standards.) person to the assessment team in large schools

In Kutaisi, the principal did not show up at all; nobody was helping the assessors; while one assessor was putting the tests away, the second Recommendation to solve the sampling assessor was bringing the children in; problem in large schools was to offer and there seemed no adherence to the GORBI add one (additional) person to list. each team, somebody who will do the selection from the list to make sure In Kutaisi, observer noticed that a participation of randomly selected second grade child could not do half children in the study. of the test. As assessor was bringing another child from the classroom, the teacher asked loudly can we give you another child to test.

In Khobi, the assessor mentioned that the school principal moved down the list by 3-4 children saying that many children had flu. The assessor mentioned that if they were allowed to select a student, then they could control the process better.

Random sampling according to the rules was the issue: almost

14 everywhere it was noted that 3-4 students down the list “were sick”; in some places principals were asking, “Can we give you a better student?” There is clearly a threat to the random feature of the selection process. The randomness of the selection is a BIG QUESTION.

Protecting the interests of children Project Recommendation Solution

Protecting the interest of children Children’s selection should not be was the issue: In one school when commented by anyone; nobody should the assessor was bringing another say whether this is a strong or weak Recommendation was child from the classroom, the teacher student, or an inclusive one; nobody accepted; asked in front of this child and the should propose to replace a child. entire classroom heard: “Can we give you a different child instead?” Prior to entering a classroom, make clear to the principal and teacher that In one school, when children were there should be no discussion of a leaving earlier from the group tests child’s strengths or weaknesses. There because they were done before the should be no discussion of any others, the assessor would say loudly characteristics of a child (“strange and specifically to each of them: behavior,” “problems at home,” etc.) “Good girl” or “Good boy”! that could possibly impact selection. The only question is whether the child On several occasions there were is present and there should be no other discussions in front of a class that discussion. If such discussion occurs, inclusive children should not be the assessor should confront the speaker included in the test. and make sure the child feels protected. This is a negative psychological We request that such cases be reported pressure on those who need more to G-PriEd. time or are not best students, as well as an attempt to discriminate Children with special needs (inclusive) children. – should be included, if they are randomly sampled, and should follow the general rules. If the student is integrated – then the student should do the test with the assigned special teacher’s help. These children should not be excluded. Assessors should ask if the student is inclusive, or integrated – and then respond accordingly. The assessor should put a note beside the student’s name regarding his/her status.

Assessors should be neutral in their language and relationship with the students, so that they are impartial. They should avoid biased comments like, “You are so great. You must be good at math!” but rather remain neutral

while also encouraging and comforting the child.

Before the test Project Recommendation Solution

The Principal brought students into To avoid any misunderstandings and Recommendation was the room. The assessor did not ask ensure that assessors are testing students accepted the names and last names, assuming they selected randomly, assessors that students are the ones selected should be strictly instructed to ask the from the list by the principal of student’s first name and last name. random selection; During an individual test, the Assessors were re-instructed to finish Recommendation was assessors allowed the next student to with one student first and then start with accepted come into the room before finishing the next one. Tests should not be filling out the assessor’s form for the accessible for students until the assessor previous student tested. While she instructs them to read it. was working on the form, the student started to read (silently) the test that was on the table. None of the students had pens when The assessor should ask students to they arrived. Assessor sent them to bring pens before the test administration Recommendation was their classrooms to get pens. starts. accepted

G-PriEd asked GORBI staff to have extra pens with them to use if needed.

Third grade students taking the math test will need rulers. GORBI should buy enough rulers so that assessors can hand them out and collect them during the third grade math test.

Reading instructions Project Recommendation Solution

The assessor did not read instructions The assessors were re-instructed to Recommendation was precisely and missed several follow the reading assessment accepted; important points. Some students had guidelines precisely; to guess what they had to do. The assessor should word by word read instructions to the students. Assessor should ensure that everybody understands the instructions before starting the test.

During the test Project Recommendation Solution

Students asked questions during the Students should be informed in the Recommendation was test, e.g., “Can you tell me what this beginning of the test that they should accepted; word means?” All other students raise a hand (not call the assessor) if

16 turned towards the one speaking. they have questions. They should be also informed that the assessor cannot help them to do the test and explain the definitions of the words, etc.

Students were distracted due to the Additional rules and procedures need to The recommendation was following reasons: be introduced and communicated with accepted; • Noise from the outside during respective parties: break time was heard in the • Principals have to be informed that testing room. they cannot enter the room during • From time to time other students the test; were opening the room door. • When showing the sample • The Principal and staff person questions for the math tests, entered the room during in the assessor should make sure they middle of the test and created walk between the rows of desks and noise show the sample questions to every • Another assessor, who was child; observing the process was • Nobody – including the assessors - chatting (whispering) to her should be talking/whispering during colleague during the test the test. The process of test administration It is recommended that assessors look at The recommendation was does not pay equal attention to all all the students, walked along between accepted; children: desks, in order to be attentive to all In many instances with the group test students equally. in both reading, and especially math, children were trying to look at each other’s responses, but the assessor was not paying attention. Children in the last row never got to ask a question or receive attention from the assessor, because the assessor did not walk between the rows of desks and did not come close to every child; the best, loudest, and most active students usually sat in the front rows and they got loud while asking questions. After the test Project Recommendation Solution

When students finished the test, they Each section of the test has its own The recommendation was started to look at other sections of the time, students should not be allowed to accepted; test. look at other sections during the time for one specific section. They have to be informed about this in the beginning and then the assessor should also control it

Math test Project Recommendation Solution

Math test content and instructions: The recommendation was made to The recommendation was Pages and pages with instructions change/re-design math test accepted; and examples are confusing to administration instructions and make it children; they need to be removed; a easier and simpler for student to The instruction section verbal explanation by an assessor is understand; that was in student’s form

17 sufficient. was removed from the tests and separate forms In almost every case, while Assessors need to begin test were created for test explaining the examples, the administration with something like administrators called: assessors do not make any this: “Hello, we are now starting the “administrator’s form”; connection with the test; they do not math test; you will have X tasks in your this form has all the explain that these are the typical test. Tasks are formulated differently. instructions that assessor ways of giving responses, circling Let’s go through some examples to see should read loudly for the answers, or seeing the tasks. They how the tasks look like and how you students prior to the start off as if they were starting doing should formulate your response. startup of the test; also a a test. set of posters/grade were developed for each grade level as a part of the instruction delivery process; The first grade instructions are two- It is recommended to revise the format The recommendation was stage in some tasks, but an assessor of the first grade math test to make it accepted: was giving instructions clearly for much clearer, closer to the EGMA the one stage only. When children format. Assessors were re- first need to identify a missing instructed to give 2 stage number in the first number line, and instructions carefully. then circle it in the second number line, instructors simply say only the first part of the instruction. Children are confused as to what they should do. Also, the first and second parts of such tasks do not have the equally attracting format, and child misses one of them. For instance, in this case above, the second number line is not clearly depicted and children simply missed it. The grade 2 math written instructions It was recommended to move to the The recommendation was changed the children’s attitudes to verbal instructions for the second grade. accepted and the 2nd the test. Because most of them could grade math test not read instructions or comprehend administration structure it, they become confused and was redesigned and nervous. became verbal; the same format as the 1st grade test;

Observation Visits in Minority schools

General observation notes

The assessors had good rapport with the students, and the students were organized and ready when their turn came for each assessment. It was difficult to ascertain if the assessor read the instructions fully/correctly as the testing took place in Azeri language, but the timing was correct and they appeared well organized. Testing started on time.

Overall, student literacy skills varied widely; one student couldn't do a single exercise, while some others were quite proficient. Testing took place in a classroom in the main part of the school. It was adequately lit with sufficient space. While relatively quiet between classes, the testing environment was very noisy when students changed classes approximately every hour, and several students changing classes walked into the testing room while the assessment was taking place. A teacher made some attempts to quell this upon the request of the assessor, but it would have been more effective if someone from the school was posted in the hallway to ensure students didn't enter the room while the assessment was in process (the available teacher spoke Azeri and Turkish (no Georgian and very limited English, so this was difficult to convey to her (she actually is the school's English teacher but was able to understand almost no English).

Second Stage of the Observation The recommendations discussed at the feedback session on February 21, 2103 were submitted to GORBI and they were tasked to adjust the test administration according to the recommendations provided by the project. G-PriEd continued observing the quality of reading and math test administration in the field and documenting observation notes on the quality of test administration process. Observation findings from the second stage are summarized in the table 7 below: Table 7. Second stage of the observation

Positive things observed Negative things observed Recommendation about sampling procedures was There was just one case when the sampling was taken in consideration and in almost all schools questioned in multi-campus school; observed; the sampling was done accurately, according to the test admin. guidelines; There was a case when assessor forgot to conduct the oral portion of the test with two of The third person helping the sampling of student the students. After this was pointed out, she was very helpful; that person organized the went to retrieve the students and conducted the students as they were coming and leaving the room oral portion of the assessment; in a timely manner without creating noise outside while waiting; There was a case when mobile phone was ringing during the test; Assessors were delivering improved the quality of During the oral portion to assess a student's test administration and provided instructions ability to decipher syllables, the assessor was consistently and accurately; nodding her head in rhythm to the syllables; this could artificially enhance the student's ability to divide words into syllables;

Most of the assessors observed by G-PriEd Reading group test is composed of 2 or three administered the reading and math tests according assignments. Observation showed that assessors to the test administration guidelines. provided all instructions at once, while the instructions said different approach.

Recommendation to improve the test administration process

More instructional trainings are needed for assessors to prepare them for accurate data collection stage. Assessors need to follow the Test Administration Guidelines precisely, according to the guidelines developed by the G-PriEd project. One of the common observations that came out from all school visits was

19 Reading Test Administration in Grades 3- 4 in Rustavi school # 6; that assessors need more time to become skilled in delivering the instructions naturally. They also need to pay more attention to every single detail of the test administration manual and digest all information before they go the administer tests at school. Data Entry G-PriEd prepared the software program to facilitate baseline data entry and analysis. The software is based on a free resource at eEgra.com as the basis and is decoded and adapted to ensure its applicability with the G-PriEd diagnostic assessment tools of reading and math. It is housed by a web domain www.classroomassessments.ge. Gorbi assessors will enter the baseline data into the software program and hand over to G-PriEd together with hard copies of test forms completed during the baselines study. Finally, G-PriEd will transfer fully employed software with baseline results to USAID.

Annex 1: Sampling Strategy

Selection Strategy of G-PriEd Intervention Schools and Link With the Impact Study

Introduction

The Georgia Primary Education Project (G-PriEd) is a five-year activity that provides comprehensive assistance to the primary education system to improve reading and math competencies of Georgian students in grades 1-6. This will be achieved by supporting instructional improvements, testing and improving standards, use of technology, and development of subject experts in reading and math in schools.

This paper provides the strategy and rationale for the selection of intervention schools1 for the pilot and expansion phases of G-PriEd activities. It describes the random sampling technique for public schools2 clustered by two major features (size and language of instruction) within 123 geographic strata of Georgia.

The paper also serves as the basis of the sampling strategy for the independent impact study of G- PriEd’s pilot intervention. The objective of the pilot impact study is to evaluate learning outcomes of individual students in grades 1-6 in G-PriEd’s pilot schools. Therefore, the sample for the impact study will sample elements from the G-PriEd pilot and expansion samples (using a random sampling methodology) as described below.

Description of G-PriEd Interventions

G-PriEd will work in approximately 100 schools in its pilot phase and approximately 200 schools in its expansion phase to introduce, assess, and finalize its interventions as they relate to improved reading and math outcomes for students in grades 1-6. During the pilot phase, G-PriEd interventions will affect approximately 13,000 students, and in its expansion phase, approximately 27,000 students.

Project interventions include direct training of teachers in assessment-based teaching, differentiated instruction, and support of teacher learning circles; introduction of classroom formative/diagnostic assessments; and supplemental leveled reading materials, math manipulatives, and e-learning resources. G-PriEd’s support for the teacher learning circles includes direct training over a four-month period for math and Georgian-language teachers, training of principals, and training of school-based teacher circle facilitators.

Rationale for the Proposed School Sampling Approach for G-PriEd Interventions

Study Population4

1Intervention schools are those in which the G-PriEd training program will be implemented. 2By request of the MES, the project is implemented only in public schools. 3Apkhazeti schools are defined in this document as the additional 12th geographic region. 4A sample population is the total (complete) collection of elements. A study population is that group of elements from which the sample is actually collected. A summary characteristic of a population is a parameter.

Over the life of the project, G-PriEd expects to work with up to approximately 300 schools (and with approximately 40,000 grade 1-6 students5), specifically:

For the pilot: approximately 100 schools and all of their grade 1-6 students as of January 2013 (approximately 13,000 students).

For the expansion: approximately 200 schools and all of their grade 1-6 students as of September 2014 (approximately 27,000 students).

Sampling Unit6

Stage 1 of sampling: grade 1-6 students who are enrolled in public schools as of September 2012

Stage 2 of sampling: public schools with 1-6 grade students identified through the stage 1 of the sampling.

Sampling Frame7

Tables 6 and 7 below include a summary of the sampling frame for G-PriEd’s sample, which covers all grade 1-6 students in public schools of Georgia.

Sampling Method

To select participating schools, G-PriEd has used a stratified random sampling approach. In stratified sampling, the population is first divided into homogeneous groups, known as “strata” (singular “stratum”). Three consecutive steps were used to stratify students and schools by the three parameters:

Region: Schools can differ from region to region for a variety of different reasons, including regional culture, density of student population, availability of teacher resources, etc. It is important to learn regional particularities of the learning process; therefore, the 12 strata were created by 12 regions of Georgia, of which 11 are the current geographic regions, and one- Apkhazeti, is made up by the schools under the mandate of the Government of Apkhazeti in exile.

School size: For Georgia, with the large number of rural small size schools, small classrooms, combined classes, and merged small size schools, it is important to identify the influence of the size of schools on learning outcomes of students. Therefore, three different types of schools were to be identified in each of the 12 strata, with the following categories:

Small size schools: 1-299 students

Medium-size schools: 300-599 students

5 These number will change once, based on this sampling strategy, the actual schools with the actual number of students are selection for the project interventions for both stages: pilot and expansion. 6A sampling unit is that element or set of elements considered for selection in some stage of sampling. In a simple single-stage sample, the sampling units are the same as the elements. In a three stage sampling approach (e.g., census blocks, households, adults), however, all three are sampling units while only the last is an element. The terms primary sampling unit, secondary sampling unit, and final sampling unit designate these success stages. 7Sampling frame is the actual set of units from which a sample has been drawn: in the case of a simple random sample, all units from the sampling frame have an equal chance to be drawn and to occur in the sample. In the ideal case, the sampling frame should coincide with the population of interest.

Large schools: 600 and more students

Language of instruction; approximately 18 percent of Georgia’s population is ethnic minority; and there are 307 public schools that provide instruction in the languages of ethnic minority. It is important to implement the reading and math programs in these schools, and observe the trends of learning outcomes. There were two categories of schools identified with this variable:

Schools with only Georgian language of instruction

Schools with other than Georgian language of instruction

As a result, 12 regional strata were selected, each of which had further stratified by the three different size of a school (36 strata); each of the 36 strata were further stratified by the two types of the language of instruction. Because only three out 12 regions have more than one language of instruction, the total number of strata is 4; where 9 regions have 3 strata, and 3 regions have 6 strata each (9*3+3*6). The table # 1 depicts the 45 strata:

Table # 1: Total 45 strata of schools for implementation of G-PriEd activities

Region School Size # of strata

1. 1-299 1

Apkhazeti 2. 300-599 2

3. >=600 3

1. 1-299 4

Adjara 2. 300-599 5

3. >=600 6

1. 1-299 7

Guria 2. 300-599 8

3. >=600 9

1. 1-299 10

Tbilisi 2. 300-599 11

3. >=600 12

1. 1-299 13 Imereti 2. 300-599 14

Region School Size # of strata

3. >=600 15

1. 1-299

Georgian 16

Ethnic-Minority 17

2. 300-599

Kakheti Georgian 18

Ethnic-Minority 19

3. >=600

Georgian 20

Ethnic-Minority 21

1. 1-299 22

Mtskheta-Mtianeti 2. 300-599 23

3. >=600 24

1. 1-299 25 Racha-Lechkhumi & Kvemo 2. 300-599 26 Svaneti 3. >=600 27

1. 1-299 28

Samegrelo & Zemo Svaneti 2. 300-599 29

3. >=600 30

1. 1-299

Georgian 31

Ethnic-Minority 32

2. 300-599

Samtskhe-Javakheti Georgian 33

Ethnic-Minority 34

3. >=600

Georgian 35

Ethnic-Minority 36

Kvemo Kartli 1. 1-299

Region School Size # of strata

Georgian 37

Ethnic-Minority 38

2. 300-599

Georgian 39

Ethnic-Minority 40

3. >=600

Georgian 41

Ethnic-Minority 42

1. 1-299 43

Shida Kartli 2. 300-599 44

3. >=600 45

Once 45 strata are identified, the first step of reckoning the random sample size of 1-6 students have started, using the initial assumption, that G-PriEd resources for the pilot implementation is sufficient only for 13,000 students, or approximately 100-120 schools. The 2012 EMIS data were used for the number of students and public schools in each strata.

A total number of 13,000 students were distributed proportionally to the ratio of the 1-6 student numbers on the corresponding strata. For example, the number of 1-6 students from Guria region is proportionate to the ratio of Guria’s 1-6 students to the total number of 1-6 students in Georgia. The same principle is used in all strata. Additional consideration was given to the fact that at least 200 students of grades 1-6 in each region in each size of school was presented, to allow to have the sufficient sample size for conclusions by these categories.

The second step engaged defining the sample size of schools by dividing the student number above to the average number of 1-6 students per school in the given strata. Where the number of schools would be less than 1, at least 1 ne school was still selected in every strata, given that there was more than 0 student in that strata.

Table 2 below identifies the sample size (or number of public schools) to participate in the pilot and expansion phases through the following process:

Column a lists the total number of public schools per strata (source: EMIS Agency)

Column b lists the total number of grade 1-6 students in each strata (source: EMIS Agency)

Column c calculates the average number of grade 1-6 students per school by dividing b/a. This calculation is conducted in every stratum.

Column d calculates the proportion of grade 1-6 students in each stratum by comparing individual numbers of that stratum to the total numbers. For example, the number of students in Ajara (23483)) is 9,03 percent of all students in grades 1-6 in Georgia (260,060). Further, 38 percent (or 8, 988 out of 23, 483) of students of Ajara participate in the small-size schools. The same principle applies to those strata that are set for ethnic minority regions. For instance, in Kakheti, only 3 percent (303) of 1- 6 students participate in schools with ethnic language of instruction, and 97 percent- in the Georgian schools; at the same time, in Kakheti 34 percent of students enrolled in middle-size schools have the language of instruction in the language minority language.

Column e calculates the number of 1-6 students to be included in G-PriEd activities in each region proportionally to their representation. For example, for Ajara with total 23,483 students in grades 1-6 which is 9.03% out of all students 1-6 in Georgian language public school, G-PriEd will target 1,174 students, or 9,03 percent of 13,000 total samples.

Column f adjusts the student numbers upwards in the smallest regions to ensure at least 200 are included in each region.

Column g lists the number generated by dividing the number of students to be included in G-PriEd (Column f) by the average number of students per school (Column c). This is the number of schools expected to be included in the G-PriEd pilot stage;

Column h lists the hand-adjusted numbers of schools to be included in G-PriEd pilot.

This table will be presented to the EMIS, and the EMIS will be asked to randomly identify the calculated number of schools in each of 45 strata. The final numbers and decisions on the sample size will be made thereafter.

Table 2: Expected Number of 1-6 Grade Students and schools by 45 selected strata of samples

Hand- Hand- Percent Initial # of Total # Total # Average adjusted # adjusted # from Stipulation schools of of 1-6 # of 1-6 of of schools the of the Pilot to be Region schools students students benefiting to be total Sample included students included

a b c d e f g h

1. 1-299 15 1 044 70 100% 52 200 2,9 3

Apkhazeti 2. 300-599

3. >=600

Sub-Total for Apkhazeti 15 1 044 70 0,40% 52 200 2,9 3

1. 1-299 185 8 988 49 38% 449 9,2 9

Adjara 2. 300-599 29 5 529 191 24% 276 1,4 2

3. >=600 15 8 966 598 38% 448 0,7 1

Sub-total for Ajara 229 23 483 103 9,03% 1 174 1 174 11,4 12

1. 1-299 85 4 190 49 60% 209 4,2 4

Guria 2. 300-599 10 2 177 218 31% 200 1,0 1

3. >=600 2 658 329 9% 33 0,1 1

a b c d e f g h

Sub-total for Guria 97 7 025 72 2,70% 351 351 5,0 6

1. 1-299 19 1 767 93 2% 88 0,9 1

Tbilisi 2. 300-599 36 8 026 223 10% 401 1,8 2

3. >=600 119 67 224 565 87% 3 360 5,9 6

Sub-total for Tbilisi 174 77 017 443 29,62% 3 850 3 850 9,0 9

1. 1-299 299 14 385 48 41% 719 15,0 15

Imereti 2. 300-599 45 9 137 203 26% 457 2,3 2

3. >=600 24 11 617 484 33% 581 1,2 1

Sub-total for Imereti 368 35 139 95 13,51% 1 757 1 757 18,0 18

1. 1-299 134 9 654 72 44% 483 483 6,7 7

Georgian 121 9 351 77 97% 468 6,0 6 Kakheti Ethnic-Minority 13 303 23 3% 15 0,6 1

2. 300-599 42 8 250 196 37% 412 412 2,1 3

a b c d e f g h

Georgian 36 5 453 151 66% 273 1,8 2

Ethnic-Minority 6 2 797 466 34% 140 0,3 1

3. >=600 9 4 178 464 19% 209 209 0,4 2

Georgian 7 2 960 423 71% 148 0,3 1

Ethnic-Minority 2 1 218 609 29% 61 0,1 1

Sub-total for Kakheti 185 22 082 119 8,49% 1 104 1 104 9,0 12

1. 1-299 76 3 121 41 56% 111 2,7 3

Mtskheta-Mtianeti 2. 300-599 7 1 449 207 26% 52 0,2 1

3. >=600 3 1 046 349 19% 37 0,1 1

Sub-total for Mtianeti 86 5 616 65 2,16% 281 200 5,0 5

1. 1-299 66 1 416 21 89% 179 8,3 8 Racha-Lechkhumi & Kvemo 2. 300-599 1 170 170 11% 21 0,1 1 Svaneti 3. >=600 0% 0

a b c d e f g h

Sub-total for Racha-Letchkhumi and Kvemo 67 1 586 0,61% 200 12,0 Svaneti 24 79 9

1. 1-299 207 11 221 54 54% 561 10,3 10

Samegrelo & Zemo Svaneti 2. 300-599 20 3 776 189 18% 189 1,0 1

3. >=600 13 5 625 433 27% 281 0,6 1

Sub-total for Samegrelo and Zemo Svaneti 240 20 622 86 7,93% 1 031 1 031 12,0 12

1. 1-299 188 8 778 47 68% 439 439 9,4 9

Georgian 83 3 961 48 45% 198 4,2 4

Ethnic-Minority 105 4 817 46 55% 241 5,3 5

Samtskhe-Javakheti 2. 300-599 12 2 389 199 18% 119 300 1,5 2

Georgian 5 912 182 38% 115 0,6 1

Ethnic-Minority 7 1 477 211 62% 185 0,9 1

3. >=600 4 1 789 447 14% 89 300 0,7 2

a b c d e f g h

Georgian 3 1 521 507 85% 255 0,5 1

Ethnic-Minority 1 268 268 15% 45 0,2 1

Sub-total for Samtskhe-Javakheti 204 12 956 64 4,98% 648 648 10,0 13

1. 1-299 184 11 971 65 34% 598 599 9,2 9

Georgian 41 4 769 116 40% 238 2,1 2

Ethnic-Minority 143 7 202 50 60% 360 7,2 7

2. 300-599 43 9 747 227 28% 487 487 2,2 2

Kvemo Kartli Georgian 20 4 959 248 51% 248 1,0 1

Ethnic-Minority 23 4 788 208 49% 239 1,2 1

3. >=600 26 13 201 508 38% 660 660 1,3 2

Georgian 19 10 254 540 78% 513 1,0 1

Ethnic-Minority 7 2 947 421 22% 147 0,4 1

Sub-total for Kvemo Kartli 253 34 919 138 13,43% 1 746 1 746 13,0 13

a b c d e f g h

1. 1-299 120 7 672 64 41% 383 6,0 6

Shida Kartli 2. 300-599 35 7 192 205 39% 359 1,7 2

3. >=600 8 3 707 463 20% 185 0,4 1

Sub-total for Shida Kartli 163 18 571 114 7,14% 928 928 8,0 9

Total 2 081 260 060 125 100% 13 000 13 188 115,5 121

Link with the impact study:

Once the schools are identified by names through the EMIS, the sampling of students will be conducted for an independent base-line and follow-up activities. The base-line will be conducted in January 2013. The first follow-up- will be conducted in May, 2013. The second follow-up will be conducted in January, 2014, and the final follow-up will be conducted in May 2014.

The impact study will sample students in the given 45 strata. It will be suggested to select at least 200 students in each of these strata; and in proportion of the total number of students identified in each stratum. It will be suggested to select at least 10 percent of the total student population included in the pilot, or approximately 1,318 students.

The impact study will also be advised to divide the sample size proportionally between grades 1 through 6, or approximately 220 students per grade.

Annex 2. Test Administration Guidelines

General Procedures I. Introduction

The goal of the diagnostic assessment in reading and math in primary grades is to check the progress of students toward meeting learning goals and to identify gaps in the primary students’ knowledge in a specific subject. Diagnostic assessments provide information that allows teachers to adjust classroom instruction and/or provide more personalized instruction or intervention based on an individual student’s ability. Initially, results of the assessment will provide baseline data regarding the state of students’ reading and math skills in Georgia; therefore, the results will inform G-PriEd’s planned interventions including training, providing learning resources and establishing Teacher Learning Circles in G-PriEd pilot schools. In the initial stage, diagnostic assessments in reading and math will be conducted among approximately 3,000 primary grade students in 122 pilot schools and a comparable number of students and schools in control schools. The results of future assessments will be compared to the baseline in order to determine the impact of the results of G-PriEd interventions on students’ reading and math competencies. The diagnostic assessment will also be used by teachers in the classroom. Teachers will be given the opportunity to use diagnostic assessments in math and reading in primary grades in their classrooms with all or some students in order to identify the needs of each student and to adapt specific, individually- selected approaches/methods to meet these needs. In this case, the diagnostic assessment procedures and rules differ from the baseline assessment. Analysis of data obtained by teachers during classroom assessment will be done at the classroom level and only the teacher will have access to it. The present manual describes the procedures for administering diagnostic assessment in reading and math in primary grades, which will help to identify the changes introduced by G-PriEd and the impact project interventions achieve. II. General Procedures of Test Administration

Creating the Proper Environment for Test Administration

Administration of the diagnostic assessment in reading and math should be implemented in specifically designated room/rooms that have a calm, quiet environment where students will not be distracted. At the same time, there should be no disruption in the regular teaching process at school due to testing. It is essential that a student be provided detailed instruction during the test in order to prevent any obstacles to understanding the task. It is important that the instruction given to a student verbally be identical to the instruction written in the beginning of the test. This does not apply to the first and the second grades, since at this point students are not yet proficient in reading and will not be able to read written instructions. Therefore, only verbal instruction will work in these two grade levels. Intimidating the students, raising the voice and using the wrong tone during the testing are unacceptable behaviors. On the other hand, student behavior such as disorder, noise, cheating, private conversations, chatting over the mobile phone are unacceptable. In the course of testing, the assessors should not have private conversations with representatives of school administration. Testing of the first and second grade students should be conducted in separate stages. The 3rd, 4th, 5th and 6th grade students can be seated together during the tests.

Assessors should maintain a neutral attitude when communicating and talking with students. They should not make biased comments, such as, “You look so smart that you should be good at math.” Teachers should encourage the students to do their best. Preparing for Testing

The assessor should ask students to bring a pen, a pencil, eraser and a ruler to the testing room. If the students do not have the necessary materials, the assessor should have a sufficient quantity of pens, pencils, rubbers and rulers available for use. The assessor should use a pen for individual tests in reading. Testing Room The room where tests will be conducted should: • not be located in a noisy area; • be in accordance with a usual learning environment; • be equipped with a sufficient quantity (at least 6) desks/tables and chairs for the students and a table and a chair for the assessor; • have sufficient space (approximately up to 75 centimeters) between the desks/tables. III. The Number of Students Involved in the Selection and Testing Process

Selection of students is a multi-step process. Initially, schools were selected by a stratified random method. Schools of various capacities, curricula and languages of instruction were selected in a representative way in all regions of Georgia. In all, 122 pilot and 120 control schools were selected. At the next stage, students’ selection method inside each school is determined based on the size of the school. This method is based on random selection of students that is absolute requirements for test administration. Testing will be conducted in grades 1-6 with a minimum of 1 and maximum of 6 students per grade level, which will result in a minimum of six and a maximum of 36 students per school. Chart 1 below indicates the number of tests and students distributed by schools of various capacities. Chart 1 1 2 3 4 5 6 School size by Number of Total Individual Group test Group test a number of students to be number of test Reading Math students in examined at each students to Reading grades 1-6 level (grades 1-6) be examined at school 1-99 1 6 6 6 6 100-199 2 12 12 12 12 200-299 3 18 18 18 18 300-399 4 24 24 24 24 400-499 5 30 30 30 30 500+ 6 36 36 36 36

• Column 1 of Chart 1 indicates school size by a number of students in grades 1-6.

• Column 2 is a kind of a coefficient determining the number of students to be assessed in each grade of the school of indicated size. • Column 3 indicates the total number of students to be examined at school. • Columns 4, 5 and 6 indicate types of test in reading and math and the required number of students in order to administer the test of this type. A complete list of students by grade level is important for selection. In case there are several classes in the same grade at the school, a list of each parallel class needs to be entered into the unified list of the appropriate level. For instance, if there are three Grade 3 classes and there are 25 students in each class, the Grade 3 unified list should include all 75 students. Thus the school should create and provide to the assessor six unified lists based on each level. It is these kinds of unified lists that should be used for the random selection of the appropriate number of students. The assessor should obtain the lists from the school administration or principal beforehand. The assessor should use the “systematic random selection” method for student selection: After determining the overall number of students of grades 1-6 from each pilot and control school, the number of students to be selected at each grade level is determined based on the coefficient given in Chart 1, i.e., at least one student per class or 6 students per school and at most 6 students per class and 36 students per school. A sampling interval is then determined for each school. With systematic random sampling strategy, we first calculate the sampling interval by dividing the total number of students of grade 1-6 in particular school (411) by the number of grades we want in the sample (6) and by number of students we want in sample per grade (5). Sampling interval = 411/6/5=14 In this case sampling interval is 14. We then select a number between 1 and the sampling interval from the random number table (in this case 6). The first selected student in the sampling frame (students’ list) is #6. We then count down the students list starting with student #6 and select each 14th student. The second selected student from the students’ list is #20 (6+14), the third selected student is # 34 (20+14); the fourth selected student is # 48 (34+14); the fifth selected student is # 62 (48+14). The number of students, interval and random counting number was determined for the pilot and control schools by G-PriEd. This information is provided to the assessor beforehand (Annex 1). IV. Student Selection Process

The announcement of the student selection should take place in the classroom. Before entering the classroom, it should be made clear to a principal and a teacher that they should avoid discussing the strengths and weaknesses of a student selected by a random selection principle. Generally, a teacher and a principal should refrain from characterizing a selected student, especially in a negative context such as, strange behavior”, “problems at home”, which could affect the selected student. The assessor should enter the classroom himself/herself and read the name/names of randomly selected student/students from the list. There may be two possible reasons for an assessor to substitute a student with another one: 1) absence of the student from school and 2) categorical refusal of the student to participate in testing; The assessor or assessment coordinator should escort students from the classroom to the assessment room and back to the classroom in schools where one or two students/class are assessed. In schools where 3-6 students/class are selected, an additional assistant will be required to ensure organization of issues associated with student selection, student transition from classroom to the place of testing and a suitable environment in the testing room. If a randomly selected student has special needs (inclusive or integrated), he/she should participate in the testing process with the rest of students. A special needs student should not be rejected and isolated from 36

the testing process. The assessor should find out whether the category to which such student belongs is inclusion or integration. If the student is integrated, he/she should complete the test with the help of a teacher assigned to him/her. The assessor should make a note of the special needs status of the student in the student section of the test form. It should be noted that the same student should pass both math and reading (individual and group) tests. For the sake of simple and flexible implementation of this process, G-PriEd has created an illustrative testing schedule (Annex 2). The illustrative schedule enables conducting both reading and math tests in a given class on the same day. V. Duties of the Parties Involved in Assessment

The G-PriEd representative provides the coordinator and the assessor with the tests and collects the completed, blank and/or spoiled tests. The project representative also: • Ensures the collating, printing and inserting into envelopes. Several sealed envelopes are designed for each school according to grade levels. The school number, grade level and number of tests are written on each envelope. The number of tests corresponds to the number of students; one test can be added to every level in case one test is spoiled; • Passes the sealed envelopes to the assessment coordinator (including each reserve test); • On completion of the assessment process and data entry process, collects the completed test and the corresponding processed electronic data; • Monitors the assessment process through unscheduled visits. The school principal is responsible to contribute to the assessment process and at the same time to: • Cooperate with assessment coordinator and assessor; • Prepare beforehand room/rooms for testing with appropriate environment for conducting a test; • Prepare unified lists of grade 1-6 students according to the above principles and provide it to the coordinator by e-mail beforehand if possible. If impossible to prepare beforehand, the principal should have the lists of students ready for the assessor on the day of test administration; • Be familiar with the testing schedule to ensure students arrive according to the schedule. The Assessment Coordinator is responsible for cooperating with the assessor, helping the assessor perform his/her duty, monitoring the assessment process and ensuring high quality. The assessment coordinator provides supervision of several schools, including: • Coordinates with school principal on the date of testing and selection of a room/rooms for testing; • Creates a schedule for testing (date, time, name of the assessor, number of students to be assessed in the class) and delivers it to school principal, assessor and G-PriEd project representative; • Provides each assessor with a typical assessment schedule based on the size of a school each of the above are going to work (Annex 2). • Delivers tests inserted into sealed envelopes on the day of testing or a day ahead to assessor; • Monitors the assessment process in a way that does not impede or hinder the process; • Ensures solution of any administrational problems related to the process together with school principal and G-PriEd; • Verifies accuracy of random selection made by assessor and correctness of the name and grade level indicated in the test form; • Checks the tests administered by the assessor as well as scoring made by assessor and makes sure that the quality of test administration is maintained; • Creates the observation schedule and makes an unscheduled visit to one or several schools to observe the test administration process. • Changes the schedule if a class has a written test or exam or for any other valid reason;

• Changes the schedule if there are two shifts at school; Required skills: The Assessment Coordinator should have extensive knowledge of reading and math test assessment and the scoring method. The Assessor is responsible for ensuring a favorable, student-friendly environment for the assessment; explain test instructions, conduct assessment, collect tests; assess tests, grade with scores and enter data into the database, as necessary. Assessor: • Goes to school an hour before the test starts; • Ensures the room environment is appropriate for test administration; • In case the testing room does not meet the requirements, the assessor immediately informs the coordinator and school principal of the issue and waits until it is solved; • Selects students by classes using the random selection method; devises a schedule with the list of selected students arranged by the time of their testing; delivers the schedule to the principal so that he/she invites students to test according to the schedule and not all at once; • Based on the list the assessor has delivered to the principal, enters the following information on the test cover before students are called out: student name and grade. The students are provided a test before the testing starts. • It is important that assessor creates a friendly environment during the testing; it is unacceptable to intimidate, use an intimidating tone, or shout at students. On the other hand, disorder, noise, cheating, private conversations, chatting over the mobile phone are unacceptable too! Required skills: The assessor should have extensive knowledge of reading and math test assessment, scoring method, and the method for data entry into database. Prior to the test, the assessor: • Provides detailed instruction to students regarding reading and math tests. After having read the instruction, assessor asks a question: “Do you have any questions?” The assessor should give students several minutes to look at the test forms. • If a student still has questions regarding the instruction, the assessor answers them and only after he/she has made sure the instruction is clear to all students, says: Please start working. (This is the moment the time counting starts). VI. Factors to Consider When Conducting Individual Tests

• There will be individual and group tests in reading and only a group test in math. • There will be a copy of an individual test where assessor will record a student’s oral answers. • Assessor should be sitting in front of a student so that the student cannot see what the assessor marks on his/her copy. The assessor should clearly understand what the student says and the student, in turn, should understand the instructions given by the assessor. • It is important that the students not be hampered by the stopwatch used by the assessor. To avoid this, the assessor should be holding the stopwatch so that the students cannot see it. When the time has expired, the assessor should tell the student, “Thank you. It’s enough for now.” • It is desirable that assessor finish the test with a positive comment, e.g. “I liked the way you tried to read distinctly,” or “I liked your enthusiasm when you read letters.” The student should not know his/her score. • If the assessor finishes an individual test earlier than planned, the assessor should continue to conduct the test.

• The students’ assessment data should be safeguarded so that other students do not have access to information on how their classmates did on the same test.

VII. Factors to Consider When Conducting Group Tests

• When providing instructions and showing a sample, the assessor should walk between the desks so that the sample is visible to all students. • The assessor should warn students that using mobile phones during the test is not allowed; otherwise, tests will be withdrawn from them. • The assessor should inform students that they are allowed to get his/her attention by raising their hands in case they have any question. The assessor should approach and talk to them quietly, without disturbing the others. He/she should not correct mistakes. In case students ask him/her about the correct answer, he/she should avoid answering by a comment as follows: “Do whatever you think is correct.” The assessor has no right to prompt or make any hint about the correct answer during the test. • The assessor should make a note of a question asked by a student in the special section of the assessor’s reporting form, “Student Questions Regarding the Test” and indicate the task the question concerned. • The assessor should be walking between the desks to make observations and to ensure orderliness. • If the assessor notices during the test that a student has omitted a page or an important section of a test, he/she should quietly draw a student’s attention on the omission and say: “Don’t forget to work on this page too.” • The assessor should make sure students do not copy from one another. In order to avoid cheating, assessor should periodically walk between the lines. If he/she notices an attempt of cheating, he/she should deprive the student of his/her test and keep it in a separate file. He/she should make a record in the “Notes” section of the assessor’s report form and in the students’ list. The student should go back to his class. He/she will be substituted either immediately or later. • In case of completion of a test ahead of time, the assessor should not praise a student/ students who have completed the test sooner than others, as this will result in de-motivating the other students who have not yet finished their work. • If a student completes the test ahead of time, the assessor should make sure he/she leaves the room noiselessly. • If any student requests to leave the room due to an urgent need, assessor should withdraw the test from him and record the times of exit and return on his/her test “exited at --- (time) and returned at --- (time)”. No other student is allowed to go out before the one who has already gone out returns so that they are deprived of any opportunity to talk to each other in the corridor. • Student assessment data should be safeguarded so that no other students are able to find out how their classmates did on the same test. • The assessor should make a note of students who refused to fill in the test on the student list and on the assessor’s reporting form. • The assessor should specify the start time (hour) of the test, the time of test completion by the first student and the time the last student completed the test on the reporting form. VIII. Stage after Completion of Testing

• In the case where scoring and data entry are the assessor’s duty, the assessor will collect each test, score it and enter the data into electronic database. Only after completing these tasks, will he/she deliver the tests to assessment coordinator. It is desirable that test results be processed by the end of the testing day so that the tests can be delivered to the coordinator as soon as possible. As a last

resort, the stage of test assessment and data entry should be finished by the following morning and be ready to be delivered to coordinator. • After the assessor has finished the data entry, the assessor inserts the tests into the same envelope that was provided and presents them to the coordinator. The assessor should present the same number of test that had been delivered to him/her, including the blank and completed forms. The number of forms in the envelope should match the number written on the envelope. • The Assessor is responsible for both data confidentiality and test security. • The Assessment Coordinator should count the tests as soon as he/she receives them, compare them to the initial number specified on the envelope and insert them into envelopes or a file according to grade levels/schools. It is unacceptable to rewrite test materials or duplicate them in any other way! It is unacceptable to open the sealed envelopes before testing! IX. Conducting Testing at Schools with Special Needs Students

One of the G-PriEd pilot schools, School #198, belongs to the category of schools for students with special educational needs. G-PriEd consulted with members of the MES multidisciplinary group prior to test administration in this school to understand the needs of the students. Considering the specific learning levels of the students in this school, the test administration was planned as following: students were not given their grade level test but rather the level of test based on the ability of the student to pass any grade level test. For example, a student in Grade 6 could take a Grade 2 test if Grade 2 was his/her ability level. Following the recommendations of the school psychologist, the MES multidisciplinary group members and teachers of this school, the Grades 1, 2, and 3 students were not considered for testing. On the designated testing day, the test administration was conducted with Grades 4, 5, and 6 students, who started with the Grade 1 level test, with the assumption that after the first grade level test, the students would be able to cope with upper grade tests. However, the results showed that even the first grade level test was difficult for the special needs students. X. Frequently Asked Questions

1. Question: How do you create a comfortable environment for a student in which he/she will be able to demonstrate his/her skills? Answer: First of all, explain to the student that you represent an organization trying to make learning more effective and entertaining for students and now are asking him/her to help you. Tell him/her you are going to have a word game with him/her. The mentioned process should be simple and clear for the student. Have a friendly conversation with the student before the test. Ask him/her whether he/she likes his/her school and whether he/she has any brothers or sisters studying at the same school. Students need feedback. They will try to gain your approval. You should encourage him/her. Don’t make him/her feel he/she has made a mistake. 2. Question: What do we do if a student asks us: “Why did you select me?” “Why should it be me?” Answer: Tell him/her that he/she had been put on the list as a result of random selection and he/she can help you. A student is seeking your feedback and you should be encouraging him/her and thanking him/her for support. 3. Question: What do I say if a student tasks if the testing results will affect his/her school grades?

Answer: Explain that the present assessment is in no way related to his/her school grades. Tell him/her you are working for an organization trying to make learning more interesting and effective students. To make it possible, you need his/her support and are grateful for that. 4. Question: What do I do if a student asks why he was interrupted when reading while the time for testing had not expired yet? Answer: Tell him/her you had received enough information from him/her so you would now move to another issue. 5. Question: How should we answer a student’s question about why we are using the stopwatch? Answer: Tell him/her you are trying to find out the amount of time he/she needs to complete the task. 6. Question: What do we do in case 1) a student refuses to participate in testing or 2) is absent from school? Answer: If a student is absent or refuses to participate, during the selection process when counting the students from the list using the selection interval, move to the next student in line. An example of a random selection of students is as follows: Random number:3 Interval: 4 Total number of students to be selected within a class: 2 Total number of students on the list: 10

Consequently, students to be selected are: the 3rd, the 7th. However, if it turns out that the third student is not present in the classroom or has refused to participate, then select the next in line – the 4th student. Student # according to the list Name Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 Student 7 Student 8 Student 9 Student 10 7. Question: How do we substitute a student whom we have caught cheating with another student? Answer: If a student is suspected of cheating, rebuke the student and continue conducting the test. If cheating actually took place, withdraw the test from the student, keep it separately and have the student go back to his/her classroom. In this case, assessment will be planned with another student. This student is going to be selected by a random selection process either on the same day or later, in accordance with the test schedule. If the schedule does not allow conducting additional testing, the assessor should inform the coordinator about the problem to find the best solution, for instance, to continue work at school or any other option. 8. Question: Is conducting a test in math or reading optional? Which test should be conducted by which assessor? Answer: All assessors will take the same training in both reading and math test administration. The assessors will be organized according to their reading and math knowledge/ skills. READING DIAGNOSTIC TEST ADMINISTRATION GUIDE

Introduction

The reading diagnostic assessment, developed by the USAID Georgia Primary Education Project (G-PriEd), is a systematic process which provides a comprehensive overview of what a student knows, is able to do, and is learning to do. The assessment information provides the foundation for decision-making and planning for instruction and learning. G-PriEd will use the results from the assessment to plan relevant interventions, including a professional development program for teachers in G-PriEd targets schools, provision of appropriate instructional aids, and engagement of parents and community in the school learning environment. At a later stage in the project, classroom teachers will be able to use the diagnostic tests to assess student progress toward learning goals. Diagnostic evaluation of reading comprises of eight type tests: • Segmenting word into syllables • Segmenting word into phonemes • Recognition of letters • Reading of word • Reading of text • Vocabulary

- 42 -

• Understanding-comprehension of narrative text • Understanding-comprehension of informational text The guidelines in this manual provide the detailed instructions and procedures for each type of test are included in the diagnostic evaluation of reading skills. Segmentation of a Word into Syllables

The test on the segmentation of word into syllables is conducted only in Grade I and II. This is an individually, orally administered test. Duration: 1 minute (60 seconds). Objective

The test on the segmentation of a word into syllables evaluates the skill of a student to divide an orally-pronounced word into smaller parts (syllables). Such evaluation provides teachers with the information on how well the student processes basic linguistic skills before learning the alphabet. Syllabication in the pre-reading stage will significantly contribute to the process of learning to read. Materials Required for Test Administration

• One printed copy of the test form per student on which the assessor will note the student’s name and surname; • Clipboard. Test form should be attached to the board in a way that only the assessor can see the test and not the student participating in the test; • Stopwatch (or timer, for timing the test – one minute); • Pencil for marking the answers.

Number of Topics/Assignments

The test form contains 20 words to be segment;, the words are written on one side of the paper. Procedures

• This is an individual test. • Test is conducted only in the oral form. • Do not show the students the paper on which the assessor marks the answers. The Student only listens and pronounces the words said by the assessor. • Test duration – 1 minute (60 seconds). As soon as you pronounce the word, turn on the stop-watch. After 60 seconds, give the student a chance to finish segmentation of the last word pronounced into syllables. If the last word is correctly segmented into syllables consider the relevant point for the last word. • If student spends more than 3 seconds for any specific word, move to the next word. • The assessor marks words with the forward slash (/) in the column “student says” to indicate the syllables the student pronounced. • At the end of 1 minute, assessors mark the last syllable students read by placing a bracket] after the final syllable. • Once all tests have been administered, the assessor counts the total number of correctly named syllables and enters this score in the “Total Score” line at the bottom of the test form.

- 43 -

Instructions provided by the Assessor to the student

Tell the student: “Now, I will name the words. You have to divide the words into syllables. First listen carefully to the word pronounced by me, and then say each syllable you hear. For example, if I say the word “dapa” (Blackboard), you must say in the following way: da-pa. Let’s try it together with me”. Assessor pronounces the word: “movida” (came). Assessor and student should both say together: მო / ვი / და (Mo / vi / da). The teacher claps for each syllable pronounced by the student. Assessor says: “Now, let’s try another word: კალამი (pen - kalami)“. Student must say კა / ლა / მი (ka / la / mi). „You can clap. Do you understand what to do? Let’s start! “ Following the completion of the test tell the student: “Well done, thank you very much.”8 Highlighting the Answers and scoring

• Student does not participate in the marking of syllables on the paper. • Student gets one point for each correctly named syllable. • If a student pronounces more than one syllable together he/she gets a point only for the first correctly named syllable (student does not receive points for the pronunciation of syllables which were tied with the first syllable). For example: word „საათი“(clock - Saati). Student says in the following way: „საა-თი“ (saa-ti). In this case student gets 1+1=2 points. Compare the way the student segments the word into syllables with the sample provided in the column. Consider other acceptable options provided below. There are words encountered in the text, which contain combination of vowels and consonants. Example 1: „თაგვი“(tagvi - mouse). Student can say: /თა/ /გვი/ (ta/ / gvi/). In this case the student’s answer is considered as correct, as the student segmented the word in accordance with the vowels. In the case when a student segments the word according to the vowels, but distributes the consonants among syllables incorrectly this case is still considered a correct answer. For example: /ხუჭ/ /უ/ /ჭა/ (Khuch/ /u/ /cha/). Example 2: „აიღო“ (aigo – took). Student may say: /აი/ /ღო/ (/ai/ /go/). For such answer student will get 2 points and not 3 points. Segmentation of Word into Phonemes

The test on the segmentation of a word into phonemes is conducted only for Grade I. This is an individually, orally administered test. Duration: 1 minute (60 seconds).

Objective

The test on the segmentation of a word into phonemes evaluates the skill of a student to divide an orally- pronounced word into phonemes. Such evaluation provides assessor with the information on how well the student processes basic linguistic skills before learning the alphabet. Syllabication in the pre-reading stage will significantly contribute to the process of learning to read. Materials Required for Administration

8 Georgian syllabication differs significantly from English; therefore, translating the examples in this section into English would not result in a meaningful explanation.

- 44 -

The test form contains 20 words to be segmented; the words are written on one side of the paper. Procedures

• Student only listens and pronounce the word named by the assessor. • Test duration – 1 minute (60 seconds). As soon as you pronounce the word, turn on the stopwatch. After 60 seconds, give the student a chance to finish the segmentation of the last word pronounced. If the last word is correctly segmented, consider the points for that word. • If a student spends more than 3 seconds for any specific word move to the next word. • In order to mark the pronounced phonemes in the column student says/,the assessor makes vertical line between the letters separated by the student in the process of segmentation: ღ|ო|ბ|ე (fence –g|o|b|e). • The assessor crosses the incorrectly pronounced or missed letters with the slash (/) in the column “student says”. • After one minute the assessor puts the rectangular bracket after the last word segmented by the student (]). • Once all tests have been administered, the assessor counts the total number of correctly named phonemes and enters this score in the “Total Score” line at the bottom of the test form.

Instructions provided by the Assessor to the student

Tell the student: “Now we’ll play with words. First, listen to the word pronounced carefully. Then say each phoneme/letter heard separately. For example, if I tell the word: კარი (door), you must segment it in the following way: კ / ა / რ / ი (k/a/r/i). Let’s try another word: სკოლა (school). Assessor and the student shall say jointly: ს / კ/ ო / ლ / ა. (s/k/o/l/a). Now, you try one more word: ჩიტი (bird). Student must say:ჩ / ი / ტ / ი (ch/i/t/i). Is it clear for you? Let’s start.” Following the completion of the test tell the student: “Well done, thank you very much.” Highlighting the Answers and Scoring

• The student does not participate in the process of marking. • The student gets one point for each correctly named phoneme. • If the student pronounces more than one phoneme together (without segmentation), he/she gets only one point For example: ბ|ა|ნ|ა|ნი (Banana - b|a|n|a|na). In the presented case, the student made a mistake. He/she has not segmented the last two phonemes (n and a). Therefore, student gets 5 points instead of 6. • Points are not deducted for the pronunciation of extra phonemes.

- 45 -

• If the student incorrectly pronounces phoneme/phonemes, note this down on your paper. Example: ფარი (Shield – pari). The student said /ბ/ა/რ/ი/ (b/a/r/i). In this case the student gets three points. The assessor will write down in the comment’s column “note” the way how the student pronounced the word. • Immediately after the test completion (before you start the test with the next student), sum up the points and record at the bottom if the test. Recognition of Letters

Test on recognition of letters is conducted only for the Grades I and II. This is an individually, orally administered test. Duration: 1 minute (60 seconds) for Grade I, and 30 seconds for Grade II. Objective

The test on letter recognition evaluates the ability of the student to recognize and read each letter of the alphabet. The test provides us with the information on skills of the student to comprehend-utilizing the alphabetic principle to decode letters. Materials Required for Administration

• 2-3 test form for the student • One printed copy of the test form per student on which the assessor will note the student’s name and surname; • Clipboard. Test form should be attached to the board in a way that only the assessor can see the test and not the student participating in the test; • Stopwatch (or timer, for timing the test – one minute); Pencil for marking the answers

Number of Topics/Assignments

The test forms contain all thirty three letters of the Georgian alphabet and each of the letters are repeated twice - once in the first half of the test form and second time in the second half. Note: Proceeding from the format of Georgian alphabet, one letter from the 33 letters will be encountered only once. This will be the “simple” letter, which does not have complex outline and is used relatively often. Procedures

• This is an individual test. • Provide the student with the table of letters (student’s paper) and read out the instructions. • Test duration - 60 seconds for Grade I, 30 seconds for Grade II. Turn on the stopwatch as soon as the student reads the first letter. • The assessor highlights only incorrectly pronounced letters with the forward slash (/). • If the student pronounces a word one more time correctly after previously pronouncing it incorrectly, the assessor makes another backward slash (\) in the way to produce (X). The above sign means self-correction and is not considered as mistake. • After 60/30 seconds (depending on the grade level), the assessor writes a bracket (]) after the last letter pronounced by the student.

- 46 -

• Following the completion of the test, the assessor counts the total number of correctly named letters and writes down the number on the scoring sheet. Instructions Provided by the Assessor to the Student

Tell the student: “Look at this paper; there are alphabetic letters written on the paper. You have to loudly read each of the letters you can see. You must start reading from the beginning of the paper (assessor indicates to the first letter) and read the whole horizontal line, then move to the second line (assessor indicates the direction with the finger). If you can’t remember any letter, I will tell you the letter, and then you will continue pronouncing the letters loudly. Is it clear? Let’s start!” If the student loses the point of reading, help to continue and indicate to the next letter to be read. After 60/30 seconds (depending on the grade level), tell the student: “Let’s finish. Well done, thank you very much.” Scoring

• The student does participate in the process of marking. • The student gets 1 point for each correctly read letter. If the student: • Incorrectly pronounces the letter, cross the relevant letter with a forward slash (/) and consider it as an incorrect answer. • Corrects the mistake, cross the relevant letter with two (forward and backward) slashes (X) and consider it as a correct answer. • Spends more than 3 seconds on any letter, you pronounce the relevant letter and consider it as an incorrect answer. • Misses the letter, circle the letter and consider as incorrect answer. For recording of final the results, the assessor tallies only the correctly pronounced letters. The number of incorrectly read/missed letters is deducted from the number of letters read correctly during 60 seconds (Grade I) or 30 seconds (for Grade II). Reading of Words

Test on the reading of words is conducted only for the Grades I and II. This is an individual test, with the duration of 1 minute (60 seconds) for Grade I as well as for Grade II. Objective

The objective of the test on reading of words is to determine the student’s skill in correctly and fluently reading separate words not connected to each other. Materials Required for Administration

• Two-three copies of the test for students; • One copy of the test/ student for the assessor on which the assessor will note the student’s name and surname; • Clipboard. Test form should be attached to the board in a way that only the evaluator can see the printed text and not the student participating in the test;

- 47 -

• Stopwatch (or timer, for timing the test – one minute); • Pencil for highlighting the answers. Number of Topics/Assignments

Test for the Grade I contains 60 words, and for Grade II – 120 words. Procedures

• This is an individual test. Test duration - 1 minute (60 seconds). • Read out instructions for the student and give him/her the student’s paper. • Cross the words read incorrectly by the student with the forward slash (/). • If the student pronounces a word one more time correctly after previously pronouncing it incorrectly, the assessor makes another backward slash (\) in the way to produce (X). The above sign means self-correction and is not considered as mistake. • As soon as student starts reading of the first word, turn on the stopwatch. After 60 seconds, give the student a chance to finish reading the word started before the expiration of one minute. Put a rectangular bracket (]) on the test form after the last read word. • After completion of the test, record the student’s results on the assessor’s paper. • Note: The student is allowed to read the letters loudly and integrate them into the words. Instructions Provided by the Assessor to the Student

Tell the student: “Let’s now read the words loudly. Start reading the words from the top of the paper (assessor indicates o the first word) and read the entire line; then move to the second line (assessor shows with finger). If any of the words is difficult to read, I will help you read the word, then you can continue reading. Is it clear? Then, let’s start.” For the demonstration of the above, follow your finger from left to right for the first two lines of words. Start timing when the student starts reading (and not while the instruction is being provided). Note: The student has to read/pronounce the whole word. If the student reads in phonemes or syllables and is not able to integrate them or incorrectly integrates them into the word, it is counted as a mistake. In the above case, cross the word and consider it as an incorrect answer. Following the completion of the test tell the student: “Well done, thank you very much.” Highlighting Answers and Scoring

• The student does participate in the process of scoring • The student gets 1 point for each correctly read word. If the student: • Incorrectly pronounces the word, put the forward slash (/) over the relevant word and consider it as an incorrect answer. • Corrects the mistake, put two (forward and backward) slashes (X) on the relevant word and consider it as a correct answer. • Spends more than 3 seconds on any word, you pronounce the relevant word and consider it as an incorrect answer.

- 48 -

• Misses the word, circle the letter and consider it as an incorrect answer. • For recording the final results, the assessor tallies only the correctly pronounced words. The number of incorrectly read/missed words is deducted from the number of words read correctly during the 60 seconds. Reading of Text

The text reading test is conducted for grades II, III, IV, V and VI. This is an individual test, duration: 1 minute (60 seconds). Objective

The objective of the text reading test is to determine whether the student can read text fluently and correctly. This is an assessment of smooth/fluent reading skills. A low result in the text reading test is one of the important indicators that a student will have difficulty comprehending text. Number of Topics/Assignments

The test comprises one narrative text. The length of the sentence and words used in text should be appropriate to the grade and age level of the student. Text should fit on one page. Materials Required for Administration

• The text reading test is conducted individually. • Test duration – 1 minute (60 seconds). • Turn on the stop-watch when the student starts reading the first paragraph and not when the student starts reading the text title. Note: Some students will not read title. We do not assign points for reading or not- reading the title. If student reads the title, put a sign (+) to the right of the title. • Cross the words read incorrectly with the forward slash (/). • If the student pronounces a word one more time correctly after previously pronouncing it incorrectly, the assessor makes another backward slash (\) in the way to produce (X). The above sign means self-correction and is not considered as mistake. • After 1 minute give the student a chance to finish reading the sentence started before expiration of 1 minute. Put rectangular bracket (]) after the last read word. • Following the completion of the test, tell the student: “Well done, thank you very much.” • During the tallying of points, you also have to note whether the student was reading with consideration of punctuation and with appropriate intonation.

- 49 -

Instructions for the Student

Tell the student: „Now I will give you a paper with one story written on the paper. You have to read the story loudly, in the way you usually read the text. You will start reading from this section (assessor indicates by finger to the first word). If it is difficult to read any word, I will help and tell you the word. Then you will continue reading. Try to complete assignment with attention and diligence. Is everything clear? Then, let’s start.” Scoring

• The student does not participate in the process of marking. • The student gets 1 point for each correctly read word. If the student: • Incorrectly pronounces the word, put the slash (/) over the relevant word and consider it as an incorrect answer. • Corrects the mistake, put two (forward and backward) slashes (X) on the relevant word and consider it as a correct answer. • Spends more than 3 seconds on any word, you pronounce the relevant word and consider it as an incorrect answer. • Misses the word, circle the letter and consider it as an incorrect answer.

For recording of the final results, the assessor tallies only the correctly pronounced words. The number of incorrectly read/missed words is deducted from the number of words read correctly during the 60 seconds. Vocabulary

The vocabulary test is conducted for the Grades II, III, IV, V and VI. This test is administered in groups. Duration – 10 minutes. Objective

The vocabulary test assesses how well the student understands the meaning of a word in the context of a sentence. Materials Required for Administration

• One printed copy of the vocabulary test for each student on which the assessor writes the student’s name and surname; • Watch or timer to time 10 minutes; • One copy of instructions to be read to the students by the assessor. Number of Topics/Assignments

Grade II - 12 assignments; grade III - 15 assignments, grades IV-VI - 20 assignments. Procedure

• The vocabulary test is conducted in groups. • Test duration – 10 minutes.

- 50 -

• This test comprises several types of assignments:

o In some sentences words are written in bold. The student will correctly understand the meaning of the word.

o In some sentences the word is missing. The student will correctly choose the missed word from the list of words. • At the beginning of the test, the vocabulary tests are distributed to the students. Then the assessor reads the instructions related to the specific task to the students. Instructions Provided by the Assessor to the Students:

After the welcoming remarks, distribute the work papers to the students and tell them:”Students, the papers that were distributed to you, contain 12/15/20 sentences (depending on grade level). There are words missing in some of the sentences, and you must choose the most appropriate word from the list of words. There are words written in bold in some of the sentences. You must identify the word from the list of words as a meaning of the word in bold in the sentence.” “Please, look at your paper. Let’s discuss the examples provided below. Example one: Coward rabbit concealed itself in the bushes at the sight of the bear. In this sentence word coward means: a) shy b) timid c) fast You have to circle answer b) timid, as in the above sentence word “coward” means “timid”. Example two: The bee buzzes, the horse ______. a) Howls b) Twitters c) Neighs You must circle the word c) Neighs.” Is everything clear, students? Then, now it is your turn. Please, circle the answer which is you think is correct. You are allocated 10 minutes for this exercise. I will remind you when the time is over. Now please show me your pens. Are you ready? Then you can start.” Scoring

• Student gets 1 point for each correctly circled answer • Points are assigned after the completion of the test; therefore, the scoring is not the responsibility of the assessor Comprehension of Narrative Text

The test on comprehension of narrative text is conducted for grades III, IV, V and VI. This test is administered in groups. Duration – 30 minutes. Objective

Comprehension of narrative text is one of the important components of the reading skill. The test on comprehension of narrative text evaluates the skill of the student to understand the text and draw conclusions. Materials Required for Test Administration

- 51 -

• One printed copy of test on understanding-comprehension of narrative text for each student; assessor writes down the name and surname of the student on these papers; • Watch or timer to time 30 minutes; • One copy of instructions to be read to the student by the assessor. Number of Topics/Assignments

There are 9 questions in the test forms for Grade III, 12 – for Grade 4 and 15 – for Grades 5 and 6. Procedure

• Test on comprehension of narrative text is conducted in groups. • Test duration – 30 minutes. • At the beginning of the test, the test papers of test on comprehending narrative text are distributed to the students. Then the teacher reads the instructions related to the test to the students. • The assessor watches the time allocated to the students. After 15 minutes the assessor must say: “You have 15 minutes for the completion of the assignment”. • Thirty minutes after the test starts, the assessor says: “Time is over. Please, stop writing and pass in your papers. Thank you.” Instructions Provided by the Assessor to the Students

Tell the students: “Students, you must read the story silently and then answer 9/12/159 questions about the story. Each question has three possible answers. Choose the answer which is correct in your view. You can work on test papers and make notes. If possible, please clearly mark the correct answers. You have 30 minutes (half an hour) for the completion of the test. I will watch the time and remind you when the time expires? Do you understand what to do? Now all of you show me your pens. Are you ready? Then, you can start. Scoring

• The student gets 1 point for each correctly circled answer. • Points are assigned after the completion of the test and this is not the responsibility of the assessor. Comprehension of Informative Text

The test on the comprehension of informational text is conducted for Grades III, IV, V and VI. This test is administered in groups. Duration – 20 minutes. Objective

The skill of comprehending of informational text is an important component of literacy and the National Curriculum. It evaluates the ability of the student to understand the informational text and make conclusions based on it. Materials Required for Test Administration

9The number of questions differs for the different grades. There are 9 questions for Grade 3, 12 questions – for Grade 4, and 15 – for t Grades 5 and six.

- 52 -

• One printed copy of the test on comprehension of informative text for each student; the assessor writes down the name and surname of the student on these papers; • Watch or timer to time 20 minutes; • One copy of the instructions to be read to the student by the assessor Number of Topics/Assignments

There are 10 topics on the test form. Procedure

• The test on the comprehension of informational text is conducted in groups. • Test duration – 20 minutes. • At the beginning of the test, the papers of test on understanding-comprehension of informational text are distributed to the students. Then teacher reads for the students instructions related to the test. • Test comprises of two parts. Students read the text silently. Then they answer questions related to the read text. • Assessor watches the time allocated to the students. After 10 minutes assessor must say: “you have 10 minutes for the completion of the assignment”. • In 20 minutes after the test start the assessor says: “Time is over. Please, stop writing and pass your papers. Thank you.” Instructions Provided by the Assessor to the Students

Tell the students: “The students’ papers were distributed; the test comprises of two parts. The first part of the paper is the informational text, the second part, the interpretation of words. Please, read silently. Each part is followed by 5 questions. In total, you have to answer 10 questions. Each question has three possible answers. Choose the answer which is correct in your view. You can work on test papers and make notes. If possible, please clearly mark the correct answers. You have 20 minutes for the completion of the test. I will watch the time and remind you of the expiration of time. Do you understand what to do? Now all of you show me your pens. Are you ready? Then, you can start. The assessor watches the time allocated to the students. After 10 minutes the assessor must say: “You have 10 minutes for the completion of the assignment”. In 20 minutes after the test start the assessor says: “Time is over. Please, stop writing and pass your papers. Thank you.” The assessor shall collect test materials. Scoring

• The student gets 1 point for each correctly circled answer. • Points are assigned after the completion of the test and this is not responsibility of the assessor.

- 53 -

Table of indicators for evaluation of reading skills

Copy of Grades, for which test is conducted Test Test for the Test name Organization test for duration student 1 2 3 4 5 6 assessor

Segmenting Individual word into X 1 minute _ + syllables Verbal

Segmenting Individual word into X 1 minute _ + phonemes Verbal 1 minutes Recognition of Individual (grade I) X X + + letters Verbal 30 seconds ( grade II)

Reading of Individual X X 1 minute + + words Verbal Individual Reading of text X X X X X 1 minute + + Verbal Group Vocabulary X X X X X 10 minutes + _ Written _ Understanding- Group comprehension X X X X 30 minutes + of narrative text Written

Understanding- comprehension Group X X X X 20 minute + _ of informational Written text

MATH DIAGNOSTIC TEST ADMINISTRATION GUIDE

Introduction

The math diagnostic assessment, developed by the USAID Georgia Primary Education Project (G-PriEd), is a systematic process which provides a comprehensive overview of what a student knows, is able to do, and is learning to do. The assessment information provides the foundation for decision-making and planning for instruction and learning. G-PriEd will use the results from the assessment to plan relevant interventions, including a professional development program for teachers in G-PriEd target schools, provision of appropriate instructional aids, and engagement of parents and community in the school learning environment. At a later stage in the project, classroom teachers will be able to use the diagnostic tests to assess student progress toward learning goals. Instructions for Math Test Administration

First and Second Grade Diagnostic Math Test Administration

Administration of math tests in Grade 1 and Grade 2 is similar except for the total test duration: in Grade 1 the total time is 45 minutes, while in Grade 2 – 60 minutes. Administration in Grades 1-2 is different from Grades 3-6. Timing for Tests

Total time for testing is 45 minutes (Grade 1) or 60 minutes (Grade 2), including time for the assessor to give students detailed instructions aloud and clearly (approximately 10 minutes), read all 24 problems and organize one 5-minute break. It is recommended to have the break in the middle of the testing process, i.e. after completion of the first 9 problems. Time for reading the problem situations is defined for each problem and is shown below the problems (please refer to Grade 1 or Grade 2 Diagnostic Test form). It should be noted that the time defined for a problem is the maximum time and should cover reading a problem situation to the students twice, students’ solving the problem, and, if required, giving individual directions. If all the students perform the task ahead of the set time, then the assessor should immediately start the next problem. After expiration of the designated time for each problem, the assessors should start the next problem regardless of whether everybody completed the task. Process of Giving Instructions

The assessor has the assessor’s form and a stack of posters with samples of the solutions. Grade 1and Grade 2 Diagnostic Tests do not have textual problems’ situations. Situations are read out loudly by the assessor. The assessor reads the problem situation from the assessor’s form and walks in a classroom allowing everyone to see the poster with the sample of the solution. After students review the sample, the assessor answers the students’ questions. The students will have many questions. The assessor will explain only the procedural issues, i.e. how to do the tasks. As for the problem content, s/he should not help them. If the students have questions related to the content, the assessor should say “Think about it, and do as you think best.”

The assessor hands out the tests with the names of the students already written on the tests (compiled by the assessor prior to testing). The assessor tells the students: “I will read each task twice. Please, listen to me carefully and perform your task quietly. Do not say an answer aloud!” The assessor reads each problem’s situation to the students twice (or three times, if necessary) and walks between the desks while the students are solving the problem. If s/he notices that someone does not understand how to do the task, s/he will explain the situation to that student individually. When the assessor sees that all students are ready to start the next task, s/he will start reading the following problem situation, etc. Test Administration in Grades 3-6

Administration of math tests in grades 3-6 is similar except for the number of problems per grade: for Grade 3 and 4 there are 28 problems and for Grade 5 and 6 - 32 problems. Timing for Tests

Total time for testing in Grades 3-6 is 45 minutes, including time for the assessor to give students detailed instructions aloud and clearly (approximately 10 minutes).

Process of Giving Instructions

The assessor is holding the evaluation form and a package of posters with samples of problem solutions. The assessor reads the problem situation from the evaluation form and walks in the classroom allowing everyone to see the samples of problem solutions.

After students review the samples, the teacher answers the students’ questions. The students will have many questions. The assessor will explain only the procedural issues, i.e. how to do the tasks. As for the problem content, s/he should not help them. If the students have questions related to the content, the assessor should say “Think about it, and do as you think best.” Competencies’ matrix

I II III IV V VI Competence grade grade grade grade grade grade Numbers and operations

Counting X X Number identification X X X X X X Number comparison X X X X X X Operations on numbers X X X X X X Algebra and Patterns

Patterns X X X X Algebra X X X X X Expressing relations between X quantities

Geometry and Space Perception

Geometrical figures X X X X X X Area X X Probability, Statistics and Data Analysis

Data analysis X X X X X

Annex 1: Student Selection Strategy

Random selection of students in each pilot and control school is a major requirement.

In order to comply with the random selection strategy, the selection interval is calculated for each school as follows: an overall number of grade 1-6 students of a given school is divided by the number of primary grade level classes determined by a school size coefficient, in other words,

School capacity Coefficient (Overall number of students of Number of students to be examined grades 1-6) in a class (grades 1-6) 1-99 1 100-199 2 200-299 3 300-399 4 400-499 5 500+ 6

For example, an overall number of grade 1-6 students at Abkhazia school #1 is 134; overall, there are 6 classes at the primary stage in this school; the number of students to be examined in each class is 2:

Sampling interval is 11=134/6/2

After the interval is determined, a random number is identified. For example, 3 is a random number for Abkhazia school #1. Consequently, the first student randomly selected from the selection list for each grade level is the 3rd student. Then, 11 students will be counted and the 14th will be selected, since 11 is the interval for this school. Therefore, the following selected student will be the 14th student in the list.

Students to be selected: the 3rd and the 14th

If the 3rd student is absent from school or refuses to be tested, the next student will be selected, that is the 4th student.

According to the above random selection strategy, a chart has been prepared, specifying the overall number of students to be selected within a school, the number of students selected per grade level, and the interval and random number.

ANNEX 2: LIST OF PILOT AND CONTROL SCHOOLS

Table 1. List of pilot schools

School Name Ratio of Total # of Sampling frame - Sampling students/grade students to starting point on interval be assessed the list

Abkhazia School #1 2 12 11 2 Abkhazia School #11 2 12 12 3 Abkhazia School #7 2 12 9 4 36

Shuaghele School 1 6 10 2 Paposhvilebi School 1 6 4 3 Oladauri School 1 6 4 3 Gora School 1 6 4 1 Chkhutuneti School 1 6 8 2 Chakvi School #2 2 12 10 3 Batumi School #27 1 6 12 2 Batumi School #23 2 12 8 4 Agara School 1 6 3 3 Peria School 1 6 16 4 Bobokvati School 2 12 16 5 Batumi School #8 6 36 24 6 120 Kokhnari School 1 6 6 1 Gogolesubani School 1 6 6 2 Chaisubani School 1 6 7 3 Amaghleba School 1 6 6 4 Nasakirali School 3 18 16 5 Ozurgeti School #2 5 30 14 6 72 Tbilisi School #198 Special Needs Tbilisi School #215 2 12 16 2 Tbilisi School #102 3 18 14 3 Tbilisi School #45 6 36 14 4 Tbilisi School #64 5 30 14 5 Tbilisi School #87 6 36 24 6 Tbilisi School #22 5 30 16 1 Tbilisi School #133 4 24 13 2

Tbilisi School #171 4 24 13 3 210

Kitskhi School 1 6 1 Gorisa School 1 6 2 Geguti School 1 6 3 Dilikauri School 2 12 11 4 Bori School 1 6 1 Banoja School 1 6 2 Bazaleti School 1 6 3 Tsirkvali School 1 6 4 Kursebi School 1 6 1 Shukruti School 1 6 2 Patskhanakanevi School #3 1 6 3 Kutaisi School #35 2 12 9 4 Rodinauli School 1 6 1 Kvakhchiri School 1 6 2 Katskhi School 1 6 3 Tskaltubos School #1 4 24 13 4 Chiatura School #2 3 18 16 1 Kutaisi School #32 5 30 14 1 174

Sagarejo School #3 1 6 2 Pshaveli School 2 12 11 3 Ikalto School 2 12 10 4 Apheni School #2 2 12 9 5 Arashendi School 2 12 8 6 Shilda School #2 1 6 2 Mughanlo School 2 12 11 3 Bakurtsikhe School 2 12 14 4 Gurjaani School 2 12 14 5 Ganjalo School 3 18 14 6 Karajala School 6 36 15 4 Lambalo School 6 36 19 6

186

Mtskheta School #2 2 12 11 3 Sakramuli School 1 6 2 Magharoskari School 1 6 1 Mtskheta School #1 3 18 15 4 Dusheti School #2 3 18 16 5

Sasashi School 1 6 1 Orbeli School 1 6 2 Nikortsminda School 1 6 3 Nasperi School 1 6 1 KvedaTsageri School 1 6 2 Ambroluari School #2 1 6 3 Lasuriashi School 1 6 1 Dekhviri School 1 6 2 Oni School 2 12 14 3 60 Nakhunao School 1 6 2 ShuaKhorgi School 2 12 9 3 Nakhurtsilao School 1 6 3 Lenjeri School # 2 1 6 3 KirtskhisTemi 1 6 2 Zugdidi School #11 1 6 3 Chkhoria School 1 6 4 Bandza School 2 12 9 1 KhabumesTemi 2 12 11 2 Chkhorotsku School #2 2 12 9 3 Kakhati School #2 1 6 1 Martvili School #2 2 12 14 4 Zugdidi School #2 5 30 16 5 132 Spasovka School #1 1 6 1 Gomaro School 1 6 2 Varkhani School 1 6 3 Zanavi School 1 6 4 Balkho School 1 6 1 Diliska School 1 6 2 Gorelovka School #2 1 6 3 Saghamo School 1 6 4 Eshtia School #2 1 6 5 Borjomi School #1 3 18 12 4 Akhalalaki School #4 3 18 16 5 Akhaltsikhe School #1 5 30 16 6 Akhalkalaki School # 2 3 18 15 3 138

Rustavi School #1 2 12 10 1

Vaziani School #1 2 12 10 2 Chapali School 2 12 10 3 Ashkala School 1 6 1 Geta School 1 6 2 Khuldari School 1 6 3 KvemoOrozmani 1 6 1 ZemoOrozmani 1 6 3 Chivtkilisi 1 6 4 Rustavi School #6 4 24 14 5 Rustavi School #3 3 18 16 6 Rustavi School #4 6 36 29 5 Gardabani School #3 5 30 14 6 180 PataraMejvirskhevi School 1 6 3 Kaspi School #4 2 12 11 4 Dvani School 1 6 2 Kvakhvreli School 2 12 10 4 KvemoGomi School 1 6 1 Pkhvenisi School 1 6 2 Kaspi School #1 3 18 14 3 Gori School #8 2 12 15 4 Gori School #12 3 18 16 5 96 1,464

Table 2.List of Control Group Schools

Sampling Sampling Ratio of interval frame - students/grade Total # of starting students to point on School Name be assessed the list

Abkhazia #12 Public School 2 12 8 2 Abkhazia #14 Public School 1 6 3 Abkhazia #15 Public School 1 6 4 24 Kobuleti. Jikhanjuri Public School 2 12 11 1 Shuakhevi. Nenia Public School 1 6 2 Shuakhevi. Nagazeuli Public School 1 6 3 Khelvachauri. Zedagantiadi Public School 1 6 1 Khelvachauri. Korolistavi Public School 1 6 2 Kobuleti. KvedaSameba Public School 1 6 3 Batumi. #29 Public School 1 6 2 Batumi. #31 Public School 1 6 4 Qeda. Akho #1 Public School 1 6 3 Khelvachauri. Makhinjauri Public School 3 18 13 4 Kobuleti. Khutsubani Public School 2 12 15 5 Batumi. #10 Public School 6 36 16 6 126 Chokhatauri. Basileti Public School 1 6 1 Chokhatauri. Erketi Public School 1 6 2 Chokhatauri. Vanzomleti Public School 1 6 3 Chokhatauri. Chakhouri Public School 1 6 4 Ozurgeti. DabaLaituri Public School 3 18 15 5 Ozurgeti. DabaUreki Public School 2 12 15 6 54 Tbilisi #123 Public School 3 18 13 1 Tbilisi #144 Public School 3 18 12 2 Tbilisi #17 Public School 1 6 3 Tbilisi #154 Public School 4 24 13 4 Tbilisi #126 Public School 6 36 16 5 Tbilisi #182 Public School 6 36 18 6 Tbilisi #186 Public School 6 36 14 1

Tbilisi #167 Public School 5 30 13 2 Tbilisi #210 Public School 4 24 14 3 228 Zestafoni. Boslevi Public School 1 6 1 Zestafoni. Futi Public School 1 6 2 Terjola. Gogi Public School 1 6 3 Sachkhere. Argveta Public School 1 6 4 Tkibuli. Gurni Public School 1 6 1 Kutaisi #22 Public School 1 6 2 Tskaltubo. Varcikhhesebi Public School 1 6 3 TskaltuboZedameskhetiBrotseula Public School 1 6 4 TskaltuboSakuloia Public School 1 6 1 Chiatura. Itkhvisi #1 Public School 2 12 9 2 Chiatura. Perevisi Public School 1 6 3 Chiatura. Khreiti Public School 1 6 4 Kharagauli. Biriti Public School 1 6 1 Kharagauli. Vardzia Public School 1 6 2 Kharagauli. Leghvani Public School 1 6 3 Kutaisi #33 Public School 3 18 16 4 Kutaisi #38 Public School 3 18 16 1 Kutaisi #21 Public School 4 24 15 1 156 Gurjaani. Kalauri Public School 2 12 9 2 Telavi. Kuri Public School 2 12 10 3 Telavi. Kurdghelauri Public School 2 12 10 4 Lagodekhi. Baisubani Public School 1 6 5 Sagarejo. Tokhliauri Public School 1 6 6 Sagarejo. Udabno Public School 1 6 2 Sagarejo. Mughanlo Public School 1 6 6

Gurjaani. Vejini Public School 2 12 13 4 Gurjaani #2 Public School 3 18 16 5 Lagodekhi. Kabali #2 Public School

3 18 16 6 Telavi. Telavi #9 Public School 5 30 15 4 Sagarejo. Iormughanlo Public School

6 36 19 6 174 Dusheti. Lafanaantkari Public School 1 6 3 Dusheti. Kvesheti Public School 1 6 2

Mtskheta. Saguramo Public School 2 12 9 1

Mtskheta. Tsolkani Public School 2 12 12 4 Mtskheta. Tsikhisdziri Public School 3 18 13 5 54 Lentekhi. Rtskhmeluri Public School 1 6 1 Tsageri. Lailashi Public School 1 6 2 Ambrolauri. Kvatskheti Public School 1 6 3 Lentekhi. Choluri Public School 1 6 1 Oni. Ghari Public School 1 6 2 Ambrolauri #1 Public School 2 12 12 3 Tsageri. Bardnali Public School 1 6 1 TsageriOkhureshi Public School 1 6 2

54 Martvili. Kurzu Public School 1 6 2 Khobi. KvemoKvaloni #1 Public School 1 6 3 Mestia. Chuberi Public School 1 6 1 Chkhorotsk. KirtskhisTemi #2 Public School 1 6 2 Zugdidi. #9 Public School 1 6 3 Zugdidi. Jumri Public School 1 6 4 Martvili. Salkhino Public School 1 6 1 Chkhorotsk. NafichkhovosTemi Public School 2 12 10 2 Chkhorotsk. Zumi Public School 1 6 3 Zugdidi. Zedaetseri Public School 1 6 1 Senaki #7 Public School 2 12 13 4 Zugdidi #3 Public School 5 30 15 5 108 Ninotsminda. Gorelovka #3 Public School 1 6 1 Adigeni. Abastumani Public School 1 6 2 Adigeni. DidiSmadi Public School 1 6 3 Adigeni. Zarzma Public School 1 6 4 Akhalqalaqi. Barateli #1 Public School 1 6 1 Akhalqalaqi. Diliska #2 Public School 1 6 2 Ninotsminda. Fogi Public School 2 12 9 3 Ninotsminda. Satkhi Public School 2 12 11 4 Ninotsminda. DidiKhanchala Public School 1 6 5 Borjomi. DabaTsagveri Public School 2 12 10 4 Akhalqalaqi #1 Public School 3 18 13 5

Akhaltsikhe #5 Public School 6 36 18 6 Akhaltsikhe #3 Public School 4 24 14 3 156 Rustavi #12 Public School 2 12 14 1 Gardabani. Vakhtangisi #2 Public School 1 6 2 Bolnisi. Talaveri #2 Public School 1 6 3 Tsalka. Nardevani Public School 2 12 9 1 Bolnisi. Kvemobolnisi #2 Public School 2 12 14 2 Marneuli. Tsofi Public School 1 6 3 Dmanisi. Safarlo Public School 1 6 1 Dmanisi. Iaghuflo Public School 1 6 3 Tsalka. Kazilkilisi Public School 1 6 4 Rustavi #26 Public School 3 18 12 5 Rustavi #12 Public School 2 9 6 6 Rustavi #18 Public School 3 18 12 6 Rustavi #20 Public School 5 30 15 5

147 Gori. PataraGarejvari Public School 2 12 11 3 Kaspi. Ferma Public School 2 12 15 4 Kareli. Bebnisi Public School 1 6 2 Gori. Uplistsikhe Public School 1 6 4 1 6 1 Gori. KvemoKhviti Public School 1 6 2 Kaspi #2 Public School 3 18 15 3 Gori. Shindisi Public School 2 12 14 4 Gori #10 Public School 4 24 16 5 102

Annex 3: Typical School Assessment Scheme

Chart 1: Number of students involved in testing by school size School type by a number of students A B C D E F Grade levels 1-99 100-199 200-299 300-399 400-499 500+ Number of students involved in testing Grade I 1 2 3 4 5 6 Grade II 1 2 3 4 5 6 Grade III 1 2 3 4 5 6 Grade IV 1 2 3 4 5 6 Grade V 1 2 3 4 5 6 Grade VI 1 2 3 4 5 6 Total 6 12 18 24 30 36

Chart 2: Test schedule in A type (1-99) school Time Grade Reading Test Time Grade Math Test

Day I Day I 10:00 - 10:10 I Student I 10:00-11:00 V+VI Student I+I Group

10:10 –10:20 II Student I 11:00 -12:00 I Student I Group

10:20-10:30 II Student I Group part 12:00 -13:00 II Student I Group

30 0 4 III Student I Individual 13:00--14:00 Student I+I Group 10: _1 : 0 III+IV part 10:40 -10:50 IV Student I Individual part 10:50 -11:50 III + IV Student I+I Group part 11:00– 11:10 V Student I Individual part 11:10– 11:20 VI Student I Individual part 11:20-12:20 V + VI Student I+I=2 Group part Time 2,5 hours Time 4hours needed: needed:

Chart3: Test schedule in B type (100-199) school

Time Grade Reading Test Time Grad Math Test e Day I Day 1

10:00 - 10:10 I Student I 10:00 – 11:00 III + IV 4 students Group 10:10 - 10:20 Student II 10:20 – 10:40 II Student I Individual 11:00 – 12:00 I 2 students Group part 10:40 -11:50 Student II Individual 12:00 – 13:00 II 2 students Group part 11:50 -11: 00 Student I+II Group 11:10 _11:20 III Student I Individual part 11:20 _11:40 Student II Individual part 11:40 _11:50 IV Student I Individual part 11:50 _12:00 Student II Individual part 12:00 _13:00 III + IV 4 students Group

Time 3hours Time 3hours needed needed Day II Day II 10:00 - 10:10 V Student I Individual 12:00 – 13:00 V + VI 4 students Group part 10:10 - 10:20 V Student II Individual part 10:20 –10:40 VI Student I Individual part 10:40 -10:50 VI Student II Individual part 11:00-12:00 V +VI Student I+II Group

Time 2 hours Time 1hour needed needed

Chart4: Test schedule in C type (200-299) school

Time Grade Reading Test Time Grade Math Test

Day 1 Day 1 10:00 - 10:10 I Student I 10:00 – 11:00 III + IV 6 students Group

10:10 - 10:20 I Student II 11:15-12:15 II 3 students Group 10:20–10:30 I Student III 12:15- 13:15 I 3 students Group

10:30 -10:40 II Individual Student I part 10:40-10:50 II Individual Student II part 10:50 – 11:00 II Individual Student III part

11:00-11:10 II 3 students Group 11:15 – 11:25 III Individual Student I part 11:25 – 11:35 III Individual Student II part 11:35 – 11:45 III Individual Student III part 11:45-11:55 IV Individual Student I part 11:55-12:05 IV Individual Student II part 12:05- 12:15 IV Individual Student III part 12:30-13:30 III + IV Group 6 students Time needed3,5 hours Time needed3 hours

Day II 10:00-11:00 10:00-11:00 V+VI 6 students Group Individual 11:00 - 11:10 V Student I part Individual 11:10 - 11:20 V Student II part Individual 11:20 –11:30 V Student III part Student I Individual 11:30 -11:40 VI part Individual 11:40-11:50 VI Student II part Individual 11:50 – 12:00 VI Student III part 12:00- 13:00 V+VI 6 students Group

Time 2 hours Time needed 1 hour needed

Chart5: Test schedule in D type (300-399) school

Time Grade Reading Test Time Grade Math Test

Day 1 Day 1 Individual 4+4=8 10.00-10:10 I Student I 10:00-11:00 III+IV Group part students I Individual 10:10-10:20 Student II 11:00-12:00 I 4 students Group part I Individual 10:20-10:30 StudentIII 12:00-13:00 II 4 students Group part

I Individual 10:30-10:40 StudentIV part Individual 10:40- 10:50 II Student I part II Individual 10:50- 11:00 Student II part II Individual 11:00-11:10 StudentIII part II Individual 11:10-11:20 StudentIV part 11:20-11:30 II 4 students Group Individual 11:30-11:20 III Student I part III Individual 11:20 - 11:30 StudentII part III Individual 11:30 - 11:40 StudentIII part III Individual 11:50 – 12:00 StudentIV part Individual 12.00-12:10 IV Student I part IV Individual 12:10-12:20 StudentII part IV Individual 12:20-12:30 StudentIII part IV Individual 12:30-12:40 StudentIV part 12:40-13:40 III+IV 8 students Group Time needed3,5 hours Time needed3 hours Day 2 Day 2 10:00-11:00 10:00 – V-VI 4+4=8stude Group

11:00 nts 11:00-11:10 V Individual Student I part V Individual 11:10-11:20 Student II part V Individual 11:20-11:30 Student III part V Individual 11:30-11:40 Student IV part VI Individual 11.40-11:50 Student I part VI Individual 11:50-11:00 Student II part VI Individual 12:00-12:10 Student III part VI Individual 12:10-12:20 Student IV part 12:20 – 12:30 V+VI 8 Students Group

Time needed - 2,5 hours Time needed -1 hour

Chart6: Test schedule in E type (400-499) school Time Grade Reading Test Time Grade Math Test

Day 1 Day 1 Individual 10.00-10:10 I Student I 10:00 – 11:00 III 5 students Group part I Individual 10:10-10:20 Student II 11:00-12:00 I 5 students Group part I Individual 10:20-10:30 Student III 12:00-13:00 II 5 students Group part I Individual 10:30-10:40 Student IV part I Individual 10:40-10:50 Student V part II Individual 10:50-11:00 Student I part II Individual 11:00- 11:10 Student II part II Individual 11:10- 11:20 Student III part II Individual 11:20-11:40 Student IV part II Individual 11:40-11:50 Student V part 11:50-12:00 II 5 students Group Individual 12:00-12:10 III Student I part III Individual 12:10- 12:20 Student II part III Individual 12:20 - 12:30 Student III part III Individual 12:30 – 12:40 Student IV part III Individual 12:40-12:50 Student V part 13:00-14:00 III 5 students Group Time Time 3 hours 4 hours needed needed Day 2 Day 2 Individual 10:00 – V+ VI 5+5=10 10.00-10:10 IV Student I Group part 11:00 students IV Individual 11:00 – IV 5+5=10 10:10-10:20 Student II Group part 12:00 students IV Individual 10:20-10:30 Student III part

IV Individual 10:30-10:40 Student IV part IV Individual 10:40 – 10:50 Student V part 10:50-11:50 IV 5 students Group 12:00-12:10 V Individual Student I part V Individual 12:10-12:20 Student II part V Individual 12:20-12:30 Student III part V Individual 12:30-12:40 Student IV part V Individual 12:40– 12:50 Student V part VI Individual 12.50-13:00 Student I part VI Individual 13:00-13:10 Student II part VI Individual 13:10-13:20 Student III part VI Individual 13:20-13:30 Student IV part VI Individual 13:30-13:40 Student V part 5+5=10 13:40-14:40 V+VI Group students Time needed4,40 hours Time needed 2 hour

Chart7: Test schedule in V type (500+) school

Time Grade Reading Test Time Grade Math Test

Day 1 Day 1 10.00-10:10 I Student I Individual part 10:00 – 11:00 II 6students Group 10:10-10:20 I Student II Individual part 11:00 – 12:00 I 6students Group 10:20-10:30 I Student III Individual part 10:30-10:40 I Student IV Individual part 10:40-10:50 I Student V Individual part 10:50- 11:00 I Student VI 11:00- 11:10 II Student I Individual part 11:20- 11:30 II Student II Individual part 11:30-11:40 II Student III Individual part 11:40-11:50 II Student IV Individual part 11:50-12:00 II Student V Individual part 12:00-12:10 II Student VI Individual part 12:10 – 12:20 II 6 students Group

Time Time needed 2 hours 2.20 hours needed Day 2 Day 2 10:00 – 11:00 III 6student Group 10:00-11:00 s 11:00 – 12:00 IV 6 Group 11:00-11:10 III Student I Individual part students 11:10-11:20 III Student II Individual part 11:20-11:30 III Student III Individual part 11:30-11:40 III Student IV Individual part 11:40-11:50 III Student V Individual part 11:50-12:00 III Student VI Individual part 12.00-12:10 IV Student I Individual part 12:10-12:20 IV Student II Individual part 12:20-12:30 IV Student III Individual part 12:30-12:40 IV Student IV Individual part 12:40-12:50 IV Student V Individual part 12:50 -13:00 IV Student VI Individual part 6+6=12 13:00- 14:00 III+IV Group Students Time Time needed 2 hours 3 hours needed Day 3 Day 3 11:00-11:10 10:00-11:00 V 6student Group

s 11:00-12:00 VI 6 Group 11:10-11:20 V Student I Individual part students 11:20-11:30 V Student II Individual part 11:30-11:40 V Student III Individual part 11:40 -11:50 V Student IV Individual part 11:50 – 12:00 V Student V Individual part 12.00-12:10 V Student VI Individual part 12:10-12:20 VI Student I Individual part 12:20-12:30 VI Student II Individual part 12:30-12:40 VI Student III Individual part 12:40- 12:50 VI Student IV Individual part 12:50-13:00 VI Student V Individual part V+VI 6+6=12 Group 13:000-14:00 Students Time 3 hours Time needed- 2 hours needed