CHAPTER 9 Sample Implementation in TIMSS 2019 Sylvie LaRoche Pierre Foy Overview Rigorous sampling of schools and students was a key component of the TIMSS 2019 project. Implementing the sampling plan was the responsibility of the National Research Coordinator (NRC) in each participating country. NRCs were supported in this endeavor by the TIMSS 2019 sampling consultants, Statistics Canada, and the Sampling Unit of IEA Hamburg. Sampling consultants conducted the school sampling for most countries and trained NRCs in using the Windows® Within-School Sampling Software (WinW3S) provided by IEA Hamburg to implement within-school sampling. As an essential part of their sampling activities, NRCs were responsible for providing detailed documentation describing their national sampling plans (sampling data, school sampling frames, and school sample selections). The documentation for each TIMSS participant was reviewed and completed by the sampling consultants, including detailed information on coverage and exclusion levels, stratification variables, sampling participation rates, and variance estimates. The TIMSS & PIRLS International Study Center and the TIMSS 2019 Sampling Referee, Dr. Keith Rust of Westat, Inc., used this information to evaluate the quality of the samples. TIMSS 2019 marked the beginning of the TIMSS transition to computer based assessment, with countries having the option of administering the new computer-based version of the 2019 assessment, known as eTIMSS, or the paper-and-pencil version as in previous assessment cycles (paperTIMSS). In order to control for mode effects while linking the two versions to the TIMSS achievement scales and to safeguard the measurement of trends from previous assessments, eTIMSS countries also provided a separate sample of bridge data (see Chapter 3 of this volume). This chapter gives a summary of the major characteristics of the national samples for TIMSS 2019, followed by a summary of the major characteristics of the bridge samples for trend countries that participated in eTIMSS. More detailed descriptions of the sample design for each country, including details of population coverage and exclusions, stratification variables, and schools’ sampling allocations, are provided in Appendix 9A: Characteristics of National Samples. TIMSS & PIRLS International Study Center Lynch School of Education CHAPTER 9: SAMPLE IMPLEMENTATION METHODS AND PROCEDURES: TIMSS 2019 TECHNICAL REPORT 9.1 Target Population As described in Chapter 3 (Sample Design), the international target populations for the TIMSS 2019 fourth and eighth grade assessments were defined as the grades that represented 4 and 8 years of formal schooling, respectively, counting from the first year of primary or elementary schooling. Countries could assess either one or both student populations. In addition, at the fourth grade for the TIMSS 2019 cycle, countries could administer a less difficult mathematics assessment, consisting of one third of the items from the regular assessment and two-thirds less difficult items, along with the regular fourth grade science assessment. Albania, Bosnia and Herzegovina, Kosovo, Kuwait, Morocco, North Macedonia, Montenegro, Pakistan, Philippines, and Saudi Arabia chose to administer the less difficult mathematics assessment at the fourth grade while South Africa administered the less difficult mathematics assessment at the fifth grade. Exhibits 9.1 and 9.2 present the grades identified as the target grades for sampling by each country, and include the number of years of formal schooling that the grades represent and the average age of students in the target grades at the time of testing. For most countries, the target grades did indeed turn out to be the grades with 4 and 8 years of schooling, i.e., fourth and eighth grades, respectively. However, in England and New Zealand, children begin primary school at an early age.1 Therefore, these countries administered the TIMSS fourth grade assessment in the fifth year of schooling. The TIMSS eighth grade assessment for England and New Zealand was administered in the ninth year of schooling. Norway chose to assess its fifth and ninth grades to obtain better comparisons with Sweden and Finland. To provide a better match with the demands of the assessments, South Africa and Turkey availed themselves of the option to assess students at a higher grade. South Africa administered the TIMSS fourth grade with less difficult mathematics assessment at the fifth grade and Turkey administered the TIMSS fourth grade assessment at the fifth grade. South Africa administered the eighth grade assessment at the ninth grade, as did its benchmarking provinces of Gauteng and Western Cape. Exhibit 9.1: National Grade Definition – TIMSS 2019 – Fourth Grade Years of Average Age Country’s Name Country Formal at for Grade Tested Schooling Time of Testing Albania Grade 4 4 10.0 Armenia Grade 4 4 9.9 Australia Year 4 4 10.1 Austria Grade 4 4 10.4 1 Given the cognitive demands of the assessments, TIMSS wants to avoid assessing very young students. Thus, TIMSS recommends assessing the next higher grade (i.e., fifth grade for fourth grade TIMSS and ninth grade for eighth grade TIMSS) if, for fourth grade students, the average age at the time of testing would be less than 9.5 years and, for eighth grade students, less than 13.5 years. TIMSS & PIRLS International Study Center Lynch School of Education CHAPTER 9: SAMPLE IMPLEMENTATION METHODS AND PROCEDURES: TIMSS 2019 TECHNICAL REPORT 9.2 Exhibit 9.1: National Grade Definition – TIMSS 2019 – Fourth Grade (continued) Years of Average Age Country’s Name Country Formal at for Grade Tested Schooling Time of Testing Azerbaijan Grade 4 4 10.3 Bahrain Grade 4 4 9.8 Belgium (Flemish) Grade 4 4 10.0 Bosnia and Herzegovina Grade 4 4 10.1 Bulgaria Grade 4 4 10.7 Canada Grade 4 4 9.9 Chile Basic 4 4 10.1 Chinese Taipei Grade 4 4 10.2 Croatia Grade 4 4 10.5 Cyprus Grade 4 4 9.8 Czech Republic Grade 4 4 10.4 Denmark Grade 4 4 10.9 England Year 5 5 10.2 Finland Grade 4 4 10.8 France CM1 4 9.9 Georgia Grade 4 4 10.1 Germany Grade 4 4 10.4 Hong Kong SAR Primary 4 4 10.1 Hungary Grade 4 4 10.5 Iran, Islamic Rep. of Grade 4 4 10.2 Ireland Fourth Class 4 10.4 Italy Primary Grade 4 4 9.6 Japan Grade 4 4 10.4 Kazakhstan Grade 4 4 10.4 Elementary School Korea, Rep. of 4 10.5 Grade 4 Kosovo Grade 4 4 9.9 Kuwait Grade 4 4 9.7 Latvia Grade 4 4 10.8 Lithuania Grade 4 4 10.7 Malta Year 5 4 9.8 Montenegro Grade 4 4 9.8 Morocco Grade 4 4 10.1 Netherlands Group 6 4 10.1 New Zealand Year 5 4.5 - 5.5 10.0 North Macedonia Grade 4 4 9.8 TIMSS & PIRLS International Study Center Lynch School of Education CHAPTER 9: SAMPLE IMPLEMENTATION METHODS AND PROCEDURES: TIMSS 2019 TECHNICAL REPORT 9.3 Exhibit 9.1: National Grade Definition – TIMSS 2019 – Fourth Grade (continued) Years of Average Age Country’s Name Country Formal at for Grade Tested Schooling Time of Testing Northern Ireland Year 6 4 10.4 Norway (5) Grade 5 5 10.7 Oman Grade 4 4 9.7 Pakistan Grade 4 4 10.6 Philippines Grade 4 4 10.1 Poland Primary 4 4 10.3 Portugal Grade 4 4 10.0 Qatar Grade 4 4 9.9 Russian Federation Grade 4 4 10.8 Saudi Arabia Grade 4 4 9.9 Serbia Grade 4 4 10.6 Singapore Primary 4 4 10.4 Slovak Republic Grade 4 4 10.4 South Africa (5) Grade 5 5 11.5 Spain Grade 4 4 9.9 Sweden Grade 4 4 10.8 Turkey (5) Grade 4 5 10.6 United Arab Emirates Grade 4 4 9.7 United States Grade 4 4 10.2 Benchmarking Participants Ontario, Canada Grade 4 4 9.8 Quebec, Canada Grade 4 4 10.1 Moscow City, Russian Fed. Grade 4 4 10.8 Madrid, Spain Grade 4 4 9.9 Abu Dhabi, UAE Grade 4 4 9.7 Dubai, UAE Grade 4 4 9.9 TIMSS & PIRLS International Study Center Lynch School of Education CHAPTER 9: SAMPLE IMPLEMENTATION METHODS AND PROCEDURES: TIMSS 2019 TECHNICAL REPORT 9.4 Exhibit 9.2: National Grade Definition – TIMSS 2019 – Eighth Grade Years of Average Age Country’s Name Country Formal at for Grade Tested Schooling Time of Testing Australia Year 8 8 14.1 Bahrain Intermediate 2 8 13.8 Chile Grade 8 8 14.2 Chinese Taipei Grade 8 8 14.3 Cyprus Grade 8 8 13.8 Egypt Grade 8 8 13.9 England Year 9 9 14.0 Finland Grade 8 8 14.8 France Quatrième 8 13.9 Georgia Grade 8 8 13.8 Hong Kong SAR Secondary 2 8 14.1 Hungary Grade 8 8 14.6 Iran, Islamic Rep. of Grade 8 8 14.1 Ireland Second Year 8 14.4 Israel Grade 8 8 14.0 Lower Secondary Italy Grade 3 8 13.7 Lower Secondary Japan Grade 2 8 14.4 Jordan Grade 8 8 13.9 Kazakhstan Grade 8 8 14.3 Korea, Rep. of Middle School Grade 2 8 14.5 Kuwait Grade 8 8 13.8 Lebanon Grade 8 8 14.0 Lithuania Grade 8 8 14.7 Malaysia Form 2 8 14.3 Morocco Middle School Year 2 8 14.5 New Zealand Year 9 8.5 - 9.5 13.9 Norway (9) Grade 9 9 14.7 Oman Grade 8 8 13.9 Portugal Grade 8 8 14.0 Qatar Grade 8 8 14.0 Romania Grade 8 8 14.8 Russian Federation Grade 8 8 14.8 Saudi Arabia Grade 8 8 13.9 Singapore Secondary 2 8 14.3 TIMSS & PIRLS International Study Center Lynch School of Education CHAPTER 9: SAMPLE IMPLEMENTATION METHODS AND PROCEDURES: TIMSS 2019 TECHNICAL REPORT 9.5 Exhibit 9.2: National Grade Definition – TIMSS 2019 – Eighth Grade (continued) Years of Average Age Country’s Name Country Formal at for Grade Tested Schooling Time of Testing South Africa (9) Grade 9 9 15.5 Sweden Grade 8 8 14.8 Turkey Secondary 4 8 13.9 United Arab Emirates Grade 8 8 13.7 United States Grade 8 8 14.2 Benchmarking Participants Ontario, Canada Grade 8 8 13.8 Quebec, Canada Secondary 2 8 14.2 Moscow City, Russian Fed.