Incorporating out-of-school youth in international large-scale assessments – the case of PISA for Development Michael Ward OECD Development Co-operation Directorate & Directorate for Education and Skills World Bank, Washington – 3-4 February 2020 Measuring Learning in National Household Surveys This presentation • PISA and PISA for Development, a brief introduction • Rationale for a PISA household survey • The assessment structure (sampling, questionnaires, skills and subjects covered) • Data collection design • Results • Technical/operational challenges • Lessons learned and next steps Key documents The Programme for International Student Assessment (PISA) • 15-year-old students in 7th grade or above from randomly selected schools take two-hour tests (CBA and PBA) in the key subjects: reading, mathematics and science plus optional assessments. • Tests are competency based and internationally comparable. • Background questionnaires for students, schools, teachers and parents provide context which can help interpret the results. PISA design • Launched in 1997 by OECD - assessments conducted every three years since 2000 - Reading, Mathematics, Science and other optional domains • Major/minor domains 2000 2003 2006 2009 2012 2015 2018 2021 Read Read Read Read Read Read Read Read Math Math Math Math Math Math Math Math Science Science Science Science Science Science Science Science Prb Slv Prb Slv Collab. Prb. Global Creative Slv Competence thinking 5 Participating countries and economies in PISA 2018 Over half a million 15-year-olds from 79 countries and economies All 37 OECD member states and 42 partner countries/economies OECD countries Partner countries and economies EXAMPLES OF PISA OUTCOMES – THREE MAIN TYPES Score points Basic indicators: B-S-J-Z (China) Mean reading performance 550 Singapore Tab I.4.1 530 Macao (China) Estonia Hong Kong (China)* Ireland Canada Finland Countries/economies statistically Korea 510 Poland significantly above the OECD average United States* Sweden New Zealand Japan United Kingdom AustraliaDenmark Germany Norway Chinese Taipei Slovenia Countries/economies NOT statistically France Belgium 490 Portugal* significantly different from the OECD Czech Republic Switzerland Netherlands* Austria average Russia CroatiaLatvia Lithuania Belarus Iceland Hungary Italy 470 Luxembourg Israel Countries/economies statistically Turkey Ukraine significantly different below the OECD Greece Slovak Republic Chile average 450 Malta Serbia United Arab Emirates 430 Romania Costa Rica Uruguay Moldova Mexico Bulgaria MontenegroJordan Malaysia 410 Colombia Brazil AlbaniaQatar Brunei Darussalam Peru Argentina Bosnia and Herzegovina Saudi Arabia 390 North Macedonia Thailand Kazakhstan Baku (Azerbaijan) Georgia Panama 370 Indonesia Morocco Countries/economies with an asterisk* did not meet response-rate Kosovo Lebanon standards, but further analyses could exclude a large bias in the 350 ! published results due to non-response. Philippines Dominican Republic 330 B-S-J-Z (China) 550 Contextual indicators: Singapore High performance Reading performance and High equity Higher Performance Higher 530 equity in PISA 2018 Hong Kong (China) Chinese Taipei Estonia Poland Ireland Finland Macao (China) Korea Canada New Zealand Sweden 510 United Kingdom Germany Japan United States Australia Belgium Slovenia Denmark Norway 490 France Czech Republic Portugal Austria Netherlands Croatia Hungary Switzerland Russia Lithuania Italy Belarus Latvia Iceland 470 Luxembourg Israel Ukraine Turkey Some countries combine Slovak Republic Greece Chile equity and excellence 450 Malta Serbia Romania Uruguay 430 United Arab Emirates Reading performance (in score points) score (in performance Reading Moldova Costa Rica Mexico Jordan Montenegro Malaysia Bulgaria 410 Colombia Albania Brunei Darussalam Brazil Qatar Bosnia and Herzegovina Peru Argentina Saudi Arabia Thailand North Macedonia 390 Baku (Azerbaijan) Georgia Kazakhstan Panama 370 Indonesia Low performance Morocco Lebanon Kosovo 350 Low equity Philippines Dominican Republic 330 25 20 Percentage 15of variation in performance10 explained by social-economic5 statusGreater equity 0 Score-point differences (PISA 2018 – PISA 2015) -35 -25 -15 15 25 35 45 -5 5 North Macedonia Turkey Macao (China) performance Indicators on trends: Change Singapore Jordan United States Moldova Lebanon Hungary Poland United Kingdom Kosovo Sweden Brazil Chinese Taipei Slovak Republic Qatar Estonia Lithuania Peru Czech Republic Malta Denmark Albania Australia between 2015 and 2018 in mean reading Austria Costa Rica United Arab Emirates Hong Kong (China) Ireland Mexico OECD average-36 Korea New Zealand Belgium Montenegro Romania tone darker a in shown are 2018 PISA and 2015 PISA between S Chile differences significant tatistically Portugal Finland Canada France Iceland Croatia Switzerland Italy Israel Latvia Uruguay Greece Slovenia Germany Luxembourg Bulgaria Japan Colombia Fig I.8.1 Norway Dominican Republic Russia Thailand Netherlands Georgia Indonesia RATIONALE FOR A PISA HOUSEHOLD SURVEY – PISA FOR DEVELOPMENT PART ONE Three principle drivers of PISA for Development (1) • OECD Strategy on Development 2012 • make OECD instruments available to a wider range of countries Three principle drivers of PISA-D (2) Increasing demand to participate in PISA PISA cycle Members Non-Members Total 2000 28 15 43 2003 30 11 41 2006 30 27 57 2009 34 40 74 2012 34 30 64 2015 35 36 71 2018 37 42 79 2021 35 50 85 Three principle drivers of PISA-D (3) • Inclusion of PISA as a source of data for global monitoring of SDG 4. Global Indicator 4.1.1.c Proportion of children and young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex • Equates to: Level 2 in PISA (at least 407 points for reading; 420 points for mathematics) PISA for Development International Seminar – London, 25 September 2019 PISA-D informed by the experience of the many low- and middle-income countries already in PISA… • PISA results highlight large differences in student performance • Some of the contextual factors measured by PISA are not adequately capturing the social and economic context in which students learn, teachers work and schools operate in lower and middle-income countries • Because out-of-school rates are high in middle income countries indices of coverage in these contexts can be as low as 50%; in low income countries average indices of coverage are 30% PISA-D Implementation: 2014-2020 • Nine countries from Africa, Asia and Latin America • OECD and international contractors • Development partners • In-school and out-of-school components • Capacity building of participating countries • International and national reports PISA-D’s Achievements PISA is now more relevant to Low PISA now more accessible to Low and Middle Income Countries and Middle Income Countries ▪ Increased the resolution of ▪ Partnerships for participation the PISA tests at the lower established end of the student ▪ Better preparation of performance distribution – countries and their Level 2 and below populations ▪ Captured a wider range of ▪ Capacity building and peer-to- social and economic contexts peer learning established ▪ Collaboration regarding ▪ Incorporated an assessment analysis of data, reporting and of out-of-school 14-16-year- dissemination and use of olds results established 17 What changes in PISA after PISA-D? PISA-D informed by: • PISA experience • Participating countries • Other assessments PISA PISA-D Outputs of PISA-D (instruments & approach for incorporating out-of- school youth in PISA) enhancing PISA from 2021 cycle onwards PISA for Development International Seminar – London, 25 September 2019 PISA-D a success, but the results reveal the extent of the global learning crisis (PISA-D, reading, 15-year-olds students in Grade 7 or above) Percentage of 15-year-olds not Level 1a Level 1b Level 1c Below Level 1c Level 2 Level 3 Level 4 Level 5 Level 6 included in the PISA sample OECD average 11 Zambia 64 Senegal 71 Paraguay m Honduras 59 Guatemala 53 Ecuador 39 Cambodia 72 100 80 60 40 20 0 20 40 60 80 100 % RATIONALE FOR A PISA HOUSEHOLD SURVEY PART TWO 100 % 10 20 30 40 50 60 70 80 90 0 Germany Hong Kong (China) Slovenia Brunei Darussalam covered by PISA What wasthe problem we set out to address? Malta Finland Coverage of the national 15-year-old population (Coverage Index 3) Ireland Czech Republic Singapore Moldova North Macedonia Montenegro Belgium Russia Estonia Greece Qatar Chinese Taipei Kazakhstan United Arab Emirates Spain Iceland France Netherlands Norway Japan Lithuania Poland Hungary Australia Chile Croatia Switzerland Austria New Zealand Latvia Serbia Macao (China) OECD average Korea Denmark Belarus Portugal Luxembourg Ukraine Lebanon Canada Percentage Slovak Republic United States Sweden Indonesia United Kingdom Italy Saudi Arabia Kosovo Georgia countries OECD across 15 entire theof 88.2% On Bosnia and Herzegovina OECD average OECD B-S-J-Z (China) of 15 Israel Argentina Uruguay Albania - Peru year Dominican Republic Romania Turkey represents 2018 PISA , Thailand - - olds Malaysia year Bulgaria Viet Nam Fig I.3.1 - Philippines population old Mexico Brazil Morocco Costa Rica Colombia Jordan Panama Baku (Azerbaijan) What we wanted to achieve Because out-of-school rates are high in many countries, indices of coverage in low-and-middle income countries especially can be as low as 30%, we wanted to achieve: • An approach and methodology for incorporating