User Guide for the International Database
Total Page:16
File Type:pdf, Size:1020Kb
IEA International Computer and Information Literacy Study 2018 USER GUIDE FOR THE INTERNATIONAL DATABASE Ekaterina Mikheeva Sebastian Meyer Editors I IEA International Computer and Information Literacy Study 2018 User Guide for the International Database III Ekaterina Mikheeva · Sebastian Meyer Editors IEA International Computer and Information Literacy Study 2018 User Guide for the International Database Edited by Ekaterina Mikheeva Sebastian Meyer International Association for the Evaluation of International Association for the Evaluation of Educational Achievement Educational Achievement Hamburg, Germany Hamburg, Germany With contributions from Sabine Tieck Sabine Meinck International Association for the Evaluation of International Association for the Evaluation of Educational Achievement Educational Achievement Hamburg, Germany Hamburg, Germany Oriana Mora Wolfram Schulz International Association for the Evaluation of The Australian Council for Educational Research Educational Achievement Camberwell, Victoria, Australia Hamburg, Germany IEA Secretariat Keizersgracht 311 1016 EE Amsterdam, the Netherlands Telephone: +31 20 625 3625 Fax: + 31 20 420 7136 email: [email protected] Website: www.iea.nl ISBN/EAN: 978-90-79549-34-4 © International Association for the Evaluation of Educational Achievement (IEA) 2020 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, electrostatic, magnetic tape, mechanical, photocopying, recording or otherwise without permission in writing from the copyright holder. The International Association for the Evaluation of Educational Achievement (IEA), with headquarters in Amsterdam, is an independent, international cooperative of national research institutions and governmental research agencies. It conducts large-scale comparative studies of educational achievement and other aspects of education, with the aim of gaining in-depth understanding of the effects of policies and practices within and across systems of education. For more information about the IEA ICILS 2018 International Database contact: International Association for the Evaluation of Educational Achievement (IEA) Überseering 27 22297 Hamburg Germany email: [email protected] Website: www.iea.nl Design by Becky Bliss Design and Production, Wellington, New Zealand V Contents List of tables and figures vii Chapter 1: Overview of ICILS 2018, its data, and implications for analysis 1 1.1 Main objectives and scope 1 1.2 The design in brief 2 1.3 Analyzing the data 3 1.3.1 Resources and requirements 3 1.3.2 Estimation requirements 4 1.3.3 Levels and units of analysis 6 1.3.4 Limitations of the international database 7 1.4 Contents of the user guide 8 Chapter 2: The ICILS 2018 international database files 9 2.1 Overview 9 2.2 ICILS 2018 database 10 2.2.1 Student data files (BSG) 11 2.2.2 School data files (BCG) 11 2.2.3 Teacher data files (BTG) 12 2.2.4 National contexts survey data file 12 2.3 Records included 13 2.4 Survey variables 14 2.4.1 Identification variables 14 2.4.2 Administration variables 15 2.4.3 Achievement item variables 16 2.4.4 Achievement test scores 17 2.4.5 Questionnaire variables 17 2.4.6 Indices, ratios, and indicators derived from the questionnaire data 18 2.4.7 Summary scales and derived variables from the questionnaires 18 2.4.8 Weighting and variance estimation variables 19 2.4.9 Database creation variables 21 2.5 Coding of missing data 21 2.5.1 Omitted response codes (SPSS: 9, 99, 999, ...; SAS: . ) 21 2.5.2 Not administered response codes (SPSS: 8, 98, 998, …; SAS: .A) 22 2.5.3 Not reached response codes (SPSS: 7; SAS: .R) 22 2.5.4 Logically not applicable response codes (SPSS: 6, 96, 996, …., SAS: .B) 22 2.6 Codebook files 23 2.7 Two versions of the ICILS 2018 international database 23 Chapter 3: Weights and variance estimation for ICILS 2018 25 3.1 Overview 25 3.2 Sampling weights 25 3.2.1 Student weight variables 25 3.2.2 Teacher weight variables 26 3.2.3 School weight variables 26 3.2.4 Selecting the appropriate weight variable 26 3.2.5 Single level analysis 26 3.2.6 Merging files from different levels 27 VIvi ICILS 2018 USER GUIDE FOR THE INTERNATIONAL DATABASE 3.2.7 Multi-level analysis 28 3.2.8 Importance of using weights for data analysis 30 3.3 Variance estimation 32 3.3.1 Variance estimation variables in the ICILS 2018 international database 33 3.3.2 Selecting the appropriate variance estimation variables 33 3.3.3 Example for variance estimation 34 3.3.4 Estimating sampling variance with jackknife repeated replication 34 3.3.5 Comparing groups and statistical significance testing 35 3.3.6 Importance of using the correct variance estimation method 37 Chapter 4: Analyzing the ICILS 2018 data using the IEA IDB Analyzer 39 4.1 Overview 39 4.2 Merging files with the IEA IDB Analyzer 40 4.2.1 Merging data from different countries 41 4.2.2 Merging school and student data files 45 4.2.3 Merging school and teacher data files 46 4.2.4 Merged data files for the example analyses 46 4.3 Performing analyses with the IEA IDB Analyzer 47 4.4 Performing analyses with student-level variables 49 4.4.1 Student-level analysis without achievement scores 49 4.4.2 Student-level analysis with achievement scores 53 4.4.3 Student-level linear regression analysis 56 4.4.4 Calculating percentages of students reaching proficiency levels 60 4.4.5 Computing correlations with context or background variables 62 and achievement scores 4.4.6 Calculating percentiles of students’ achievement 64 4.5 Performing analyses with teacher-level data 66 4.6 Performing analyses with school-level data augmented with school-level data 71 4.7 Trend analyses 75 References 77 Appendices 79 Appendix A International version of the ICILS 2018 questionnaires 79 Appendix B National adaptations of international questionnaires 172 Appendix C Variables derived from the survey data 230 VIIvii List of tables and figures Tables Table 2.1 Countries participating in ICILS 2018 9 Table 2.2 ICILS 2018 data file names 10 Table 2.3 Location of identification variables in the ICILS 2018 international database 15 Table 2.4 Location of administration variables in the ICILS 2018 international database 16 Table 2.5 Location of weighting variables in the ICILS 2018 international database 20 Table 2.6 Location of variance estimation variables in the ICILS 2018 international 21 database Table 2.7 Disclosure risk edits for sampling, identification, and tracking variables 24 Table 2.8 Disclosure risk edits for school questionnaire variables 24 Table 2.9 Disclosure risk edits for student questionnaire variables 24 Table 3.1 Weight variables in student achievement and questionnaire file (BSG) 25 Table 3.2 Weight variables in teacher questionnaire file (BTG) 26 Table 3.3 Weight variables in school and ICT coordinator questionnaire file (BCG) 26 Table 3.4 Number of participating students per participating school (student cluster 29 size) Table 3.5 Number of participating teachers per participating school (teacher cluster 30 size) Table 3.6 Student-level variance estimation variables in the student achievement and 33 questionnaire file (BSG) Table 3.7 Teacher-level variance estimation variables in the teacher questionnaire file 33 (BTG) Table 3.8 School-level variance estimation variables in the school and ICT coordinator questionnaire file (BCG) 33 Table 4.1 Possible merging of data between different file types in ICILS 2018 41 Table 4.2 Statistical procedures available in the Analysis Module of the IEA IDB 47 Analyzer Table 4.3 Fields for variable selection in the Analysis Module of the IEA IDB Analyzer 48 Table 4.4 Example of student-level analysis without CIL achievement scores originally 49 published in the ICILS 2018 international report Table 4.5 Example of student-level analysis with CIL achievement scores originally 53 published in the ICILS 2018 international report Table 4.6 Percentages of students at each proficiency level of CIL achievement, 60 originally published in the ICILS 2018 international report Table 4.7 Example of teacher-level analysis, originally published in the ICILS 2018 67 international report Table 4.8 Example of school-level analysis, originally published in the ICILS 2018 71 international report viii Figures Figure 3.1 Example of unweighted analysis in SPSS 31 Figure 3.2 Example of weighted analysis using the IEA IDB Analyzer 31 Figure 3.3 Example of incorrect variance estimation in SPSS 34 Figure 3.4 Example of weighted analysis using the IEA IDB Analyzer 34 Figure 4.1 IEA IDB Analyzer main window 40 Figure 4.2 IEA IDB Analyzer Merge Module: selecting countries 42 Figure 4.3 IEA IDB Analyzer Merge Module: editing country list 43 Figure 4.4 IEA IDB Analyzer Merge Module: selecting file types and variables 44 Figure 4.5 SPSS Syntax Editor with merge syntax produced by the IEA IDB Analyzer 45 Merge Module Figure 4.6 IEA IDB Analyzer setup for example student-level analysis without plausible 51 values Figure 4.7 IEA IDB Analyzer output for example student-level analysis without 52 achievement scores Figure 4.8 IEA IDB Analyzer setup for example student-level analysis with 54 achievement scores Figure 4.9 IEA IDB Analyzer output for example student-level analysis with CIL 55 achievement scores Figure 4.10 IEA IDB Analyzer Excel output including significance test results for example student-level analysis with CIL achievement scores 56 Figure 4.11 IEA IDB Analyzer setup for example student-level regression analysis with achievement scores 58 Figure 4.12 IEA