NATIONAL CENTER FOR EDUCATION STATISTICS Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey U.S. Department of Education Office of Educational Research and Improvement NCES 98-053 Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey Editors T. Scott Murray Statistics Canada Ottawa, Ontario, Canada Irwin S. Kirsch Educational Testing Service Princeton, New Jersey, U.S.A. Lynn B. Jenkins Northford, Connecticut, U.S.A. National Center for Education Statistics Office of Educational Research and Improvement NCES 98-053 For sale by the U.S. Government Printing Office Superintendent of Documents, Mail Stop: SSOP, Washington, DC 20402-9328 Pre-publication copy Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey Contents Preface ...........................................................................................................................11 Chapter 1: Introduction—Irwin S. Kirsch and T. Scott Murray.......................13 1.1 Survey Goals, Definitions, and Methods ................................13 The International Adult Literacy Survey: A brief history ..............................................................14 Defining and measuring literacy ....................................17 Conducting the survey ....................................................18 Reporting the results........................................................20 1.2 Objectives of This Report .........................................................20 1.3 A Framework for Evaluating Data Quality .............................21 1.4 Organization of This Report .....................................................22 Part I: Sample Design Error and Non-response Bias ...............................23 Chapter 2: Sample Design—Nancy Darcovich ..................................................25 2.1 Introduction ................................................................................25 2.2 Target Population and Frame Coverage ..................................26 2.3 Sample Design ...........................................................................29 2.4 Concerns About Non-probability-based Sampling ................36 2.5 Conclusions ................................................................................39 Chapter 3: Survey Response and Weighting—Nancy Darcovich ...................41 3.1 Introduction ................................................................................41 3.2 Survey Response .......................................................................42 3.3 Weighting ...................................................................................46 3.4 Benchmarking............................................................................50 3.5 Internal and External Checks ...................................................51 3.6 Sampling Error...........................................................................53 3.7 Conclusions ................................................................................53 3 Contents Contents Chapter 4: Non-response Bias—Nancy Darcovich, Marilyn Binkley, Jon Cohen, Mats Myrberg, and Stefan Persson ...............................55 4.1 Introduction ................................................................................55 4.2 Canadian Study of Non-response Bias ...................................55 4.3 United States Study of Non-response Bias .............................61 4.4 Swedish Study of Non-response Bias......................................67 4.5 Conclusions ................................................................................71 Part II: Non-sampling Error ..........................................................................73 Chapter 5: Data Collection and Processing—Nancy Darcovich and T. Scott Murray ............................................................................75 5.1 Introduction ................................................................................75 5.2 Model Procedures Manuals and Instruments..........................76 Background questions.....................................................76 Literacy tasks ...................................................................76 Standardized non-response coding ................................77 5.3 Field Operations ........................................................................77 5.4 Scoring ........................................................................................83 Analysis of pilot survey data ..........................................83 Quality control .................................................................86 Intra-country rescoring ...................................................87 Inter-country rescoring ...................................................88 5.5 Data Capture, Data Processing, and Coding ..........................89 5.6 Conclusions ................................................................................90 Chapter 6: Incentives and the Motivation to Perform Well—Stan Jones ................................................................93 6.1 Introduction ................................................................................93 6.2 Participation Bias .......................................................................94 6.3 Motivation to Perform Well ......................................................96 6.4 Omit Rate ....................................................................................96 6.5 Not Reached Rate ......................................................................97 6.6 Time Taken on Test ...................................................................98 6.7 Conclusions ................................................................................99 4 Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey Contents Part III: Construct Validity and Proficiency Estimation.......................... 101 Chapter 7: The Measurement of Adult Literacy—Irwin S. Kirsch, Ann Jungeblut, and Peter B. Mosenthal......................................... 105 7.1 Introduction ............................................................................. 105 7.2 The Young Adult Literacy Survey......................................... 106 Understanding task difficulty ...................................... 111 7.3 The U.S. Department of Labor Literacy Survey ................. 113 Prose literacy ................................................................. 113 Document literacy ........................................................ 118 Quantitative literacy ..................................................... 122 Establishing literacy levels .......................................... 125 7.4 The National Adult Literacy Survey ..................................... 129 7.5 Conclusions ............................................................................. 134 Chapter 8: Validity Generalization of the Assessment Across Countries—Don Rock ..................................................................... 135 8.1 Introduction ............................................................................. 135 8.2 Sample and Methods .............................................................. 136 8.3 Results...................................................................................... 137 8.4 Conclusions ............................................................................. 142 Chapter 9: An Analysis of Items with Different Parameters Across Countries—Marilyn R. Binkley and Jean R. Pignal........................................................................... 143 9.1 Introduction ............................................................................. 143 9.2 A Theory of the Cognitive Processes Underlying Literacy Proficiency ............................................................... 146 9.3 Application of the Theory to the IALS ................................ 146 9.4 Analysis of Items .................................................................... 147 Example 1 ...................................................................... 147 Example 2 ...................................................................... 149 Examples 3 and 4 ......................................................... 150 Examples 5 and 6 ......................................................... 152 Example 7 ...................................................................... 154 9.5 Do the Examples Fit the Theory? ......................................... 158 9.6 Conclusions ............................................................................. 160 5 Contents Contents Chapter 10: Scaling and Scale Linking—Kentaro Yamamoto.......................... 161 10.1 Introduction ............................................................................. 161 10.2 Analysis of Items Using Mantel-Haenszel Statistics ........... 162 10.3 Block Order Effects ................................................................ 164 10.4 Scaling Model ......................................................................... 166 10.5 Item Parameter Estimation ..................................................... 167 10.6 Scale Linking Procedures ...................................................... 171 10.7 Evaluation of Differential Group Performance ................... 175 10.8 Conclusions ............................................................................