Early Childhood Longitudinal Study, Kindergarten Class of 2010–11
Total Page:16
File Type:pdf, Size:1020Kb
Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 Third-Grade, Fourth-Grade, and Fifth-Grade Psychometric Report NCES 2020-123 Technical/Methodological Report U.S. DEPARTMENT OF EDUCATION A Publication of the National Center for Education Statistics at IES Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 Third-Grade, Fourth-Grade, and Fifth-Grade Psychometric Report MARCH 2020 Michelle Najarian Educational Testing Service (ETS) Karen Tourangeau Christine Nord Kathleen Wallner-Allen Nancy Vaden-Kiernan Westat Gail M. Mulligan Project Officer National Center for Education Statistics NCES 2020-123 U.S. Department of Education U.S. Department of Education Betsy DeVos Secretary Institute of Education Sciences Mark Schneider Director National Center for Education Statistics James L. Woodworth Commissioner The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries. NCES activities are designed to address high-priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high-quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public. Unless specifically noted, all information contained herein is in the public domain. We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or suggestions about this or any other NCES product or report, we would like to hear from you. Please direct your comments to NCES, IES, U.S. Department of Education Potomac Center Plaza 550 12th Street SW Washington, DC 20202 March 2020 The NCES Home Page address is https://nces.ed.gov. The NCES Publications and Products address is https://nces.ed.gov/pubsearch. This publication is only available online. To download, view, and print the report as a PDF file, go to the NCES Publication and Products address shown above. This report was prepared for the National Center for Education Statistics under Contract No. ED-IES-14-C-0119 with Westat and the Educational Testing Service. Mention of trade names, commercial products, or organizations does not imply endorsement by the U.S. Government. Suggested Citation Najarian, M., Tourangeau, K., Nord, C., Wallner-Allen, K., and Vaden-Kiernan, N. (2019). Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011), Third-Grade, Fourth-Grade, and Fifth-Grade Psychometric Report (NCES 2020-123). U.S. Department of Education. Institute of Education Sciences. Washington, DC: National Center for Education Statistics. Retrieved [Month day, year] from https://nces.ed.gov/pubsearch. Content Contact Gail M. Mulligan (202) 245-8413 [email protected] CONTENTS Chapter Page LIST OF APPENDIXES ...................................................................................................... ix LIST OF TABLES ................................................................................................................ ix LIST OF FIGURES .............................................................................................................. xix LIST OF EXHIBITS ............................................................................................................. xx 1 INTRODUCTION ........................................................................................... 1-1 1.1 The Third-Grade, Fourth-Grade, and Fifth-Grade Data Collections .......................................................................................... 1-3 1.2 Data Collection Instruments and Administration of Assessments ........................................................................................ 1-4 1.3 Contents of Report .............................................................................. 1-7 2 OVERVIEW OF THE CONTENT COVERAGE AND ADMINISTRATION OF THE DIRECT COGNITIVE ASSESSMENT INSTRUMENTS ................................................................... 2-1 2.1 Content Coverage of the Cognitive Assessments ............................... 2-2 2.1.1 Reading Test Specifications ................................................ 2-2 2.1.1.1 Basic Reading Skills ............................................ 2-3 2.1.1.2 Vocabulary .......................................................... 2-4 2.1.1.3 Comprehension .................................................... 2-5 2.1.1.4 Continuity Between the ECLS-K and the ECLS-K:2011 Reading Frameworks ................... 2-6 2.1.2 Mathematics Test Specifications ......................................... 2-6 2.1.2.1 Number Properties and Operations ...................... 2-7 2.1.2.2 Measurement ....................................................... 2-7 2.1.2.3 Geometry ............................................................. 2-8 2.1.2.4 Data Analysis and Probability ............................. 2-8 2.1.2.5 Algebra ................................................................ 2-8 2.1.3 Science Test Specifications ................................................. 2-9 2.1.3.1 Scientific Inquiry ................................................. 2-9 2.1.3.2 Physical Science .................................................. 2-9 2.1.3.3 Life Science ......................................................... 2-10 2.1.3.4 Earth and Space Science ...................................... 2-10 2.2 Item and Time Allocation Across and Within Subject Areas ............. 2-10 iii Chapter Page 2.3 Mode of Administration ...................................................................... 2-13 2.4 Inclusion of Children With Diverse Language Backgrounds and Language of Administration ........................................................ 2-15 3 ANALYSIS METHODOLOGY ...................................................................... 3-1 3.1 Quality Control Procedures ................................................................ 3-1 3.2 Overview of Item Response Theory (IRT) ......................................... 3-4 3.2.1 Dichotomous Item Calibration ............................................ 3-6 3.2.2 Polytomous Item Calibration .............................................. 3-10 3.2.3 Calculation of IRT Scale Scores ......................................... 3-13 3.2.4 Selection of an IRT Model .................................................. 3-13 3.2.5 Evaluating Items Using Empirical Item Characteristic Curves ................................................................................. 3-15 3.2.6 Item Information and Measurement Precision .................... 3-18 3.2.7 Item Response Theory Estimation Using PARSCALE ........................................................................ 3-22 3.3 Construct Validity: Assessing Dimensionality ................................... 3-24 3.4 Group Differences in Item Functioning .............................................. 3-25 3.5 Development of the Kindergarten Through Fifth-Grade Longitudinal Scale .............................................................................. 3-29 3.5.1 Evaluating Common Item Functioning ............................... 3-30 3.5.2 Concurrent Calibration and Chain Linking ......................... 3-31 3.5.2.1 Methodology Approaches ................................... 3-32 3.5.2.2 Comparison of Approaches and Methodology Selection ........................................ 3-34 3.5.2.3 Chain-Linking Methodology ............................... 3-35 3.5.3 Computation of Final Scale Scores ..................................... 3-36 4 DEVELOPMENT OF THE TWO-STAGE COGNITIVE ASSESSMENT TEST FORMS ....................................................................... 4-1 4.1 Development of the Item Pool ............................................................ 4-1 4.1.1 Field-test Design ................................................................. 4-1 4.1.2 Methods Used to Analyze Data for Design of the National Assessments .......................................................... 4-2 4.1.3 Criteria Guiding the Selection of Items for the National Assessments .......................................................... 4-5 iv Chapter Page 4.2 Reading ............................................................................................... 4-8 4.2.1 Estimated Ability Levels for the ECLS-K:2011 National Samples and Target Ranges for Item Difficulties .......................................................................... 4-8 4.3 Mathematics ........................................................................................ 4-13 4.3.1 Estimated Ability Levels for the ECLS-K:2011 National Samples and Target Ranges for Item Difficulties .......................................................................... 4-13 4.4 Science ...............................................................................................