World-class Instructional Design and Assessment TM Development and Field Test of WIDA MODEL Grades 6–8 and 9–12 Prepared by: CAL/WIDA Partnership Activities Psychometrics/Research Team Center for Applied Linguistics July 17, 2014 Copyright Notice This document is owned by the Board of Regents of the University of Wisconsin System on behalf of the WIDA Consortium. This document is protected by United States copyright laws and may not be reproduced, modified, or distributed, including posting, without the prior written permission of the Wisconsin Center for Education Research (WCER) and the Board of Regents of the University of Wisconsin System. This document is for your personal, noncommercial use only. You may not alter or remove any trademark, copyright, or other notice from copies of this booklet. Fair use of this document includes reproduction for the purpose of teaching (including multiple copies). If you are not sure whether your use of this booklet falls within fair use or if you want permission to use the copyrighted WIDA document for purposes other than personal or fair use, please contact the WIDA Help Desk at [email protected] or 1-866-276-7735. © 2014 Board of Regents of the University of Wisconsin System, on behalf of the WIDA Consortium—www.wida.us. 2014 WIDA Consortium Members Alabama Michigan Northern Mariana Islands Alaska Minnesota Oklahoma Colorado Mississippi Pennsylvania Delaware Missouri Rhode Island District of Columbia Montana South Carolina Georgia Nevada South Dakota Hawaii New Hampshire Tennessee Illinois New Jersey Utah Kentucky New Mexico Vermont Maine North Carolina Virginia Maryland North Dakota Wisconsin Massachusetts Wyoming 2014 Non-member States Formally Adopting the WIDA ELD Standards Idaho Indiana WIDA advances academic language development and academic achievement for linguistically diverse students through high quality standards, assessments, research, and professional development for educators. WIDA’s vision is to be the most trusted resource in the education of Pre-kindergarten through Grade 12 language learners. Executive Summary The World-class Instructional Design and Assessment (WIDA) Measure of Developing English Language (MODEL)™ is an off-the-shelf series of academic English language proficiency assessments for English Language Learners (ELLs) in Kindergarten through Grade 12. The test for Kindergarten was developed from 2006–2008 and became available to WIDA Consortium members and non-members in October 2008. The test for Grades 1–2 and the test for Grades 3–5 were developed from 2008–2010 and became available in August 2010. The test for Grades 6–8 and the test for Grades 9–12 were developed from 2009–2011 and became available in September 2011. The purpose of this technical report is to describe the development and field test of WIDA MODEL for Grades 6–8 and 9–12. The development and field tests of WIDA MODEL for Kindergarten and Grades 1–2 and 3–5 are discussed in separate technical reports. This report about WIDA MODEL for Grades 6–8 and 9–12 provides background information about the purposes, format, and scores (Chapter 1); describes how the tests were developed (Chapter 2) and field tested (Chapter 3), presents technical properties of the field tested items and tasks (Chapter 4); explains the linking of WIDA MODEL to the WIDA English Language Proficiency (ELP) levels1 (Chapter 5); and provides an argument-based framework to support the validity and reliability of the test (Chapter 6). Summary Highlights Background Information (Chapter 1) WIDA MODEL is a series of English language proficiency assessments, which evaluates ELL students’ academic English language proficiency in the four language domains of Speaking, Listening, Reading, and Writing. All items and tasks in those sections are aligned to the WIDA ELP Standards (i.e., Social and Instructional Language, Language of Language Arts, Language of Mathematics, Language of Science, and Language of Social Studies). WIDA MODEL can be used to determine the academic English language proficiency level of students who are new to a school or to the U.S. school system and to identify and place students who are candidates for English as a Second Language (ESL) and/or bilingual services. In addition, in states that are members of the WIDA Consortium, WIDA MODEL may be used to determine tier placement on the WIDA ACCESS for ELLs® test (hereafter referred to as ACCESS), to track students’ proficiency at an 1 WIDA MODEL was developed using the 2007 ELP Standards (Gottlieb, Cranley, & Cammilleri, 2007). These standards were updated in the 2012 Amplification of the English Language Development Standards, Kindergarten- Grade 12, which can be found on the WIDA website (www.wida.us/standards/eld.aspx). iii additional time during the school year, and to replace the WIDA-ACCESS Placement Test (W- APT)™. WIDA MODEL contains both a full-length assessment (MODEL) and a Screener (MODEL Screener), which includes all tasks from the Speaking and Writing sections of MODEL but fewer items in the Listening and Reading sections. MODEL Screener was developed because stakeholders saw a need for a less time-consuming test that would still determine students’ language proficiency levels, tier placement on ACCESS, and need for ELL services. MODEL Screener, however, cannot be used to determine amount, type, or exiting of ELL services. In both MODEL and MODEL Screener, the Speaking section consists of constructed-response tasks that target progressively higher proficiency levels and are administered to individual students in an interview format. The Listening section in MODEL has multiple-choice items, is administered to individual students, and has Low, Mid, and High placement levels so students take only items that are appropriate for their proficiency level. The Reading section in MODEL is also multiple-choice and has Low, Mid, and High placement levels, but the section may be administered to individual students or to a group. The Listening and Reading sections in MODEL Screener have the same format and administration as in MODEL, but they contain fewer items and students are not placed into different levels. The Writing section in both MODEL and MODEL Screener contains two parts: Part A, which asks students to respond to open-ended questions that require only short answers; and Part B, which requires a more extended response that is administered only if students are able to meet expectations on Part A. The Writing section may be administered to individual students or to a group. After completing a test administration with a student, the test administrator uses lookup tables to convert raw scores to scale scores and proficiency levels. Scores are computed for all four language domains as well as three composite scores—Oral language (Listening and Speaking), Literacy (Reading and Writing), and Overall (all four domains). Proficiency level scores render a student’s scale score in terms of the WIDA ELP Standards. Test Development (Chapter 2) The WIDA MODEL tests for Grades 6–8 and 9–12 originally used folders of items that were retired or removed from the ACCESS operational test, resulting in a format that is patterned after the ACCESS tests for Grades 6–8 and 9–12. Due to limited availability of retired folders, additional folders were either selected from the ACCESS field test or were newly created by Center for Applied Linguistics (CAL) item writers or external item writers. iv All items underwent a series of reviews to ensure that items contained the appropriate content for each grade level and proficiency level, were appropriate and universal to people of different ethnic backgrounds, and did not contain cultural bias or sensitive topics. In addition, cognitive labs were held to collect information about administration procedures and times, accurate placement of students in Low, Mid, or High levels, quality of text and graphics, and the ability of items and tasks to elicit expected language. A number of quality checks, such as proofing and key checks, were conducted before the WIDA MODEL test forms were finalized. Field Test (Chapter 3) Field testing for Grades 6–8 and 9–12 was conducted with 1,256 students in 24 schools in four WIDA states—Illinois, Kentucky, Maine, and New Mexico—from November through December 2010. CAL hired field test administrators to assist with the testing of students. The field test administrators followed the same procedures, administration, and scoring guidelines as would be used for operational testing. Field Test Results (Chapter 4) Raw data for the Speaking, Listening, and Reading sections were entered and cleaned electronically. The items were scored dichotomously as correct or incorrect so the functioning of items could be analyzed psychometrically and total raw scores could be calculated. An outlier analysis was conducted on the data for the Listening and Reading domains to ensure that students that had discrepant performances on MODEL and ACCESS were not included in calibration and linking procedures. Following the removal of outliers, Rasch analyses revealed that, overall, the Speaking, Listening, and Reading items are productive for measurement and measure the intended construct. For the Writing sections, sets of writing samples were selected to calibrate test development staff at CAL and, later, outside consultants. After all raters were trained to reliably score the calibration samples, all writing samples were rated operationally. Linking WIDA MODEL to WIDA ELP Levels (Chapter 5) To facilitate the interpretation of WIDA MODEL scores, scores on the test were linked to scores on ACCESS so they can be understood in terms of the WIDA ELP Levels (Level 1 Entering through Level 5 Bridging). Linking studies were conducted in order to produce lookup tables, which show the proficiency level scores that correspond to students’ raw scores and scale scores for each grade, placement level, and domain. In the linking study for Listening and Reading, psychometric methods were used to link WIDA MODEL scores to ACCESS scores.