Marcia Invernizzi • Connie Juel • Linda Swank • Joanne Meier University of Virginia • Curry School of Education K Technical Reference For questions about PALS-K, please contact: Phonological Awareness Literacy Screening (PALS) 1-888-UVA-PALS (1-888-882-7257) or (434) 982-2780 Fax: (434) 982-2793 e-mail address: [email protected] • Web site: http://pals.virginia.edu ©2004–2015 by Te Rector and Te Board of Visitors of the University of Virginia. All Rights Reserved. Graphic Design: Branner Graphic Design Printed in the United States of America K Technical Reference Tis document is a supplement to the PALS-K Administration & Scoring Guide. Marcia Invernizzi • Connie Juel • Linda Swank • Joanne Meier Virginia State Department of Education University of Virginia • Curry School of Education Acknowledgments Development of the Phonological Awareness Literacy Screening (PALS-K) has been supported by the Virginia Department of Education through Virginia’s Early Intervention Reading Initiative. Without the support provided by the Department, the test devel- opment activity required for this assessment would not be possible. Te PALS Ofce would like to thank Dr. Randall Robey, Dr. Timothy Landrum, and Dr. Tonya Moon at the University of Virginia for their contribu- tions to the technical adequacy of PALS-K. Tanks go also to division representatives, principals, and teachers throughout Virginia who have participated in pilots. Tanks to their participation, the PALS ofce is able to ensure that classroom teachers have a literacy screening tool with good evidence of reli- ability and validity. We also wish to thank several generations of graduate students from the McGufey Reading Center who have contributed to the devel- opment of PALS-K tasks. Section I 5 Phonological Awareness Literacy Screening for Kindergarten (PALS-K) 5 Purposes, Uses, and Limitations 5 Overview 6 Background 6 Virginia’s Early Intervention Reading Initiative (EIRI) 7 Funding for Virginia’s Early Intervention Reading Initiative 7 Virginia’s Standards of Learning (SOL) and PALS Section II 9 Description of PALS-K 7 Domains 7 Scoring 8 Forms Section III 11 Item Development and Field-Testing 11 Phonological Awareness Tasks 11 Rhyme and Beginning Sound Awareness 13 Literacy Tasks 13 Alphabet Knowledge 14 Letter-Sound Knowledge 15 Concept of Word 17 Word Recognition in Isolation 18 Feedback from the Field 18 Outside Review 19 Advisory Review Panel 19 External Review Section IV 20 Establishing Summed Score Criteria and Benchmarks 21 Benchmarks and Discriminant Analysis (DA) Section V 22 Technical Adequacy 23 Broad Representation of Students 23 Pilots 23 Summary Statistics 24 Reliability 26 Test-retest Reliability 26 Subtask Reliability 26 Inter-rater Reliability 29 Internet Data Entry Reliability 29 Validity 29 Content Validity 30 Criterion-related Validity 31 Construct Validity 35 Diferential Item Functioning Section VI 36 Summary Section VII 37 References Section VIII 39 Endnotes Section I Phonological Awareness Literacy Screening for Kindergarten 5 Section I Phonological Awareness Literacy Screening for Kindergarten (PALS-K) In this section we never sufcient when making high-stakes decisions • provide an overview of the purpose and use of such as summer school placement or retention. PALS; • provide an overview of Virginia’s Early Intervention Reading Initiative (EIRI); Overview • describe the way that funding is allocated for divisions that participate in the EIRI; Consisting of three screening instruments, the • show how PALS supports Virginia’s Standards of Phonological Awareness Literacy Screening Learning (SOL); (PALSPreK, PALS-K, and PALS Plus for grades 1–8), • describe briefly the PALS-K instrument. measures young children’s knowledge of important literacy fundamentals, including: More detailed information about the instrument is • phonological awareness; available from our website (pals.virginia.edu). • alphabet knowledge; • knowledge of letter sounds; • spelling; Purposes, Uses, and Limitations • concept of word; • word recognition in isolation. Te major purpose of the Phonological Awareness Literacy Screening for Kindergarten (PALS-K) is to Te major purpose of PALS is to identify students identify students who perform below grade-level who are performing below grade-level expectations expectations in several important literacy funda- in these areas and may be in need of additional mentals, and thus are at risk of reading difculties reading instruction beyond what is typically pro- and delays. As a diagnostic tool, PALS-K can be vided to developing readers. Note that meeting the used to assess what students already know about the Summed Score benchmark does not imply that the English writing system and what they need to learn student is on grade level, but only that the student to become readers. PALS-K has demonstrated good met the level of minimal competency necessary to evidence of reliability and construct, concurrent, and beneft from typical classroom literacy instruction. A predictive validity. secondary and logical extension of this goal is to pro- vide teachers with explicit information about what However, like any other assessment tool, PALS-K is their students know of these literacy fundamentals so just one means to assess a student’s overall literacy that they can more efectively tailor their teaching to competence. Other important information includes their students’ needs. additional early literacy assessment data, parent information, the child’s interest in books, and teacher Tis Technical Reference includes a description of judgment. While PALS-K provides reliable screening the background and rationale underlying PALS-K, for developmental milestones in literacy acquisition, the process of task and item development and one measure of an emergent reader’s performance is feld-testing, and the technical adequacy of the 6 PALS-K Technical Reference instrument (validity and reliability). In preparing ment. Many of the same conditions from the earlier this Technical Reference, we followed the Standards initiative apply: for Educational and Psychological Testing (1999), • All students in kindergarten through second grade prepared jointly by the American Educational must be screened annually; Research Association (AERA), the American • All students not meeting the benchmark for their Psychological Association (APA), and the National grade level must receive, in addition to regular Council on Measurement in Education (NCME). classroom instruction, intervention services; Explicit instructions for the administration and • All students in kindergarten through second grade scoring of PALS-K are included in a separate who receive intervention services must be assessed PALS-K Administration and Scoring Guide. Te again during the frst screening period following results for the statewide screening for each cohort the intervention. (Note that third-grade students are available in separate annual reports. are only screened in the fall if they are new to Virginia schools, or if they received intervention services over the summer; spring screening for Background third-graders is optional); • All screening results must be reported to the PALS Te Phonological Awareness Literacy Screening Ofce at the University of Virginia via the PALS for Kindergarten (PALS-K) is the state-provided website (pals.virginia.edu). screening tool for the Virginia Early Intervention Reading Initiative (EIRI), and is designed for use In 2002, the Virginia Department of Education in kindergarten. Te purpose of the EIRI is to changed the screening period for the EIRI from fall reduce the number of children with reading prob- to spring. Also, a high benchmark was added for lems through early detection and to accelerate their frst- and second-grade students clearly performing learning of research-identifed emergent and early above grade-level expectations. Students attaining literacy skills. this high benchmark would no longer need to be screened for the EIRI. Tese changes enhance the Virginia’s Early Intervention Reading EIRI by: Initiative (EIRI) • allowing intervention services for all students Te 1997 Virginia Acts of Assembly, Chapter 924, in frst, second, and third grades to start at Item 140, initially established the Early Intervention the beginning of the school year or during the Reading Initiative. Te state initiative allocated summer; funds to help participating school divisions identify • eliminating the problem created by fall screening children in need of additional instruction and to for year-round schools and schools that start provide early intervention services to students with before Labor Day; diagnosed needs. Participating school divisions were • allowing Title I to use PALS as their screening allowed to implement the initiative in either kinder- instrument for reading services, thereby garten or frst grade. eliminating the use of a second screening; • reducing the amount of time required for In the 2000-01 legislative session, the Governor and screening. the General Assembly provided funding to expand the EIRI to third grade. Participating school divisions An EIRI timeline for PALS screening is shown in are now required to screen students in kindergarten Table 1. through third grade either with a diagnostic assess- ment approved by the Virginia Department of Education or with PALS, the state-provided instru- Section II Description of PALS-K 7 Section II Description of PALS-K In this section we briefly describe the parts of Domains PALS-K. Table 3 outlines the conceptual framework