2016 No. 062 Delaware System of Student Assessment and Maine Comprehensive Assessment System: SAT Alignment to the Common Core State Standards Final Report Prepared Delaware Department of Education Prepared Contract Number #00000475 for: Maine Department of Education under: Contract Number #00000478 Authors: Yvette Nemeth Date: December 16, 2016 Hillary Michaels Carrie Wiley Jing Chen Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314 | Phone: 703.549.3611 | Fax: 703.549.9025 | humrro.org Delaware System of Student Assessment and Maine Comprehensive Assessment System: SAT Alignment to the Common Core State Standards Table of Contents Executive Summary ................................................................................................................... iv Chapter 1: Introduction ............................................................................................................... 1 Overview ................................................................................................................................ 1 Organization and Contents of the Report ............................................................................... 1 Chapter 2: Alignment Study Design and Methodology ............................................................... 2 Alignment of Assessments and Standards on Content and Performance ............................... 2 SAT Overview .................................................................................................................... 2 Prior Studies Evaluating the Alignment of the SAT to the CCSS ........................................ 3 Challenges of Evaluating the Alignment of the SAT to the CCSS ....................................... 4 HumRRO Alignment Methodology ...................................................................................... 5 Special Study Evaluating the SAT Using the CCSSO Criteria. ........................................... 7 Alignment Study Procedure .................................................................................................... 9 Panelists............................................................................................................................. 9 Training .............................................................................................................................10 Test Security .....................................................................................................................10 Materials............................................................................................................................10 Procedures ........................................................................................................................11 Ratings ..............................................................................................................................12 Mid-Course Adjustments ...................................................................................................12 Chapter 3: Alignment Results ....................................................................................................14 Criterion 1: Items Represent Intended Content .....................................................................14 Comparison of College Board Ratings and Average Panelist Ratings ...............................21 Essay Results ...................................................................................................................22 Criterion 2: Items Represent Intended Categories .................................................................25 Criterion 3: Item DOK Represents Test Specifications ..........................................................27 Panelist Agreement ...........................................................................................................27 College Board DOK and Rigor Distributions ......................................................................28 Item DOK Compared to CCSS DOK ..................................................................................30 Criterion 4: Item Sufficiency for Category Reporting ..............................................................32 Chapter 4: Special Study Comparing the SAT to Selected CCSSO High Quality Assessment Results..................................................................................................................36 Chapter 5: Panelist Debriefing and Evaluation Survey ..............................................................41 Chapter 6: Summary and Recommendations ............................................................................43 SAT Alignment Summary ......................................................................................................43 Criterion 1: Items Represent Intended Content ..................................................................43 Table of Contents i Criterion 2: Items Represent Intended Categories .............................................................44 Criterion 3: Item DOK Represents Test Specifications.......................................................44 Criterion 4: Item Sufficiency for Category Reporting ..........................................................44 Special Study Evaluating the SAT Using the CCSSO Criteria Summary ...........................44 Suggestions ..........................................................................................................................45 References ...............................................................................................................................47 Appendix A. Sample Panelist Alignment Review Materials ..................................................... A-1 Appendix B. Summary of Item Workbook Ratings ................................................................... B-1 Appendix C. Graphical Representations of the Identified and Verified CCSS by College Board and Panelists ................................................................................................................ C-1 Appendix D. List of CCSS in Reading, Writing, and Language Arts ......................................... D-1 Appendix E. List of Mathematics Grade 11-12 CCSS.............................................................. E-1 Appendix F. List of Panelist Identified Off-Grade Level Mathematics CCSS ............................ F-1 Appendix G. CCSS Identified by College Board and Panelists ................................................G -1 Appendix H. SAT Alignment Workshop Evaluation Form ........................................................ H-1 List of Tables Table 1. Professional and Demographic Characteristics of Panelists ......................................... 9 Table 2. Item Counts Reviewed by Panelists ............................................................................11 Table 3. Math CCSS Organization ............................................................................................14 Table 4. English Language Arts CCSS Organization ................................................................14 Table 5. Mean Percentage of Items with No Link to Standards .................................................17 Table 6. Number of College Board Identified Standards ............................................................17 Table 7. Test and Subscore Distribution of Items ......................................................................26 Table 8. Rigor Levels of Math Items ..........................................................................................28 Table 9. Cognitive Demand Alignment of Items and Standards .................................................31 Table 10. Subscore Intercorrelations .........................................................................................33 Table 11. Coefficient Alphas by Subscores ...............................................................................33 Table 12. CFA Fit Indices ..........................................................................................................34 Table 13. Average Standardized Factor Loadings .....................................................................35 Table 14. CFA Factor Score Intercorrelations ...........................................................................35 Table 15. ELA CCSSO High Quality Assessment Results ........................................................36 Table 16. Additional Essay Criteria ...........................................................................................38 Table 17. Math CCSSO High Quality Assessment Results .......................................................39 Table 18. Means and Standard Deviations of the ELA Panelists’ General Alignment Opinions ..........................................................................................................................41 Table of Contents ii Table 19. Means and Standard Deviations of the Math Panelists’ General Alignment Opinions ..........................................................................................................................42 Table 20. Means and Standard Deviations for the Process and Procedure Survey Questions by Panelist Group ...........................................................................................42 List of Figures Figure 1. Panelist agreement with item to primary standard content alignment. ........................16 Figure 2. Panelist agreement with item to standard content alignment. .....................................18