RESEARCH IN REVIEW 2016-1 A Framework Analysis TIMSS Advanced 2015 AND Advanced Placement Calculus & Physics Christopher Lazzaro, Lee Jones, David C. Webb, Ryan Grover, F. Tony Di Giacomo, and Katherine Adele Marino 00078_002 Christopher Lazzaro, Ph.D. – Director, Science Education, College Board Lee Jones, Ph.D. – Independent Consultant David C.Webb, Ph.D. – Associate Professor, Math Education, University of Colorado Boulder Ryan Grover – Ph.D. Candidate, University of Colorado, Boulder F.Tony Di Giacomo, Ph.D. – Director of International Research, College Board Katherine Adele Marino – Ph.D. Candidate, Pennsylvania State University About the College Board The College Board is a mission-driven not-for-profit organization that connects students to college success and opportunity. Founded in 1900, the College Board was created to expand access to higher education. Today, the membership association is made up of over 6,000 of the world’s leading educational institutions and is dedicated to promoting excellence and equity in education. Each year, the College Board helps more than seven million students prepare for a successful transition to college through programs and services in college readiness and college success — including the SAT® and the Advanced Placement Program®. The organization also serves the education community through research and advocacy on behalf of students, educators and schools. For further information, visit www.collegeboard.org. ©2016 The College Board. College Board, Advanced Placement Program, SAT, and the acorn logo are registered trademarks of the College Board. ACES and Admitted Class Evaluation Service are trademarks owned by the College Board. Visit the College Board on the Web: www.collegeboard.org. For more information on College Board research and data, visit www.collegeboard.org/research. Contents Executive Summary ................................................................................................ 5 Introduction ............................................................................................................. 8 Methodology .................................................................................................... 8 Population ........................................................................................................ 9 Purpose ............................................................................................................. 9 Overview of AP and TIMSS Advanced Frameworks ..........................................10 AP Physics 1 and 2 Framework.....................................................................10 AP Calculus Framework ................................................................................13 TIMSS Advanced 2015 Frameworks ............................................................16 Alignment Methodology ........................................................................................20 Categorical Concurrence — Content Areas ................................................20 Depth of Knowledge — Cognitive Domains ................................................22 Balance of Representation — Item Distribution ..........................................27 Alignment Analysis ................................................................................................29 Physics ............................................................................................................29 Calculus ...........................................................................................................44 Findings and Conclusion .......................................................................................61 Overall Summary ............................................................................................61 Categorical Concurrence ...............................................................................61 Depth of Knowledge .......................................................................................61 Balance of Representation .............................................................................61 Conclusion ..............................................................................................................62 References ...............................................................................................................63 Tables Table 1. Big Ideas for AP Physics 1 and AP Physics 2 ................................................................11 Table 2. Big Idea 1: Objects and Systems. ...................................................................................11 Table 3. AP Science Practices .......................................................................................................12 Table 4. Target Percentages of the TIMSS Advanced 2015 Mathematics Assessment ..........18 Table 5. Target Percentages of the TIMSS Advanced 2015 Physics Assessment ...................19 Table 6. TIMSS Advanced Physics Cognitive Domains .............................................................23 Table 7. TIMSS Advanced Mathematics Cognitive Domain and Thinking Processes ...........24 Table 8. Examples of AP Calculus Tasks Within TIMSS Advanced Mathematics Framework ..............................................................................................................25 Table 9. Number and Description of Tasks in Calculus AB and BC Exams ..............................26 Table 10. Emphasis on Cognitive Domains Across Frameworks ..............................................39 Table 11. Total Number of Tasks Corresponding to Each Domain in TIMSS Advanced Mathematics ....................................................................................................54 Table 12. Total Number of Tasks Corresponding to Each Domain in TIMSS Advanced, Expanded ........................................................................................................54 Figures Figure 1. Hierarchy of the AP Physics Framework ....................................................................13 Figure 2. Evaluation of degree of alignment: mechanics and thermodynamics ......................30 Figure 3. Evaluation of degree of alignment: electricity and magnetism .................................31 Figure 4. Evaluation of degree of alignment: wave phenomena and atomic and nuclear physics ...............................................................................................................................32 Figure 5. Topic area weighting in TIMSS Advanced 2015 Physics and AP Physics 1. ...........41 Figure 6. Topic area weighting in TIMSS Advanced 2015 Physics and AP Physics 2. ...........42 Figure 7. Topic area weighting in TIMSS Advanced 2015 Physics and AP Physics 1 and 2 combined .......................................................................................................42 Figure 8. Evaluation of degree of alignment: algebra .................................................................46 Figure 9. Evaluation of degree of alignment: calculus ................................................................47 Figure 10. Evaluation of degree of alignment: geometry............................................................48 Figure 11. Example of an application task from AP Calculus Exam..........................................55 Figure 12. Example of a reasoning task from AP Calculus Exam ..............................................56 Figure 13. Example of a justification task from AP Calculus Exam ...........................................57 Figure 14. Example of a recognize task from AP Calculus Exam ..............................................58 Figure 15. Comparison of topic area weighting in TIMSS Advanced Mathematics and AP Calculus AB .......................................................................................................................59 Figure 16. Comparison of topic area weighting in TIMSS Advanced Mathematics and AP Calculus BC .......................................................................................................................59 TIMSS Advanced 2015 and Advanced Placement Executive Summary While the results of the TIMSS Advanced 2015 assessments are not yet known, in recent years U.S. students have underperformed relative to their international counterparts on international assessments such as TIMSS and PISA. While international assessment data are frequently used to compare performance cross-culturally, the analysis of these data often lacks adequate contextualization for country-by-country comparisons. To ensure maximum utility of the assessments, an examination of each participating country’s curriculum content and standards in relation to those on the international assessment is critical. Only then can one make cross-cultural comparisons effectively. The current study investigates the alignment between AP Physics and AP Calculus frameworks as compared to the TIMSS Advanced Physics and Mathematics frameworks. Due to the rigor of AP course work and the performance of AP students versus non-AP students on college course work, we expect AP student performance on TIMSS Advanced may exceed the performance of the general population. Moreover, meaningful comparisons of country achievement rankings are not possible without a study that considers alignment, and thus, there may also be additional compelling findings from the future performance report that can make a potentially significant contribution to the literature, and to policy discussions. The methodology applied in this