State Board of Education Topic Summary s5

STATE BOARD OF EDUCATION – TOPIC SUMMARY Topic: Math Achievement Level Descriptors, Assessment Development Date: March 11, 2010 Staff/Office: Doug Kosty, Tony Alpert / Office of Assessment and Information Services Action Requested: Informational Only Adoption Later Adoption Adoption/Consent Agenda

ISSUE BEFORE THE BOARD: In preparation for this summer’s math Standards Setting, the State Board needs to adopt Policy Definitions for Achievement Levels and Mathematics Achievement Level Descriptors.

BACKGROUND: The State Board adopted new High School Mathematics Academic Content Standards in June of 2009. The Board had previously adopted new K-8 standards in December of 2007. Following the adoption of these Content Standards is a review of the Achievement Standards or cut scores. In order to ensure that Oregon’s state educational standards stay current and in line with national and international benchmarks, ODE staff and stakeholders will review the Achievement Standards (or cut scores) used to determine whether or not a student meets standards. In August 2010, ODE staff will convene stakeholders to set new Oregon Mathematics Achievement Standards. Achievement standards were last set in 2006. Best practices now include adoption of cross-subject cross-grade General Policy Definitions and grade-level Mathematics Achievement Level Descriptors prior to standards setting. ODE also recommends adoption of cross-grade Math Policy Definitions for additional clarity.

Math Standards Setting Timeline:

Date Topic Status December 2007 New K-8 Math Standards Adopted by State Board June 2009 New High School Math Standards Adopted by State Board January 2010 General Policy Definitions & Math Policy Reviewed by State Board Definitions March 2010 General Policy Definitions & Math Policy Adoption needed by State Definitions Board March 2010 High School Math Achievement Level Review needed by State Board Descriptors April 2010 High School Math Achievement Level Adoption needed by State Board Descriptors April 2010 Grades 3-8 Math Achievement Level Descriptors Review needed by State Board May 2010 Grades 3-8 Math Achievement Level Descriptors Adoption needed by State Board August 2010 Math Standards Setting - Math Achievement Dependent on previous State Level Descriptors and Policy Definitions refined Board action October 2010 New Math Cuts Scores and refined Math Adoption needed by State Board Achievement Level Descriptors and Policy Definitions 2010-11 school year Testing using new content standards and cut Dependent on previous State scores Board action

The timeline above places State Board adoption of the new cut scores in October, the same timeframe as the start of the 2010-11 testing year. This will minimize the need to retroactively apply cut scores to student tests.

If the Standards Setting is delayed past August, ODE will either have to:  Retroactively apply cut scores to student scores, or  Assess students on the previous math standards while teaching to the new standards for another year. For more information and background on the standards setting process, please see Appendix A.

1 National Best Practice – Policy Definitions for Achievement Levels

Over the past months, ODE staff have worked with stakeholders to refine the General Policy Definitions and the Mathematics Policy Definitions first presented to the Board in January. In this process, ODE staff are following a model presented by national expert Marianne Perie from the National Center for the Improvement of Educational Assessment. Her work is referenced further in the attached appendices.

In order to ensure that Oregon standards will be consistent across grades and subjects, ODE has worked with education partners to develop, review, and refine the following General Policy Definitions:

General Policy Definitions for Achievement Levels:  Level 1: Students do not demonstrate mastery of grade-level knowledge and skills required for proficiency.  Level 2: Students demonstrate partial mastery of grade-level knowledge and skills required for proficiency.  Level 3: Students demonstrate mastery of grade-level knowledge and skills required for proficiency.  Level 4: Students demonstrate mastery of grade-level knowledge and skills exceeding the requirement for proficiency.

These definitions would apply to all grades and all subjects to provide consistency across the system. The development of such General Policy Definitions is supported by national best practices.

Subject specific Policy Definitions foster the process of developing consistent achievement level descriptors across grades. The following Math Policy Definitions would apply to all grades and would serve as the basis for similar statements in the other content areas.

Mathematics Policy Definitions for Achievement Levels:  Level 1: Students demonstrate limited mastery of mathematical knowledge and skills through the direct application of a concept or procedure in simplified and familiar situations with occasional success.  Level 2: Students demonstrate partial mastery of mathematical knowledge and skills through the direct application of concepts and procedures in familiar situations with regular success. They are able to explain some of their steps.  Level 3: Students demonstrate mastery of mathematical knowledge and skills through selecting from an assortment of strategies and integrating concepts and procedures in a variety of situations with consistent success. They are able to explain steps and procedures.  Level 4: Students demonstrate mastery of mathematical knowledge and skills through the use of multiple reasoning strategies and apply them in new and complex situations with consistent success. They are able to analyze their strategies and solutions. For more information and background on General and Math Policy Definitions for Achievement Levels, please see Appendix B. For details on stakeholder feedback and the review process, please see Appendix C.

Grade Specific Achievement Level Descriptors

Revision of the Achievement Level Descriptors is part of the implementation of the new mathematics assessments. The Achievement Level Descriptors (ALDs) at each grade will influence the upcoming achievement standards setting for mathematics, scheduled for August 2010. In order for Oregon’s standards to be accepted for federal accountability purposes, we must submit documentation to the U.S. Department of Education (U.S. DOE), and the ALDs are part of this required documentation. Achievement Level Descriptors are also used for school accountability purposes. Oregon’s new ALDs are being drafted by a team of Oregon math education experts. The Policy Definitions and new ALDs are being reviewed by education stakeholders,

2 those with mathematics expertise, and others. Below are the High School Achievement Level Descriptors. ODE staff will present the ALDs for the other grades in April.

3 High School Achievement Level Descriptors

4 Proposed Level 1* Level 2* Level 3* Level 4* Levels Students do not Students demonstrate Students demonstrate Students demonstrate General demonstrate mastery of partial mastery of grade- mastery of grade-level mastery of grade-level Policy grade-level knowledge level knowledge and knowledge and skills knowledge and skills Definitions and skills required for skills required for required for proficiency. exceeding the requirement (Apply to all proficiency. proficiency. for proficiency. grades and all subjects) Students demonstrate Students demonstrate Students demonstrate Students demonstrate Mathematics limited mastery of partial mastery of mastery of mathematical mastery of mathematical Policy mathematical mathematical knowledge knowledge and skills knowledge and skills Definitions knowledge and skills and skills through the through selecting from an through the use of multiple (Apply to all through the direct direct application of assortment of strategies reasoning strategies and grades) application of a concept concepts and and integrating concepts apply them in new and or procedure in procedures in familiar and procedures in a complex situations with simplified and familiar situations with regular variety of situations with consistent success. They situations with success. They are able consistent success. They are able to analyze their occasional success. to explain some of their are able to explain steps strategies and solutions. steps. and procedures.

 Often performs  Performs computations  Applies efficient  Fluently applies efficient Mathematics computations with real with real numbers strategies for performing strategies for performing Achievement numbers accurately. accurately. Orders or computations with real computations with real Level Inconsistently orders locates real numbers numbers. Orders and numbers. Able to justify Descriptors or locates real on a number line. locates real numbers on the result using numbers on a number  Evaluates and a number line. mathematical properties. High line. simplifies expressions  Consistently evaluates,  Consistently evaluates, School  Inconsistently involving real numbers creates, and determines creates, justifies, and Algebra simplifies expressions and/or algebraic equivalent expressions determines equivalent involving real numbers symbols. This excludes involving real numbers expressions involving real and/or algebraic factoring and complex and /or algebraic numbers and/or algebraic symbols. expressions. symbols. symbols.  Sometimes solves  Solves problems  Represents linear  Represents, solves, linear equations given involving linear relationships and solves interprets the meaning of, the equation. Identifies equations using tables, problems involving and converts among slope from a graph. graphs, and symbols. systems of linear representations of linear Solves systems of Determines slope. equations using tables, equations and inequalities. linear equations using Often solves linear graphs, and symbols. Solves problems involving tables and graphs. inequalities and graphs Solves linear inequalities systems of linear  Identifies and the solution. and graphs the solution. inequalities using tables, inconsistently graphs  Identifies and graphs Able to work with graphs, and symbols. linear, quadratic, and linear, quadratic, and function notation.  Distinguishes between, exponential functions exponential functions  Distinguishes between, manipulates, graphs, and using tables, graphs, using tables, graphs, manipulates, graphs, applies linear, quadratic, and equations. and equations. and applies linear, and exponential functions, Students can Students identify the quadratic, and including making sometimes find the domain and range from exponential functions in inferences or predictions, domain and range a graph. routine and non-routine using tables, graphs, and given the graph.  Solves routine situations using tables, equations. Students  Sometimes solves quadratic functions graphs, and equations. identify the domain and quadratic functions or using one method and Students identify the range from a table, a understands the can explain why the domain and range from graph, an equation, or meaning of the method works. a table, a graph or an within a context. solutions. Identifies the axis of equation. Solves quadratic functions symmetry graphically.  Solves quadratic graphically and algebraically functions graphically and and can communicate why a algebraically and is able method was chosen and its to explain the relationship to the solution. relationship between the two strategies. Determines the vertex 5 and axis of symmetry graphically and algebraically.  Inconsistently applies  Identifies and  Understands and  Understands and applies Mathematics properties of two and compares properties of applies properties of properties of geometric Achievement three dimensional two and three geometric figures and figures and analyzes Level figures. dimensional figures. relationships between relationships between Descriptors  Inconsistently  Determines area, figures in two and three figures in two and three determines area, surface area, and/or dimensions. dimensions. High surface area, and/or volume given  Determines area,  Determines area, surface School volume given dimensions. Sometimes surface area, and/or area, and/or volume for Geometry dimensions. Seldom solves for missing volume whether or not non-routine figures. Solves solves for missing dimensions. all dimensions are given. for missing dimensions that dimensions.  Uses some theorems Solves related context- require multiple equations.  Confused by the use and can solve problems based problems.  Consistently and efficiently of theorems and can related to congruent  Consistently integrates integrates the use of sometimes solve and similar figures in the use of theorems theorems and/or algebraic problems related to two and three and/or algebraic equations to solve and congruent and similar dimensions. equations to solve justify relationships between figures in two and  Uses coordinate problems related to congruent and similar three dimensions. geometry to justify congruent and similar figures in two and three  Sometimes justifies properties and solve figures in two and three dimensions. properties and solve problems involving two dimensions, including  Uses coordinate geometry problems using dimensional figures right triangle to justify properties, explain coordinate geometry with numeric trigonometry. conjectures, and solve when the two coordinates.  Uses coordinate problems involving two dimensional figures  Identifies and performs geometry to justify dimensional figures with have numeric single transformations properties, explain numeric and symbolic coordinates. of geometric figures on conjectures, and solve coordinates.  Inconsistently a coordinate plane. problems involving two  Fluently identifies and identifies and dimensional figures with performs single and performs single numeric coordinates. composite transformations transformations of  Identifies and performs (more than 2) of geometric geometric figures on a single and composite figures on a coordinate coordinate plane. transformations (2) of plane. geometric figures on a coordinate plane.  Analyzes survey  Analyzes the strengths  Determines and  Determines and analyzes Mathematics methods and and limitations of analyzes survey survey methods and Achievement evaluates data-based survey methods and methods and evaluates evaluates data-based Level reports with gaps in evaluates data-based data-based reports. reports and is able to Descriptors understanding. reports looking at the  Compares, analyzes, communicate appropriate  Inconsistently way the data was draws conclusions suggestions for High compares and draws analyzed and about, and interprets improvement. School conclusions about displayed. independent data sets  Compares, analyzes, Statistics independent data sets  Compares and draws and data in graphical draws conclusions about, and and data in graphical conclusions about displays, including and interprets independent Probability displays, including independent data sets central tendencies, data sets and data in central tendencies and and data in graphical range, and line of best graphical displays, including range. displays, including fit, when appropriate. central tendencies, range,  Inconsistently central tendencies,  Computes, analyzes, and line of best fit, when computes and range, and line of best and interprets theoretical appropriate, and is able to interprets simple fit, when appropriate. and experimental explain how a change in the theoretical and  Computes and probabilities. Determines data set would alter the experimental interprets simple the sample space of a display and central probabilities. theoretical and probability experiment. tendencies. Sometimes experimental  Computes, analyzes, and determines the probabilities. Often interprets theoretical and sample space of a determines the sample experimental probabilities probability experiment. space of a probability with complex sample experiment. spaces.

6 POLICY QUESTIONS:  Should Oregon create Policy Definitions for Achievement Levels to provide consistency across grades and subjects?  Should Oregon raise its Achievement Standards to align more closely with national and international standards?  How high should the bar be set? What should we expect of students earning a high school diploma? What should be expected for the purpose of school and district accountability?  Should Oregon change the names of the achievement levels from Does Not Meet, Nearly Meets, Meets, Exceeds to something new?

STAFF RECOMMENDATION:

ODE staff recommend adoption of the General Policy Definitions and the Math Policy Definitions. Please note: the Board is asked to adopt the current versions of the Policy Definitions which will be used in this summer’s Standards Setting. During Standards Setting and ongoing stakeholder engagement, revisions to these Policy Definitions may be recommended. Any updates to these statements will be provided to the Board in October for review.

ODE staff recommend adoption of the High School Achievement Level Descriptors in April.

Adoption of Mathematics Achievement Level Descriptors (at all grades) are needed for agency work flow by May. The latest possible adoption date is June, 2010. Final adoption of ALDs and Achievement Standards (cut scores) is needed in October 2010.

7 Appendix A: Additional Information and Background on the Standards Setting Process. The following information was presented to the State Board at their January meeting and is provided here as additional context.

What is Oregon’s Standards Setting Process? Large scale assessments such as the Oregon Assessment of Knowledge and Skills (OAKS) are designed to provide information to students, parents, educators, policymakers, and other stakeholders about what students know and can do in a particular content area. Scores are reported using two conventions: Achievement Levels and Scale Scores. Achievement Levels assist in describing to what degree students have met the expectations that Oregon policy makers establish for student achievement in regard to state content standards. Scale Scores describe student achievement in a systematic manner that can be used to describe student achievement and student growth in achievement

Over the coming months, Oregon will re-establish the achievement expectations we have for students on statewide assessments. Following best practices, ODE staff will engage Oregon educators and stakeholders to participate in a standards setting with attention focused on a reasonable cut score given external data about national and international benchmarks. During the standards setting, the participants will review the cut-scores that are best aligned with college-readiness, national and international benchmarks, and then give input as to whether or not these proposed scores are valid. Participants meet for two to three days and engage in structured conversations that include consideration of the following: Oregon content standards, the purpose of the achievement standards and their role in new graduation requirements, the General and Math Policy Definitions, the difficulty of the content as represented by the test items, and knowledge of Oregon students.

Prior to this standards setting, revised Achievement Level Descriptors are adopted by the State Board to help guide the discussion. This year, in alignment with Marianne Perie’s recommendations, ODE is also developing overarching Policy Definitions to help further align the standards across grades and subjects.

How Does This Work Relate to the Adoption of Common Core State Standards? ODE is working with other states around the country to ensure that the Common Core State Standards eventually adopted will be in the best interest of kids. At this time, ODE staff have some concerns about the current version under review. However, there is still much work to be done to finalize the Common Core State Standards, and once finalized these new standards will be implemented over a several year period. Additional work will be needed in the future to align these with Oregon’s standards and adjust cut scores as needed, but this work is still several years down the road. This summer, Oregon will submit a plan for adoption of the Common Core State Standards including details on how these standards will be adopted and phased-in.

Review of Achievement Levels Oregon currently has the following Achievement Levels: Does Not Yet Meet, Nearly Meets, Meets, and Exceeds. As we approach the adoption of new cut scores, we will determine the appropriate level of rigor at each of these levels. This is also a good time to reassess the number and names of our Achievement Levels. NCLB requires states to set a minimum of three achievement levels and Perie, in her article, recommends no more than four levels. It seems advisable, therefore, to maintain our current four-tiered system. Naming, however, takes more consideration. As Perie states, “The terms themselves carry meaning, even without further description; therefore, naming a level is the first step in defining performance…The words chosen express the values of the policymakers and thus should be selected carefully.” [Perie, 2008]

One question before the Board this winter/spring is whether Oregon should adopt new names for our Achievement Levels. New names may be particularly advisable if the Board decides to raise the rigor of the cut scores, as indicated by initial comparison of Oregon’s achievement levels with national benchmarks. For the purposes of this docket, the place holder names of Level 1, Level 2, Level 3, and Level 4 have been used. Level 3 would be considered a “Proficient” or “Meets” level.

In our outreach with the field, ODE staff have solicited feedback on several possible new names and that information is available in Appendix C.

8 Why Raise the Rigor? Comparisons to NAEP (National Assessment of Education Progress) indicate that at least at Elementary and Middle School, Oregon’s “Meets” is well below NAEP’s “Proficient.” The inclusion of NAEP (national) and PISA (international) test items in this spring’s OAKS field test will provide additional information as to how Oregon students do compared to their peers around the country and the world. This additional information could inform the standards setting in August and the resulting cut scores.

9 Appendix B: Additional Information and Background on the General and Math Policy Definitions and Achievement Level Descriptors The following information was presented to the State Board at their January meeting and is provided here as additional context.

What are General Policy Definitions for Achievement Levels? The General Policy Definitions provide an overarching definition for each achievement level. Policy Definitions describe how rigorous and challenging the Achievement Standards (cut scores) will be for the assessments. They are not linked directly to content but are more general statements that describe rigor across grade levels and content areas. These definitions will first be used in math and then science, social sciences, reading and writing. “The definitions are consistent across all grades and subjects and help ensure a similar level of rigor is implied by the performance level for each assessment.” [Perie, 2008]

What are Mathematics Policy Definitions? The Math Policy Definitions provide an overarching definition of each achievement level for the subject of math. These definitions apply to all grades and help provide additional consistent standards across grades. Similar definitions would be developed for the other content areas.

What are Achievement Level Descriptors (ALDs)? The Oregon Achievement Level Descriptors (ALDs) explain the knowledge and skills that students typically demonstrate at specific levels of the Oregon Assessment of Knowledge and Skills (OAKS) in each grade and in each subject. The ALDs define, for example, what a student is expected to know at the “Meets” level in 5th grade Mathematics. Marianne Perie of the National Center for the Improvement of Educational Assessment calls ALDs “the foundation of standard-setting activities as they provided the explanation for how student achievement differs from one level to the next.” [Perie, 2008]

Revision of Mathematics Achievement Level Descriptors Achievement Level Descriptors are based on excerpts of the larger set of content standards and generally represent the knowledge and skills assessed at each level. Students who score at or within a particular level of achievement possess the bulk of the abilities described at that level and generally have mastered the skills described in the preceding achievement levels. Educators can use the descriptors to explain the knowledge and skills a student is expected to possess to achieve the various achievement levels for each test. Descriptors are both grade and subject specific.

Revision of the Achievement Level Descriptors is part of the implementation of the new mathematics assessments that represent the content standards adopted in 2007 - 2009. The Achievement Level Descriptors will influence the upcoming achievement standards setting for mathematics, scheduled for August 2010. In order for Oregon’s standards to be accepted for federal accountability purposes, we must submit documentation to the U.S. Department of Education (U.S. DOE), and the Achievement Level Descriptors are part of this required documentation. Achievement Level Descriptors are also used for school accountability purposes.

In preparation for drafting the Mathematics Achievement Level Descriptors a team of math education experts has reviewed state and national resources including:  The new Oregon Mathematics Content Standards  Draft Common Core State Standards  Articles on national best practices  NAEP trends  ALDs from other key states

This team is developing Math Achievement Level Descriptors for all grades.

Over the coming months, ODE staff will be presenting the subject specific math ALDs to the Board.

10 Appendix C: Additional Information on Field Engagement and Survey Results

Field Engagement: In addition to the team drafting the Achievement Level Descriptors, ODE has solicited feedback from members of our Content Panels, our Assessment Advisory Team, Math education groups, community college representatives, and education stakeholders. Some groups have provided several rounds of feedback on the Policy Definitions and Achievement Level Descriptors. Field Engagement included:  Surveys conducted between December and March o Math Content Panel, other Content Panels (English, Science, Social Sciences, and ELPA), Assessment Advisory Committee, OMEC (Oregon Mathematics Education Council), OCTM Board (Oregon Council of Teachers of Mathematics) o ODE will survey the following groups in March - Career and Technical Ed Deans, Community College Math Chairs, and Council of Instructional Administrators  Conversations at group meetings  Conversations with individual educational stakeholders Feedback from these surveys and conversations have shaped the direction of our General and Math Policy Definitions and Math Achievement Level Descriptors.

First Round of Surveys Math Content Panel Members (23 participants)  65% thought that the proposed General Policy Definitions were clear  94% thought that the High School Achievement Level Descriptors compared well or very well to the new high school content standards  81% thought the rigor of the High School Achievement Level Descriptors was consistent or very consistent with the Math Policy Definitions  73% thought that the proposed Math Policy Definitions were clear Non-Math Content Panel Members (44 participants)  69% thought that the proposed Policy Definitions were clear  The majority agreed with the rigor of the Policy Definitions Based on feedback from the field, ODE staff made adjustments and improvements to the General and Math Policy Definitions. ODE staff then solicited additional feedback on these updated statements.

Second Round of Surveys Assessment Advisory Team, OMEC (Oregon Mathematics Education Council), OCTM Board (Oregon Council of Teachers of Mathematics) (45 participants total)  81% thought that the proposed General Policy Definitions were clear or very clear  85% agreed with the rigor of the General Policy Definitions  60% thought that the Math Policy Definitions were clear or very clear  87% thought the Math Policy Definitions compared very well or mostly to the content standards  69% said that they either liked or loved our current achievement level names  74% said that they either liked or loved following name option: Level 1: Well Below Proficiency, Level 2: Approaches Proficiency, Level 3: Meets Proficiency, Level 4: Exceeds Proficiency (Hawaii)  35% said that they either liked or loved the following name option: Below Basic, Basic, Proficient, Advanced (NAEP)  12% said that they either liked or loved the following name option: Failing (HS only)/Warning, Needs Improvement, Proficient, Advanced (Massachusetts)

11 ODE will continue to solicit feedback from education stakeholders as we move forward in this process.