Review of Evaluation Studies of Mathematics and Science Curricula and Professional Development Models

Home , Connected Mathematics, John Saxon (educator), Saxon math

Abstract

This report identifies mathematics and science curricula as well as professional development models at the middle and high school levels that are effective based on their success in increasing student achievement. The goal of the study was to provide some choice to districts and schools that wanted guidance in selecting a curriculum and that wished to use effectiveness as a selection criterion. Unexpectedly, most middle and high school mathematics and science curricula did not have studies of student achievement with comparison groups, and it proved especially difficult to find effects in either math or science for subgroups by sex, minority status, and urban status. Findings strongly suggest that science curricula is more effective when it is inquiry-based, although math curricula can be effective when standards- or traditional-based.

REVIEW OF EVALUATION STUDIES OF MATHEMATICS AND SCIENCE CURRICULA AND PROFESSIONAL DEVELOPMENT MODELS

By Beatriz C. Clewell Clemencia Cosentino de Cohen The Urban Institute

and

Patricia B. Campbell Lesley Perlman Campbell-Kibler Associates, Inc.

with

Nicole Deterding Sarah Manes Lisa Tsui The Urban Institute

and

Shay N.S. Rao Becky Branting Lesli Hoey Rosa Carson Campbell-Kibler Associates, Inc.

Submitted to the GE Foundation December 2004 Acknowledgments

A number of individuals contributed to this effort in various ways. We were fortunate to have the assistance of Gerhard Salinger of the National Science Foundation; Jo Ellen Roseman of Project 2061 at the American Association for the Advancement of Science (AAAS); Joan Abdallah at AAAS; and several staff members at the Center for Science Education at the Education Development Center—Barbara Berns, Jeanne Century, Joe Flynn, Elisabeth Hiles, Jackie Miller, Marian Pasquale, and Judith Sandler —in helping us identify science curricula that might have evaluation studies. We are also deeply indebted to those who reviewed our report and offered useful suggestions for revising it: Jane Butler Kahle at Miami University of Ohio and Linda Rosen of Education & Management Innovations, Inc. Thanks are due to William Bradbury and Cara West, Urban Institute staff who helped in the production of the report. Most of all, we wish to express our appreciation for the responsiveness of curriculum developers and researchers whose curricula are reviewed in this report and who shared evaluation studies with us.

Last, but not least, we thank our program officer, Roger Nozaki and his colleague, Kelli Wells, both of the GE Foundation, for their insightful comments and suggestions on the report draft that helped to make this document more user-friendly. We thank the GE Foundation for funding this review and for taking an evidence-based approach to school reform. We think it’s the only way to go.

REVIEW OF EVALUATION STUDIES OF MATHEMATICS AND SCIENCE CURRICULA AND PROFESSIONAL DEVELOPMENT MODELS

Introduction

This report presents the findings of a review of about four hundred studies evaluating mathematics and science curricula and professional development models for middle school and high school. As requested by the GE Foundation, the main goal of this review was to identify, in response to the GE Foundation’s request, mathematics and science curricula as well as professional development models that had been deemed effective based on their success in increasing student achievement. The Foundation’s interest in these findings stems from its desire to initiate a program of funding to foster sustainable improvement in academic achievement of underrepresented and disadvantaged populations.

Historically, curriculum choice at the local level has often been made by a committee that decides which curriculum to adopt based on considerations only peripherally related to student achievement—such as state-imposed standards, recommendations of others, cost, and presentations by publishers’ representatives. Choice of professional development models follows a similar pattern. Indeed, there has been very little else available to guide school districts in their curriculum selection process, since for most curricula and textbooks the only data at hand are publishers’ figures on the number of adoptions. That has been changing. There is a growing movement to assess the effectiveness of math and science curricula through various methodologies, including content analyses, comparative studies, case studies, and synthesis studies.1 And while there have been several studies of the effectiveness of professional development practices, very few have measured the effects of these practices on student achievement.

In this document, we describe the methodology used to conduct this review, present our findings, and end with a summary of conclusions. To provide an international perspective on these topics, the report includes a brief look at the research on mathematics and science education in three countries that are similar in key dimensions to the U.S.

Methodology

Criteria for Selecting Evaluation Studies

We developed a set of criteria to guide the selection of evaluation studies to be included in our review. Studies were expected to have (1) rigorous methodological design; (2) measures of impact on student outcomes (which include, but are not limited

1 The majority of these efforts have been undertaken by the American Association for the Advancement of Science (AAAS), the National Research Council (NRC), and the U.S. Department of Education. The AAAS study used content analysis, the NRC study did not rate specific curricular math programs, and the U.S. Department of Education study reviewed middle school math programs only.

1 to, test scores); (3) comparative data, cross-sectional or longitudinal, with experimental and quasi-experimental designs preferred over others; and (4) high quality and valid data.

We offer several caveats regarding the quality of the evaluation studies that we report in this document, especially those on mathematics and science curricula. Because of the dearth of studies that met our criteria, we were forced to compromise and include a number of evaluations that did not report effect sizes; a few that did not give the statistical significance levels for differences; and several that lacked non-treatment comparison groups. In some cases we were unable to verify the quality of the data on which findings were based. It was also a source of great disappointment that so few of the studies we identified disaggregated findings by sex, race/ethnicity, or urban school location. We believe, however, that taken as a whole the studies that are included here offer useful insight into the condition of mathematics and science curricula in middle and high school.

Identification of Curricula/Professional Development Models

Using research databases such as the Education Resource Information Center (ERIC), Education Abstracts, and web sites such as Northwest Regional Education Laboratory’s Catalog of School Reform Models, we conducted a literature search of articles and reports pertaining to (1) major mathematics and science curricula used at the middle and high school level; and (2) empirical studies that examine how teacher professional development in science and mathematics affects student outcomes. The review team also gathered and reviewed relevant documents that were not accessible through traditional research outlets. It was much more the case for science than for mathematics that most of the evaluation studies of recent curricula were unpublished reports of evaluations conducted by the program developers. On the other hand, a large number of published mathematics curriculum studies were available for inclusion in this review. Appendix D contains lists of all mathematics and science curricula for which studies were sought.

Primary sources of mathematics reform models were the Northwest Regional Educational Laboratory’s database on whole-school reform models2 and Comprehensive School Reform and Student Achievement: A Meta-Analysis.3 Primary sources of the mathematics curriculum were National Science Foundation-funded projects; the U.S. Department of Education’s “What Works Clearinghouse” and the Mathematics Expert Panel; the Mathematical Sciences Education Board’s Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials; and the American Association for the Advancement of Science’s Project 2061.

2 http://www.nwrel.org/scpd/catalog/index.shtml 3 Borman, G. D., G. M. Hewes, L. T. Overman, and S. Brown. 2002. Comprehensive School Reform and Student Achievement: A Meta-Analysis. CRESPAR Report No. 59. Baltimore, Md.: CRESPAR/Johns Hopkins University. http://www.csos.jhu.edu/CRESPAR/techReports/report59.pdf,

2 In order to ensure broad coverage of the science curriculum studies, we contacted experts on science curricula at organizations such as the Education Development Center, the Technology Education Research Center (TERC), the National Science Foundation, Project 2061, and others. In view of how few published evaluations of science curricula we were able to identify, we attempted to locate more recently developed curricula that might not yet have produced published evaluation studies. Once these curricula were identified through conversations with experts in the field, we obtained contact information on the developers of these programs and approached each of them to ascertain whether or not they had evaluation data or reports on the effectiveness of their curricula that met our established criteria. Several developers either did not respond to our requests or responded that they had not yet completed evaluation studies. We were able, however, to secure a number of evaluation reports from developers and reviewed these to determine whether or not they met our criteria for inclusion in this study. We also scoured the Internet for sources of information on curricula and on relevant evaluation studies.4 Finally, to facilitate our analysis of the data, we developed matrices into which we entered relevant information on each of the studies that met our criteria. This information included the methodological design, the analytic technique, student outcome areas measured, outcome measures and instruments used, number in the sample, whether data were disaggregated by race/ethnicity and sex, and impact. We also included descriptive information about the curriculum, including the subject matter, targeted grades, curriculum name, whether or not it had a professional development component,5 and its principal instructional features.

We were able to locate very few studies of professional development models that used student achievement measures of effectiveness.6 Most of these studies were found via an extensive review of the literature, including ERIC, Education Abstracts, ProQuest, EbscoHost and others. A matrix containing descriptive information on the professional development models and on the study elements was prepared for all the professional development evaluation studies that we identified.

4 American Association for the Advancement of Science (AAAS)’s Project 2061: http://www.project2061.org/publications/articles/textbook/default.htm, Department of Education's Mathematics and Science Education Expert Panels: http://www.ed.gov/offices/OERI/ORAD/KAD/expert_panel/math-science.html, Eisenhower National Clearinghouse for Mathematics and Science Education: http://www.enc.org, Center for the Social Organization of Schools: http://www.csos.jhu.edu/, Education Development Center, Inc.’s Center for Science Education: http://cse.edc.org, Education Resource Information Center (ERIC): http://eric.ed.gov, Northwest Regional Education Lab: http://www.nwrel.org/scpd/catalog/index.shtml, National Clearinghouse for Comprehensive School Reform: http://www.csrclearinghouse.org/ National Science Resources Center: (http://www.nsrconline.org/), The Textbook League: (http://www.textbookleague.org/), University of Wisconsin-Madison’s National Center for Mathematics and Science: (http://www.wcer.wisc.edu/ncisla/), curriculum company web sites, developer web sites (i.e., at Johns Hopkins, LHS, etc.), with follow-up to authors of studies, and subscription electronic journal databases: (ProQuest Research Library Plus, JSTOR, Education Abstracts, EbscoHost, and Project MUSE) 5 Most of the data on the mathematics curricula did not provide this information, while that on the science curricula did. 6 The paucity of such studies has been mentioned by several researchers (Harlen 2004; Kennedy 1998; Marek and Methven 1991).

The International Component

Criteria for selecting countries for international comparison. Our report includes an international comparative study of a set of countries that may contribute useful information to the research at hand. The goal of this comparison is to garner additional, corroborating evidence on best practices. This required that countries be selected carefully and purposefully, based on the degree to which their experiences may be useful to the U.S. We selected nations based on two main criteria: (1) average country achievement on different mathematics and science tests (only high achievers were considered);7 and (2) similarity to the U.S. on key features of their educational systems— namely, degree of centralization of decisionmaking with respect to the curriculum; degree of centralization of decisionmaking with respect to textbook use; and degree of stratification or selectivity of the system. Focusing on those countries that performed well on average and whose educational systems were closest to the U.S. yielded three nations for comparison: Canada, Australia, and England.8 The section entitled “International Comparative Study” provides a discussion of findings, as well as additional information regarding country selection.

Identification of relevant research on selected nations. We conducted a review of the literature on the three selected countries based on several sources. These included databases such as ERIC and Proquest, as well as institutional sources—such as OECD, UNESCO, and the World Bank—and studies arising from the tests used for selection herein—Trends in International Mathematics and Science Study (TIMSS) and Programme for International Student Assessment (PISA). The review focused first on studies of characteristics of candidate educational systems and then on curriculum, pedagogy, and professional development at selected countries.

Findings of the Review

Interpreting Results: Some Things to Consider

The validity of the test. Issues for mathematics curriculum studies. During the past 15 years, mathematics curriculum development has moved in two different directions. Traditional curricula have continued the hierarchical structure of mathematics courses broken out by specific subject area: algebra, followed by geometry, followed by a more advanced algebra/trigonometry and pre-calculus. Calculus is made available for accelerated

7 Country performance was based on three tests— Trends in International Mathematics and Science Study (TIMSS) 1995 and 1999, and Programme for International Student Assessment (PISA) 2000. 8 England is not a “high achiever”, as it experiences achievement levels similar to the U.S., but it was included for reasons spelled out on page 12, under “The International Comparative Study: Conclusions.”

4 students, usually those who took algebra in the eighth grade. Standards-based curricula tend to be more interdisciplinary, providing students with a range of subject areas the first year and returning to them during each subsequent year, allowing for deeper analysis and understanding that tends to be more focused on longer-term problem solving.

The content and skills covered by most standardized achievement tests tend to reflect more closely the content and skills covered by traditional mathematics curricula than those covered by standards-based curricula, causing them to have better “content validity” for traditional curricula. Many researchers of standards-based curricula are aware of this and develop their own student achievement tests that more accurately test the skills and content of standards-based curricula. If students taking one curriculum score higher than others on both types of test, there is no question of interpretation. Beyond that, however, judgments of efficacy must take into account the content validity of the tests in order to determine which type of curriculum is more effective.

Issues for science curriculum studies. As inquiry-based science curricula have become a major tool for standards-based reform efforts across the U.S., a dilemma has arisen regarding the appropriateness and credibility of assessments to measure effectiveness of these curricula in terms of student achievement. Basically, the dilemma can be described in the words of Walker and Schaffarzick (1974), who concluded from their review of innovative curricula that: “innovative students do better when the criterion is well-matched to the innovative curriculum, and traditional students do better when the criterion is matched to the traditional curriculum” (p. 94). To address potential lack of “fit”, curriculum developers have developed their own assessments that more closely measure the intended effects on students. Results from standardized tests, however, carry greater credibility and are used by most states for accountability purposes (although less for science than for mathematics). The dilemma posed, which is similar to that faced in mathematics, does not seem, nevertheless, to be as much of a problem in science. Shymansky, Kyle, and Alport (1983), for example, in conducting their large meta- analysis, compared the results of standardized tests to those of other forms of assessment and found very small differences. More recently, Hamilton, McCaffrey, Stecher, Klein, Robyn, and Bugliari (2003) report that the differences between multiple-choice (standardized) and open-response tests of student achievement that they observed in evaluating the effects of standards-based mathematics and science instruction at 11 sites were not significant. Ruby (2001) discovered that the positive relationship of hands-on science and test scores that he found did not differ by type of test.

Measures used. The evaluation studies reported here for mathematics and science curricula used a variety of assessments—some standardized, some state-mandated, some self-developed, and some developed by others specifically for standards-based curricula. The types of assessment tools used in the evaluations are specified for each study, and judgments regarding the content validity of the assessments used should guide determinations about the effectiveness of these curricula.

5 The size of the difference. Most impact studies for both mathematics and science curricula reported the statistical significance of their results.9 Statistical significance represents the probability that an observed difference really exists and is not due to chance. It does not say anything about the size or meaningfulness of a result. (Being statistically significant is a first step—if there are no statistically significant differences then there are no differences.) If differences are statistically significant then there is another measure, called an effect size, which provides a measure of size—that is, shows how big the difference is. Effect sizes greater than .4 are considered large; between .2 and .4 are considered moderate; and less than .2 are considered small (Glass, McGaw, and Smith, 1981). Readers can also look at the size of statistically significant differences and decide for themselves how meaningful they are.

The target population or district context. Unfortunately, very few studies presented data on effects by sex, race/ethnicity, or urban school location. Where data are disaggregated by these characteristics, we have highlighted these findings, which enhance our knowledge about the effectiveness of curricula for populations or districts targeted by the Foundation for funding.

Middle and High School Mathematics Curricula: Conclusions

Our review netted 89 middle and high school curricula including eight mathematics curricula that were developed as part of whole school reform efforts (out of 31 whole school reform efforts examined) and 81 other middle and high school mathematics curricula. A total of 156 studies of student mathematics achievement with comparison group data were found for 18 of the curricula (20 percent of the total number of curricula identified). A table listing the math curricula that had credible evaluations appears in appendix A together with overviews of the 18 mathematics curricula and their impact studies. The overviews include: the type of student achievement measure used, the number and direction of the results, and, if available, the size of any differences between groups. If the results were broken out by sex and/or by race/ethnicity, this, too, is indicated. We concluded from our review of these evaluation studies that:

Most middle and high school mathematics curricula do not have studies of student achievement with comparison groups that can be found through literature or web searches.

As indicated above, studies of student achievement with comparison groups could be found for only 20 percent (18) of the curricula. Only three of the studies found specified the curriculum to which the target curriculum was being compared. The rest compared their curriculum to some unnamed curriculum, making comparisons across curricula impossible.

9 Where inferential statistics were used, only differences that reached the conservative minimum acceptable statistical significance level of .05 were included. Inferential statistics were used in all science studies with one exception, which is noted in the description.

6 If students are going to be judged on the results of an external test, the mathematics curriculum selected should cover the areas and skills that are included on that test (i.e., the test and curriculum should be aligned).

Different mathematics curricula cover different content areas at different times. Three of the 18 curricula—Saxon Math, Direct Instruction, and Advanced Placement Calculus— cover traditional mathematics subject areas (i.e., algebra, geometry) while the remaining 15 integrate traditional mathematics subject areas across years rather than covering a subject area per year. Whether a curriculum is “integrative” or “traditional” has implications for testing. As would be expected, students tend to score higher on tests focusing on the content and skills covered in their curriculum. Traditional math curricula have greater content validity than do standards-based curricula in most standardized and state tests. Integrative mathematics curricula, which are standards based, have greater content validity with standards-based tests than do most traditional curricula.

Studies of six of the curricula (Cognitive Tutor, Connected Mathematics, Interactive Mathematics Program, Prentice Hall: Tools for Success, Saxon Math and University of Chicago School Mathematics Project [UCSMP]) found that students who use the curriculum being tested scored higher than comparison students on a majority of standardized and/or state tests used as well as on a majority of the curriculum- based tests used.

One of the six curricula, Saxon, focuses on traditional course breakdowns (i.e., algebra, geometry), while four curricula—Connected Mathematics, Interactive Mathematics, Prentice Hall, and UCSMP—are integrative. The last curriculum, Cognitive Tutor, includes both traditional and integrative components. Moderate to large achievement differences between target and comparison students, as indicated by effect size, were found in favor of four of the six curricula (Cognitive Tutor, Connected Mathematics, Interactive Mathematics, and Prentice Hall). All six curricula cover middle school and three—Cognitive Tutor, Saxon, and UCSMP—cover high school as well.

The few results broken out by sex were inconsistent.

Only five of the 18 curricula—Cognitive Tutor, Connected Mathematics, Interactive Mathematics, MATH Connections, and Mathematics with Meaning—broke out results by sex. In Connected Mathematics and MATH Connections, no sex differences were found, while in Mathematics with Meaning, boys scored slightly higher than girls. In Cognitive Tutor, the results were mixed. Girls taking Interactive Mathematics were slightly more apt than boys to continue on to three or more years of mathematics.

Connected Mathematics appeared to be reducing racial/ethnic gaps.

Four Connected Mathematics studies looked at the relative growth in achievement by race/ethnicity. In two studies, African-American and Hispanic students showed greater growth than the other Connected Mathematics students. In a third study, African- American students showed greater growth than others, while in a final study Hispanic,

7 White, African-American and Asian-American students’ scores increased while Native American students’ scores decreased.

With the exception of Connected Mathematics, too few results per curriculum were broken out by race to allow us to draw general conclusions regarding racial/ethnic effects.

Five curricula presented results by race/ethnicity (Cognitive Tutor, Connected Mathematics, MATH Connections, Mathematics in Context, and Mathematics with Meaning). A MATH Connections study found no significant difference by race/ethnicity among MATH Connections students, while in Mathematics with Meaning, white students outperformed minority students. Other studies compared target students with comparison students of the same race/ethnicity. African-American Mathematics in Context students in one study were found to score better than comparison students. In one Cognitive Tutor study, African-American students scored better than comparison students, while there was no difference for Hispanic students. In a second study, Hispanic students using Cognitive Tutor did better than comparison students. As detailed in the section above, Connected Mathematics studies provided more consistent evidence that the curriculum was successful in reducing racial/ethnic gaps.

Middle and High School Science Curricula: Conclusions

We identified 80 science curricula at the middle and high school levels.10 Similar to the mathematics curriculum, which was included in eight whole-school reform efforts, we found seven science curricula that had been developed specifically as a part of whole school reform.

A total of 45 studies of student achievement in science were found that met our criteria, covering 26 percent (21) of the total science curricula identified. A table listing these curricula as well as brief descriptions of each and the results of the studies are given in appendix B. As with the mathematics curricula, the study descriptions include the type of student achievement measures used, a description of the results from each study, and, if available, the effect sizes of any differences between groups. Results by sex, race/ethnicity, limited English-proficiency (LEP) status, or urban schools are provided where available. Our review of evaluation studies of science curricula led us to the following conclusions:

As with mathematics curricula, most middle and high school curricula do not have evaluation studies of student achievement with comparison groups that can be found through published literature or web searches.

Of the 80 curricula identified, studies that met our criteria (45) could only be found for 21 curricula. None was a publisher’s textbook series. In contrast to the mathematics

10 Of the 80 science curricula identified, 59 had no evaluations or had evaluations that did not meet our criteria, or evaluations were out of print, or we did not receive a response from the developers.

8 curricula, however, many of which had been the subjects of multiple studies, most of the science curricula had only one evaluation or study, usually unpublished works available through the developer. Because the National Science Foundation (NSF) was a pioneer in developing standards-based science curricula (the first wave of these appeared in the sixties), there were published meta-analyses that examined the effect on student achievement of a large group of NSF-funded science programs—all inquiry-based—on student achievement as compared to the effect of traditional, textbook-based science curricula. We believe that the dearth of evaluation studies of single science curricula can be explained by the relatively recent development of the new generation of science curricula, many of which were also funded by NSF. Not enough time has elapsed for these curricula to have been the subject of multiple studies, and those studies that are available have been conducted mostly as evaluations by the developers of the curricula.

Science curricula based on the inquiry approach are consistently more effective than traditional science curricula as measured by student achievement.

The preponderance of evidence provided by meta-analyses and evaluations of individual curricula seem to confirm that inquiry-based science curricula produce larger effects on student achievement than do the more “traditional” science curricula. The largest study of this kind (Shymansky, Kyle, and Alton 1983), which was reanalyzed in 1990 (Shymansky, Hedges, and Woodworth), involved 81 studies (reanalysis figures). While this meta-analysis found that inquiry-based science programs had the greatest impact on student achievement and process skill development in the primary grades (with significant differences in effect sizes found at the intermediate elementary level [four through six] for attitudes and perceptions only), by the junior high and high school levels, significant impact was found on achievement, attitude, and process skills. Other meta- analyses have reported greater positive effects on student performance for inquiry- oriented science than traditional approaches for high school curricula (Weinstein, Boulanger, and Walberg 1982); inquiry-discovery teaching techniques (Wise and Okey 1983); and an inductive rather than a deductive approach to teaching (although this effect was very small) (Lott 1983).11 No meta-analysis on inquiry-based science curricula of the magnitude undertaken by Shymansky, Kyle, and Alport in 1983 (and Shymansky, Hedges, and Woodworth in 1990) has been published on more recent inquiry-based science curricula, although researchers at Education Development Center are currently conducting such a study, and the National Research Council held a meeting in May 2004 on the topic of evaluating inquiry-based science.

The direction of the effects of inquiry-based science curricula on student achievement and performance is generally positive, as shown in the individual evaluations of the curricula that are identified in this report. Programs showing the greatest positive effects

11 These earlier meta-analyses used the terms “new” and “innovative” to describe inquiry-based science curricula. The distinction between “new” and “traditional” curricula was set forth by Shymansky and his colleagues (1983), with “new” curricula (a) having been developed after 1955; (b) emphasizing the nature, structure, and processes of science; (c) integrating lab activities as an integral part of the class routine; and (d) emphasizing higher cognitive skills and appreciation of science. “Traditional” curricula were defined as (a) having been developed before 1955; (b) emphasizing knowledge of scientific facts, laws, theories, and applications; and (c) using lab activities as secondary applications of concepts previously covered in class.

9 are (1) a set of activity models for use in physical science and technology education courses in middle and high school (Designs/Designs II); (2) a comprehensive, laboratory- based program in which students in grades seven through nine construct their own knowledge through experiential, hands-on learning (Foundational Approaches in Science Teaching [FAST]); (3) curriculum materials to support the development of integrated science understanding for middle school students in urban schools (Center for Learning Technologies in Urban Schools [LeTUS]); (4) a supplemental program for average-to- gifted students in grades two through eight employing problem-based learning to engage students in the study of the concept of systems, specific science content, and the scientific research process (National Science Curriculum for High Ability Learners); (5) a program to promote understanding of physics principles in the context of experiences relating to the daily lives of high school students (Physics Resources and Instructional Strategies for Motivating Students [PRISMS]); and (6) an inquiry-based, technology-supported environmental science curriculum for high school (WorldWatcher/LATE).

It is difficult to determine the effect of these science curricula on different subgroups of students—such as girls, minority group members, and urban students.

Very few of the curricula had studies that met our criteria and disaggregated their findings by sex, language minority status, or urban location. Surprisingly, none of the studies reported data disaggregated by race/ethnicity.

Sex. In one evaluation study of Constructing Ideas in Physical Science (CIPS), participation did not appear to have closed the gender achievement gaps on multiple- choice content, process questions, or open-ended content items. One of two studies on Center for Learning Technologies in Urban Schools (LeTUS) provided evidence that participating in at least one LeTUS unit reduced the boy-girl achievement differences on statewide examinations. Evaluation data on Modeling Instruction in High School Physics show that, in terms of performance, male students consistently outperform female students. The Designs/Designs II evaluation reported that on measures of conceptual knowledge, there was no significant difference in the gains made by girls versus boys. Thus, one (LeTUS) of the four curricula that reported data by sex showed greater gains in (some areas of) achievement for female students. Two (CIPS and Designs/Designs II) showed no differences and a fourth showed larger gains for boys. A large meta-analysis of NSF-funded inquiry-based programs (Shymansky, Kyle, and Alport 1983), which was resynthesized in 1990 (Shymansky, Hedges, and Woodworth), found that the NSF- funded “new” science curricula had a significant positive effect on males but not on females in terms of composite performance; analytic skills of females, nevertheless, improved significantly in the inquiry-based science programs.

Race/Ethnicity, LEP Status, or Urban School Attendance. Interestingly, evaluations of two inquiry-based curricula reported positive results for English Language Learners (ELLs). The use of FOSS in fourth and sixth grade classes for ELLs showed a positive relationship between years in the science program and standardized test scores. The evaluation of Expeditionary Learning Outward Bound (ELOB) reported consistent gains in all science subject areas over five years for one school where the number of immigrant

10 (ELL) students grew by 22 percent. This school also had a high percentage of economically disadvantaged students (i.e., students on free and reduced lunch).

The large meta-analysis of NSF-funded inquiry-based programs (Shymansky, Kyle, and Alport, 1983), which was re-synthesized in 1990 (Shymansky, Hedges, and Woodworth), found that the NSF-funded inquiry-based programs, while having a greater effect on all students than did the traditional programs, showed (1) a much greater effect on the composite and achievement scores of urban students than on their suburban or rural counterparts and (2) a much greater effect on the analytic scores of urban students than on their suburban counterparts. WorldWatcher/Learning about the Environment (LATE) reported higher gains for urban students than for suburban students. It is surprising that none of the studies that provided disaggregated data on urban students showed separate outcomes by race/ethnicity.

Most science curricula include a professional development component.

At least 16 of the 21 science curricula for which we report evaluation studies have professional development components. Inclusion of a professional development component as a part of the curriculum is far more prevalent in science than in mathematics because science curricula tend to be more discretionary and variable than mathematics curricula (Kennedy 1998), leading developers to provide more guidance to teachers regarding the appropriate instructional approaches to be used for specific curricula.

Professional Development Programs: Conclusions

We identified 18 evaluation studies of professional development in science and mathematics that used student achievement outcomes as measures of effectiveness.12 A matrix outlining general features of these studies appears in appendix C. Our search for studies that met established criteria was facilitated by the research of Mary Kennedy (1998). The following are the conclusions that we draw from a review of these 18 studies and others:

Providing professional development for teachers of standards-based science curricula is associated with higher levels of student achievement.

The re-analysis of the large 1983 meta-analysis of inquiry-based science curricula (Shymansky, Hedges, and Woodworth 1990) found larger effect sizes on student performance measures for students of teachers in inquiry-based courses who had participated in professional development linked to the use of inquiry-based materials. (Students of teachers using inquiry who had not had professional development still outperformed students in traditional courses, but the former were outperformed by students in inquiry-based courses whose teachers had received professional development.) Similarly, an evaluation of an inquiry-based science curriculum, Project Inquiry, found that teachers who received professional development in implementing

12 Five of these were connected to specific mathematics or science curricula.

11 standards-based, inquiry-oriented instructional strategies (and who used the specific science materials linked to the program) had students who performed significantly higher on two science assessments (Rose-Baele 2003). This is also true of an evaluation of Modeling Instruction in High School Physics. This evaluation found that the students of high school physics teachers who had completed the modeling workshop series demonstrated much greater gains on a widely used physics assessment tool than physics students of the same teachers the year before participation in the professional development series and a comparison group of high school physics students a decade ago (Hestenes 2000).

Professional development that is tied to curriculum, to knowledge of subject matter, and/or to how students learn the subject is more effective in terms of improving student achievement than is professional development that focuses only on teaching behaviors.

A number of studies have concluded that the content of professional development is more important than its format and that content should be linked to subject matter knowledge, a specific curriculum, or the process of student learning. In her analysis of 12 studies of professional development models that reported effects on student achievement, Kennedy (1998) found that the models showing the largest effect sizes were those that focused on subject matter knowledge and on student learning of a particular subject. This finding was echoed in the work of McCaffrey, Hamilton, and Stecher (2001) in a large-scale study of high school standards-based mathematics reform in a large urban school district that was part of NSF’s Urban Systemic Initiative program. One of their conclusions was that in order to be effective, professional development for teachers should consider curriculum and instructional practices in combination. In the researchers’ words, “Simple prescriptions for how to teach are unlikely to be effective” (p. 10). Cohen and Hill (1998) addressed the question of whether students of teachers who received professional development focused on student curriculum scored higher on state mathematics assessments in California. They found that teachers who attended curriculum-centered workshops and who had learned about the state assessment system had students who received higher achievement scores on the state test than students of teachers who had not participated in the workshops or learned about assessment.

The amount of professional development provided is an important factor in influencing both change in teaching behavior of teachers and change in the classroom environment.

The amount of professional development provided to teachers is another factor that has received attention in the literature. While several studies suggest that professional development, to be effective, should be intensive and sustained, (Kahle and Rogg 1996; Supovitz and Turner 2000), we found only one study that specifically investigated how many hours of professional development were required to effect a change in teaching behavior (towards inquiry-based teaching practice). This study found that behavioral change was only evident after teachers had received a minimum of 80 hours of intensive professional development (Supovitz and Turner 2000). The same study found that it was

12 only after 160 hours of professional development that the teachers’ classroom environment acquired a “culture of investigation.”13 Evaluators of Project Inquiry found that the number of self-reported hours of Project Inquiry-sponsored professional development was positively associated with science achievement of students (Rose-Baele 2003). Kennedy (1998), however, cautions against adopting a “more is better” approach to professional development. She points out that of the 12 professional development models that she investigated, amount of contact time was not the most important factor determining the largest effect sizes (although, coincidentally, the most effective model reported 80 in-service contact hours, which was the minimum effective contact time found by other research [Supovitz and Turner 2000]).

Widely held beliefs about what constitutes effective professional development are not supported by research linked to student achievement.

In Kennedy’s study of 12 professional development models, she examined various features of in-service programs that have been hypothesized as being important elements of successful professional development (1998). These features are (1) program intensity as measured by total contact time with teachers (discussed above); (2) dispersal of time (whether it is concentrated or interspersed throughout the school year); (3) classroom visits by experts for consultation or coaching; and (4) whole school or individual provision of professional development. Her conclusions suggest that while professional development in science seems to benefit from distributed time (sessions throughout the academic year), the studies in mathematics do not support the hypothesis that distributed time is beneficial. Four of the programs reviewed by Kennedy provided in-class visitations, yet none produced greater influences on student learning than those that did not. Kennedy’s study also found no compelling evidence that in-service programs working with whole schools are more effective in terms of increasing student achievement in mathematics or science; in fact, the programs in her study that worked with whole schools demonstrated the smallest influences on student learning.

The International Comparative Study: Conclusions

As mentioned earlier, countries were selected based on the degree to which their experiences may be useful to the U.S. We therefore selected, first, nations that were high achieving on international tests of student achievement. Among these, we selected those whose educational system was closest to the U.S.’s. This yielded three nations— Australia, Canada, and England. Two of them—Canada and Australia—are ideal insofar as their systems of education are closest to the U.S.’s. One of them—England—is more centralized than the U.S. (with a national curriculum, for example); more selective (i.e., not a comprehensive system of education); and is not as high achieving as the others (experiencing achievement levels similar to the U.S., sometimes higher and sometimes

13 “Teacher behavior” in this context refers to teachers’ use of specific pedagogical approaches in instruction, such as inquiry-based teaching practices. “Change in the classroom environment” refers to teacher facilitation of an investigative classroom culture through seating arrangements to stimulate discussion, use of cooperative learning groups, encouraging students to explain concepts to one another, and other such practices.

13 lower). All of them, however, share other characteristics that make them good points of reference for the U.S. They are high-income, developed nations and spend similar shares of GDP on kindergarden through twelfth grade education. They are also culturally closer to the U.S. than the Asian and European countries that are generally highest achieving (e.g., Japan and Netherlands). This provides a natural (albeit partial) control for cultural disparities that may account for some of the observed achievement differences. Lastly, these nations have experienced, like the U.S., pressure from federal authorities seeking to influence educational policy.14 The conclusions below are based on a review of the relevant literatures on these three nations and are presented (selectively) as a complement to the conclusions arising from the literature review of curricula and professional development in U.S. mathematics and science discussed above.

Curriculum: Trends in the selected countries follow those found to be effective in the U.S.—shift from theory to applications, integration of subjects.

It is important to note that, like the U.S., two of the selected countries—Australia and Canada—have no national curriculum while one does (England). Australia delegates curriculum decisions to the member states, much like the U.S. Canada does the same thing, but de facto experiences convergence in curriculum coverage across provinces due to coordination through a Council of Ministers and through book purchases (the same publishers furnish books for all the provinces).

These countries have experienced a shift from theory to applications and utility. There is greater emphasis on math and science relevance and importance. There is also emphasis on integrating mathematics and science with other subjects and disciplines, as well as over time (i.e., building better course/content sequences). These changes—as well as the pedagogical ones mentioned below—often clash with testing requirements, as existing tests (usually the centerpiece of accountability efforts as well as certification of student achievement levels) tend to focus on acquired knowledge, on theory rather than on processes, or on demonstrated problem-solving skills. This goes back to the issue of “content validity” discussed earlier—that is, the degree to which the material covered in the test and the curriculum are aligned.

Pedagogy: Strategies prevalent in the selected nations are those found to be effective in the U.S. literature.

The tendency in all of these nations has been to transition, in both math and science, away from traditional textbook-based instruction and into inquiry-based, hands-on pedagogical approaches. They emphasize problem-solving skills over rote memorization, active modeling/activity-based instruction over passive textbook or lecture-based learning. There is also greater emphasis on “data analysis” and on real-world applications, particularly in mathematics. They also are moving towards increased use of technology in classroom instruction. These changes come hand in hand with a decreased emphasis on textbook use (though there is evidence of continued reliance on textbooks)

14 The case of England is an extreme example of this, as it has a national curriculum and unprecedented national government influence in education since 1988.

14 and greater diversity of materials used in the classroom (manipulative, technology, non- textbook printed materials, etc.).15 These conclusions are true of Canada and Australia and to a lesser extent of England as well.

To summarize, these nations (in particular Canada and Australia, the higher achieving of the three) seem to have shifted from a formal, traditional teaching approach to one centered on applications to the real world, on student interactions (group work) and on student-teacher interactions (interactive learning rather than lectures). This also could be described, partly, as a shift in the locus of responsibility for learning—from the teacher to the student.

Professional Development: There are virtually no studies (outside of the U.S.) of the impact of professional development on teaching practices or on student achievement, but there is widespread recognition of the importance of professional development and the need for evaluation of its impact.

Country studies indicate that professional development is offered by a variety of organizations (schools, boards, professional organizations, universities, central governments or departments of education). There is great variation with respect to all aspects of professional development opportunities—number of days/hours, funding sources, decisions regarding form and content of training, and extent to which teachers take advantage of them. There is, however, a clear emphasis on the importance of professional development (and, more generally, teacher quality) to raise student achievement. This is also true of the need to provide professional development opportunities to elementary school teachers, who often lack the knowledge and confidence needed to teach science. Professional development thus focuses, depending on the need of different teacher populations, on content knowledge and/or pedagogy. Leadership skills are another area of focus of professional development. Unfortunately, evidence on the types or forms or intensity of professional development opportunities that are effective (in these countries) is lacking. In detailed country reports on this topic recently published by OECD, all three nations mentioned the need to obtain evidence of the link between professional development and teacher practices and, ultimately, student learning.16

Summary of Conclusions

In this section, we summarize the major conclusions of this review that should be most useful to those wishing to invest in sustainable school reform in science and mathematics.

• Effective mathematics curricula in middle and high school can be either traditional or integrative (standards-based).

15 England, the lowest achieving of the three in mathematics, relies on mathematics textbooks and lecture style more than the other two countries. 16 The Canadian report was based on one province, Quebec.

15 • Effective science curricula in middle and high school should be inquiry-based rather than traditional.

• Effective professional development programs are those that focus on content rather than format and that have the following features:

Content tied to curriculum, knowledge of subject matter, and/or how students learn a subject; A minimum of 80 contact hours to effect changes in teachers’ instructional behaviors; and A minimum of 160 contact hours to effect changes in the classroom environment.

How to Use This Review

Choice of a curriculum in mathematics and science involves deciding what aspects of these subjects are important to address and emphasize in schools—this choice thus determines what students will learn. Once a decision is made, selection should be guided by how effective a particular curriculum is for the student population to be taught. The National Research Council report, On Evaluating Curricular Effectiveness: Judging the Quality of K–12 Mathematics Evaluations (2004) describes the value of this knowledge to the decisionmaking process:

Clearly, knowing how effective a particular curriculum is, and for whom and under what conditions it is effective, represents a valuable and irreplaceable source of information to decisionmakers, whether they are classroom teachers, parents, district curriculum specialists, school boards, state adoption boards, curriculum writers and evaluators, or national policymakers. Evaluation studies can provide that information but only if those evaluations meet standards of quality (p.1).17

This review of math and science curricula has tried to simplify for schools and districts the complex, time-consuming process of determining curriculum effectiveness by identifying programs that have what we consider to be credible evaluations. We have also distilled the findings of achievement-based research on professional development to a handful of principles that reflect effective practice. Once schools and districts have decided on a curriculum and an appropriate assessment tool, they might wish to collect their own impact data to evaluate how well the curriculum they choose is working with

17 Confrey, Jere, and Vicki Stohl, eds. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K–12 Mathematics Evaluations. Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials. National Research Council. Washington, D.C.: National Academy Press.

16 their own students.18 It is the responsibility of the school and district community to ensure that the content that they want students to learn is embodied in the curriculum, that the curriculum is effective for this purpose, and that appropriate measures are used to assess whether students are indeed learning what the community wants them to learn. It is an enormous task and an enormous responsibility. We hope that our review can provide some guidance and assistance in the process.

18 For those schools and districts that already have programs that they feel are effective but are not included in tables 1 and 2, we suggest that they collect their own effectiveness data via evaluation studies that adhere to the criteria established in this review. They may wish to contract with an evaluator for this purpose.

17 Bibliography

18 General Resources

Cohen, D.K., and Hill, H.C. 1998. State Policy and Classroom Performance: Mathematics Reform in California (Policy Brief RB-23). University of Pennsylvania: Consortium for Policy Reform in Education. Retrieved on 9/8/04 from http://www.cpre.org/Publications/rb23.pdf.

Glass, G.V., McGaw, B., and Smith, M.L. 1981. Meta-Analysis in Social Research. Beverly Hills, Calif.: Sage Publications.

Hamilton, L.S., McCaffrey, D.F., Stecher, B.M., Klein, S.P., Robyn, A., and Bugliari, D. 2003. Studying Large-Scale Reforms of Instructional Practice: An Example From Mathematics and Science. Educational Evaluation and Policy Analysis, 25(1), 1– 29.

Harlen, W. 2004. Evaluating Inquiry-Based Science Developments. Washington, D.C.: National Research Council. Retrieved on 10/15/04 from http://www7.nationalacademies.org/bose/WHarlen_Inquiry_Mtg_Paper.pdf.

Hestenes, D. 2000. Findings of the Modeling Workshop Project (1994–00). From Final Report Submitted to National Science Foundation. Retrieved on 10/28/04 from http://modeling.la.asu.edu/R&E/ModelingWorkshopFindings.pdf.

Kahle, J.B., and Rogg, S.R. 1996. A Pocket Panorama of the Landscape Study, 1995. Oxford, Ohio: Miami University.

Kennedy, M. 1998. Form and Substance in In-Service Teacher Education (NISE Research Monograph No. 13). Madison, Wisc.: University of Wisconsin, National Center for Improving Science Education. Retrieved on 9/15/04 from http://www.wcer.wisc.edu/NISE/Publications/Research_Monographs/vol13.pdf.

Lott, G.W. 1983. The Effect of Inquiry Teaching and Advance Organizers Upon Student Outcomes in Science Education: A Meta-Analysis of Selected Research Studies. Journal of Research in Science Teaching, 20, 437–51.

McCaffrey, D.F., Hamilton, L.S., and Stecher, B.M. 2001. Interactions Among Instructional Practices, Curriculum, and Student Achievement: The Case of Standards-Based High School Mathematics. Journal for Research in Mathematics Education, 32, 493–517.

Rose-Baele, J.S. 2003. Report of Fifth Grade Outcome Study, Science for all Students, 2001–2002 (Project Inquiry Report: No. 03-226). Arlington, Va.: National Science Foundation. Retrieved on 8/26/04 from http://www.ccsdschools.com/administration/assessment/PIpage.html.

19 Ruby, A. 2001. Hands-On Science and Student Achievement. Santa Monica, Calif.: Rand Corporation. Retrieved on 8/25/04 from http://www.rand.org/publications/RGSD/RGSD159/

Shymansky, J.A., Hedges, L.V., and Woodworth, G. 1990. A Re-Assessment of the Effects of Inquiry-Based Science Curricula of the Sixties on Student Achievement. Journal of Research in Science Teaching, 27(2), 127–44.

Shymansky, J.A., Kyle, W.C., and Alport, J.M. 1983. The Effects of New Science Curricula on Student Performance. Journal of Research in Science Teaching, 20, 387–404.

Supovitz, J.A., and Turner, H.M. 2000. The Effects of Professional Development on Science Teaching Practices and Classroom Culture. Journal of Research in Science Teaching, 37, 963–80.

Walker, D.F., and Schaffarzick, J. 1974. Comparing Curricula. Review of Educational Research, 44, 88–111.

Weinstein, T., Boulanger, F.D., and Walberg, H.J. 1982. Science Curriculum Effects in High School: A Quantitative Synthesis. Journal for Research in Science Teaching, 19, 511–22.

Wise, K.C., and Okey, J.R. 1983. A Meta-Analysis of The Effects of Various Science Teaching Strategies on Achievement. Journal of Research in Science Teaching, 20, 419–35.

20 Mathematics Curriculum Resources

Meta-analysis of Mathematics Curricula

Borman, G.D., Hewes, G.M., Overman, L.T., and Brown, S. 2002. Comprehensive School Reform and Student Achievement: A Meta-Analysis. CRESPAR Report No. 59. Baltimore, Md.: Center for Research on the Education of Students Placed At Risk, Johns Hopkins University. Retrieved 11/2/04 from http://www.csos.jhu.edu/CRESPAR/techReports/report59.pdf.

Advanced Placement (AP) Calculus

Dodd, B.G., Fitzpatrick, S.J., DeAyala, R.J., and Jennings, J.A. 2002. An Investigation of the Validity of AP Grades of three and a Comparison of AP and Non-AP Student Groups (College Board Report No. 2002-9). New York, NY: College Board.

Morgan, R. and Ramist, L. 1998. Advanced Placement Students in College: An Investigation of Course Grades at 21 Colleges (Statistical Report No. 98–13). Princeton, N.J.: Educational Testing Service.

Morgan, R. and Maneckshana, B. 2000. Advanced Placement Students in College: An Investigation of their Course-Taking Patterns and College Majors (Statistical Report No. 2000-09). Princeton, N.J.: Educational Testing Service.

Texas Education Agency. 2003. Advanced Placement and International Baccalaureate Examination Results in Texas, 2001–02 (Document No. GE03 601 08). Austin, Tex.: Author.

Cognitive Tutor

Carnegie Learning, Inc. 2001. Report of Results from Canton, Ohio (Cognitive Tutor Research Report OH-01-01). Pittsburgh, Pa.: Author. Retrieved on 10/28/04 from http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/ OH-01-01.pdf.

Carnegie Learning, Inc. 2001. Report of Results from Pittsburgh, Pennsylvania (Cognitive Tutor Research Report PA-91-01). Pittsburgh, Pa.: Author. Retrieved on 10/28/04 from http://www.carnegielearning.com/start. cfm?startpage=research/ research_reports /PA-95-01.pdf.

Carnegie Learning, Inc. 2001. Results from El Paso, Texas (Cognitive Tutor Research Report TX-00-01). Pittsburgh, Pa.: Author. Retrieved 10/28/04 from

21 http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/T X-00-03.pdf.

Carnegie Learning, Inc. 2001. Results from Lewisville, Tex.. (Cognitive Tutor Research Report TX-00-01). Pittsburgh, Pa.: Author. Retrieved 10/28/04 from http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/T X-00-01.pdf.

Carnegie Learning, Inc. 2001. Results from The Colony, Tex. (Cognitive Tutor Research Report TX-00-02). Pittsburgh, Pa.: Author. Retrieved 10/28/04 from http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/T X-00-02.pdf.

Carnegie Learning, Inc. 2002. Results from El Paso, Tex. (Cognitive Tutor Research Report TX-01-01). Pittsburgh, Pa.: Author. Retrieved 10/28/04 from http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/T X-01-01.pdf.

Carnegie Learning, Inc. 2002. Results from Moore, Okla. (Cognitive Tutor Research Report OK-01-01). Moore, Okla.: Author. Retrieved 10/28/04 from http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/

Carnegie Learning, Inc. 2004. Miami-Dade County Public Schools. Pittsburgh, Pa.: Author. Retrieved 10/28/04 from http://www.carnegielearning.com/ start.cfm?startpage=research/research_reports/miami_dade.pdf.

Koedinger, K.R., Corbett, A.T., Ritter, S., and Shapiro, L.J. 2001. Carnegie Learning’s Cognitive Tutor™: Summary Research Results. Retrieved 10/28/04 from http://www.carnegielearning.com/research/research_reports/ Summary_9.6.01.pdf

Sarkis, H. 2004. Cognitive Tutor Algebra 1. Program Evaluation Miami-Dade County Public Schools May 2004. (The Reliability Group Research Report DADEcognitive_evaluation). Retrieved 10/28/04 from `http://www.carnegielearning.com/start.cfm?startpage=research/research_reports/

College Preparatory Mathematics (CPM)

CPM Educational Program. 2003. California SAT9 results for CPM High Schools 1998– 2002 Test Results Summary. Retrieved 10/28/04 from http://www.cpm.org/info/sat9_98_02.html.

Haswell, R.W. 1995. Effectiveness of CPM vs. Traditional Math. Retrieved 9/28/04 from http://www.mathematicallycorrect.com/study1.htm.

22 Connected Mathematics (CMP)

Grant, Y., Ludema, H., Rickard, A., and Rivette, K. 2003. Connected Mathematics Project Research and Evaluation Summary 2003. Retrieved 10/28/04 from http://www.phschool.com/math/cmp/research_evaluation/.

Ridgway, J.E., Zawojewski, J., Hoover, M.D., and Lambdin, D. 2002. Student Attainment in the Connected Mathematics Curriculum. In Sharon Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 193–224). Mahwah, N.J.: Lawrence Erlbaum Associates

Contemporary Mathematics in Context: A Unified Approach (Core-Plus Mathematic Project CPMP)

Core-Plus Mathematics Project. 1998. Contemporary Mathematics in Context Student Achievement Reports, Volume I. Chicago, IL: Everyday Learning Corporation. Retrieved 11/1/2004 from http://www.wmich.edu/cpmp/pdfs/results.pdf.

Evans, C. 2001. Advanced Placement Calculus at Sturgis High School. Retrieved 10/28/04 from http://www.wmich.edu/cpmp/pdfs/Sturgis_HS_AP_Calc_Report.pdf.

Fouch, D., and Moore, D. 2001. Advanced Placement Calculus and Statistics at Traverse City High Schools. Retrieved 10/28/04 from http://www.wmich.edu/cpmp/pdfs/Traverse_City_AP_Results.pdf.

Frequently Asked Questions About the Core-Plus Mathematics Project (n.d.). Evaluation Evidence. Retrieved 10/28/04 from: http://www.wmich.edu/cpmp/faq- pieces/evidence.html.

Mariano, T. 2003. Ithaca High School CPMP Pilot Test. Retrieved 10/28/04 from http://www.wmich.edu/cpmp/pdfs/ithaca.pdf

Schoen, H.L., and Hirsch, C.R. 2002. The Core-Plus Mathematics Project: Perspectives and Student Achievement. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What Are They? What Do Students Learn? (pp. 311–343.) Mahwah, N.J.: Lawrence Erlbaum Associates.

Stucki, J. 2004. Wayzata High School, Plymouth, MN, Mathematics Program Evaluation. Retrieved 10/28/04 from http://www.wmich.edu/cpmp/pdfs/wayzata.pdf.

Verkaik, M. 2001. CPMP Student Performance at Holland Christian High School. Retrieved 10/28/04 from http://www.umich.edu/cpmp/pdfs/Holland_Chrtn_Stud_Achieve.pdf.

Direct Instruction

Grossen, B.J. 2002. The BIG Accommodation Model: The Direct Instruction Model for Secondary Schools. Journal of Education for Students Placed at Risk, 7(2), 241– 263.

Ligas, M.R., and Vaughan, D.W. 1999. Alliance of Quality Schools: 1998–99 Evaluation Report. Broward, Florida: Broward County Schools.

Edison Schools

American Federation of Teachers. 1998. Student Achievement in Edison Schools: Mixed Results in an Ongoing Enterprise. Washington, D.C.: Author. Retrieved 10/28/04 from: http://www.aft.org/research/downloads/charter/Edison1998.pdf.

Edison Schools. 2001. Fourth Annual Report. Retrieved 11/2/04 from http://www.edisonschools.com/annualreport2001.pdf.

Miron, G., and Applegate, B. 2000. An Evaluation of Student Achievement in Edison Schools Opened in 1995 and 1996. Western Michigan University: The Evaluation Center. Retrieved 11/1/04 from http://www.wmich.edu/evalctr/edison/edison.html.

Nelson, F.H., and Van Meter, N. 2003. Update for Student Achievement in Edison Schools Inc. Washington, D.C.: American Federation of Teachers. Retrieved 10/28/04 from http://www.aft.org/research/downloads/charter/Edison2003.pdf.

Integrated Mathematics, Science, and Technology (IMaST)

Center for Mathematics, Science and Technology (n.d.). Success of the IMaST Program. Normal, IL: Center for Math, Science and Technology, Illinois State University. Retrieved 11/1/04 from http://www.ilstu.edu/depts/cemast/imast/success.htm.

Interactive Mathematics Program (IMP)

IMP Students Score Higher Than Traditional Peers (n.d.). Interactive Mathematics Program Resource Center. Retrieved 11/1/04 from http://www.mathimp.org/research/evaluation/article7.html.

24 Philadelphia IMP Research Summary. (n.d.). Retrieved 11/1/04 from http://www.gphillymath.org/StudentAchievement/Reports/SupportData/PhilIMPR esearch.htm.

Research Supporting the Interactive Mathematics Program. 2004. Emeryville, Calif.: Key Curriculum Press. Retrieved 11/1/04 from http://www.mathimp.org/downloads/IMPWhitePaper.pdf.

Standardized Tests: Highlights from Current Studies of IMP Student Performance. (n.d.). Retrieved 11/1/04 from http://www.mathimp.org/research/evaluation/article2.html.

Summary of IMP vs. Non-IMP Students. (n.d.). Retrieved 11/1/04 from http://www.gphillymath.org/StudentAchievement/Reports/SupportData/PassRate Compare.pdf.

Turner, S. 1999. IMP Students Score Higher Than Their Peers on SAT9. Can This be Mathematically Correct? IMP. Retrieved 11/1/04 from http://www.mathimp.org/downloads/research/SylviaTurnerArticle.pdf.

Webb, N.L. 2002. The Impact of the Interactive Mathematics Program on Student Learning. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp.375–408). Mahwah, N.J.: Lawrence Erlbaum Associates.

Wolff, N. 1994. Test Date: October 1994 Sophomore PSAT Scores. Retrieved 11/1/04 from http://www.gphillymath.org/StudentAchievement/Reports/SupportData/ PSATScoreCompare.pdf.

MATH Connections

Cichon, D. and Ellis, J.G. 2002. The Effects of MATH Connections on Student Achievement, Confidence and Perception. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 345–374). Mahwah, N.J.: Lawrence Erlbaum Associates.

What Does Research Say About the MATH Connections Program and Student Achievement? (n.d.). Retrieved 11/1/04: http://www.mathconnections.com/evaluation/evalsum.html.

Mathematics in Context (MiC)

25 Romberg, T.A. and Shafer, M.C. 2002. Mathematics in Context (MiC)—Preliminary Evidence About Student Outcomes. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 225–250). Mahwah, N.J.: Lawrence Erlbaum Associates.

Webb, D. and Meyer, M. (n.d.). Summary Report of Student Achievement Data for Mathematics in Context: A Connected Curriculum for Grades 5–8. Madison, Wisc.: Wisconsin Center for Education Research. Retrieved 11/5/04 from http://mic.britannica.com/mic/common/MicAndStudentAchievement /mic_report.pdf.

Mathematics: Modeling Our World (MMOW/ARISE)

Abeille, A., and Hurley, N. 2001. Final Evaluation Report: Mathematics Modeling Our World (MMOW). Stoneham, Mass.: Learning Innovations. Retrieved 11/1/04 from http://www.comap.com/highschool/projects/mmow/FinalReport.pdf.

Mathematics with Meaning

Garet, M., and Le Floch, K.C. 2003. Evaluation of Mathematics with Meaning and Textual Power 2002–2003 School Year Final Report. Washington, D.C.: American Institutes for Research.

LeFloch, K.C. 2004, October. Evaluation of Mathematics with Meaning and Textual Power 2003–2004 School Year Final Report. Washington, D.C.: American Institutes for Research.

MATH Thematics

Billstein, R., and Williamson, J. 2002. Middle Grades MATH Thematics: the STEM Project. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 251–281). Mahwah, N.J.: Lawrence Erlbaum Associates.

Reys, R., Reys, B., Lapan, R., Holliday, G., and Wasman, D. 2003. Assessing the Impact of Standards-Based Middle Grades Mathematics Curriculum Materials on Student Achievement. Journal for Research in Mathematics Education, 34 (1), 74–95.

Prentice Hall: Tools for Success

26 Gatti, G.G. (n.d.). Prentice Hall Mathematics Algebra 1 National Effect Size Study Executive Summary. Lebanon, Ind.: Prentice Hall. Retrieved 11/1/04 from: http://www.phschool.com/Research/math/pdfs/national_effectsize_study_alg1.pdf

Gatti, G.G. (n.d.). Prentice Hall Mathematics Middle Grades Math National Effect Size Study Executive Summary. Lebanon, Ind.: Prentice Hall. Retrieved 11/1/04 from http://www.phschool.com/Research/math/pdfs/national_effectsize_study_ mgm.pdf.

Prentice Hall Mathematics. 2004. Program Efficacy Studies 1998–2003: Clinical Research Reports Supporting the Efficacy of the Prentice Hall Mathematics Program. Lebanon, Ind.: Prentice Hall. Retrieved 11/1/04 from: http://www.phschool.com/Research/math/pdfs/pes_1998_2003.pdf

Saxon Math: An Incremental Development

Saxon Publishers. 2004. Scientific Research Base for Saxon Math K–12: Foundational Research and Program Efficacy Studies. Norman, Okla.: Author. Retrieved 11/1/04 from: http://www.saxonpublishers.com/pdf/research/saxon_math_research.pdf.

Systemic Initiative for Montana Mathematics and Science (SIMMS)

Lott, J., Hirstein, J., Burke, M., Lundin, M., Allinger, G., Souhrada, T.A., Walen, S., and Preble, D. 2002. Curriculum and Assessment in SIMMS Integrated Mathematics. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards-Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 399–423). Mahwah, N.J.: Lawrence Erlbaum Associates.

University of Chicago School Mathematics Project (UCSMP)

Senk, S. 2002. Effects of UCSMP Secondary School Curriculum on Students’ Achievement. In Sharon L. Senk and Denisse R. Thompson (Eds.), Standards- Based School Mathematics Curricula: What are They? What do Students Learn? (pp. 425–456). Mahwah, N.J.: Lawrence Erlbaum Associates.

27 Science Curriculum Resources

Meta-analyses of Science Curricula

Lott, G.W. 1983. The effect of inquiry teaching and advance organizers upon student outcomes in science education: A meta-analysis of selected research studies. Journal of Research in Science Teaching, 20, 437–51.

Shymansky, J.A., Hedges, L.V., and Woodworth, G. 1990. A Re-Assessment of the Effects of Inquiry-Based Science Curricula of the Sixties on student Achievement. Journal of Research in Science Teaching, 27(2), 127–44.

Shymansky, J.A., Kyle, W.C., and Alport, J.M. 1983. The Effects of New Science Curricula on Student Performance. Journal of Research in Science Teaching, 20, 387–404.

Wise, K.C., and Okey, J.R. 1983. A Meta-Analysis of the Effects of Various Science Teaching Strategies on Achievement. Journal of Research in Science Teaching, 20, 419–35.

Weinstein, T., Boulanger, F.D., and Walberg, H.J. 1982. Science Curriculum Effects in High School: A Quantitative Synthesis. Journal for Research in Science Teaching, 19, 511–22.

BSCS Science: An Inquiry Approach

BSCS 2004. Evidence of Student Achievement. Colorado Springs, CO: BSCS. Retrieved 11/19/2004 from http://www.bscs.org/library/Results.pdf.

Center for Learning Technologies in Urban Schools (LeTUS)

Geier, R., Blumenfeld, P., Marx, R., Krajcik, J., Fishman, B., and Soloway, E. 2004. Standardized Test Outcomes of Urban Students Participating in Standards and Project-Based Science Curricula. In Y.B. Kafai, W.A. Sandoval, N. Enyedy, A.S. Nixon, and F. Herrera (Eds.), Proceedings of the Sixth International Conference of the Learning Sciences (pp. 206–213). Santa Monica, Calif.: Erlbaum.

Marx, R.W., Blumenfeld, P.C., Krajcik, J.S., Fishman, B., Soloway, E., Geier, R., and Tal, R.T. (in press). Inquiry-Based Science in the Middle Grades: Assessment of Learning in Urban Systemic Reform. Journal of Research in Science Teaching.

Rivet, A.E., and Krajcik, J.S. (in press). Achieving Standards in Urban Systemic Reform: An Example of a Sixth Grade Project-Based Science Curriculum. Journal of Research in Science Teaching.

28 Constructing Ideas in Physical Science (CIPS)

Smith, P.S., and Banilower, E. 2002. Constructing Ideas in Physical Science (CIPS) Evaluation Report. Chapel Hill, N.C.: Horizon Research, Inc.

DESIGNS/DESIGNS II

Sadler, P.M. 2003. Evaluation Supplement to the Final DESIGNS Report. Cambridge, Mass.: Harvard-Smithsonian Center for Astrophysics.

Event-Based Science (EBS)

Rosenbaum, H. 1996. Evaluation of Event-Based Earth Science Project for the National Science Foundation. Rockville, Md.: Montgomery County Public Schools.

Expeditionary Learning Outward Bound (ELOB)

Borman, G.D., Hewes, G.M., Overman, L.T., and Brown, S.A. 2002. Comprehensive School Reform and Student Achievement: A Meta-Analysis (CRESPAR Report No. 59.) Baltimore, Md.: Center for Research on the Education of Students Placed At Risk, Johns Hopkins University. Retrieved 8/2/04 from http://www.csos.jhu.edu/crespar/techReports/Report59.pdf.

Foundational Approaches in Science Teaching (FAST)

Curriculum Research and Development Group. 1999. Foundational Approaches in Science Teaching (FAST) Summary of Evaluations. Honolulu, HI: Author.

Foundational Approaches in Science Teaching. (n.d.). Submission to the Expert Panel on Mathematics and Science Education.

Full Option Science System (FOSS)

Amaral, O.M., and Garrison, L. 2002. Helping English Learners Increase Achievement Through Inquiry-Based Science Instruction. Bilingual Research Journal, 26(2), 213–39.

Rose-Baele, J.S. 2003. Report of Fifth Grade Outcome Study, Science for All Students, 2001–2002. (NSF Report No. 03-226). Arlington, Va.: National Science Foundation. Retrieved 8/26/04 from http://www.ccsdschools.com/administration/assessment/PIpage.html.

29 Global Lab Curriculum (GLC)

Young, V.M., Haertel, G., Ringstaff, C., and Means, B. 1998. Evaluating Global Lab Curriculum: Impacts And Issues Of Implementing A Project-Based Science Curriculum. Menlo Park, Calif.: SRI, International.

Great Explorations in Mathematics and Science (GEMS)

Sneider, C., Kurlich, K., Pulos, S., and Friedman, A. 1984. Learning to Control Variables With Model Rockets: A Neo-Piagetian Study of Learning in Field Settings. Science Education, (68)4, 463–84.

Sneider, C.I., and Ohadi, M.M. 1998. Unraveling Students’ Misconceptions About the Earth’s Shape and Gravity. Science Education, 82, 265–84.

High Schools that Work (HSTW)

Frome, P. 2001. High Schools that Work: Findings from the 1996 and 1998 Assessments. Research Triangle Park, N.C.: Research Triangle Institute. Retrieved on 10/1/04 from http://www.sreb.org/programs/hstw/ResearchReports/ RTI_study.pdf.

Kaufman, P., Bradby, D., and Teitelbaum, P. 2000. High Schools that Work and Whole School Reform: Raising Academic Achievement of Vocational Completers Through the Reform of School Practice. Berkeley, Calif.: National Center for Research in Vocational Education, University of California, Berkeley.

Integrated Mathematics, Science, and Technology Curriculum (IMaST)

Satchwell, R.E., and Loepp, F.L. 2002. Designing and Implementing an Integrated Mathematics, Science, and Technology Curriculum for the Middle School. Journal of Industrial Teacher Education, (39)3. Retrieved on 10/20/04 from http://scholar.lib.vt.edu/ejournals/JITE/v39n3/satchwell.html.

Issues, Evidence, and You (IEY)/SEPUP

Wilson, M., Sloane, K., Roberts, L., and Henke, R. 1995. SEPUP Course I, Issues, Evidence and You: Achievement Evidence from the Pilot Implementation. Berkeley, Calif.: University of California, Berkeley. Retrieved on 10/15/2004 from http://www-gse.berkeley.edu/research/BEAR/Publications/Sepup95.pdf.

30 Learning by Design (LBD)

Holbrook, J.K., Gray, J., Fasse, B., Camp, P., and Kolodner, J. 2001. Assessment and Evaluation of the Learning by Design™ Physical Science Unit, 1999–2000: A Document in Progress. Atlanta, GA: Georgia Institute of Technology. Retrieved on 10/15/04 from http://www.cc.gatech.edu/projects/lbd/Conference_Papers/html/ eval_results/evaluation_results_99-00.html.

Kolodner, J.L., Gray, J., and Fasse, B.B. 2003. Promoting Transfer through Case-Based Reasoning: Rituals and Practices in Learning by Design Classrooms. Cognitive Science Quarterly, (3)2, 183–232.

Modeling Instruction in High School Physics

Hestenes, D. 2000. Findings of the Modeling Workshop Project (1994–00). Arlington, Va.: National Science Foundation. Retrieved on 10/28/2004 from http://modeling.la.asu.edu/R&E/ModelingWorkshopFindings.pdf.

U.S. Department of Education Expert Panel. 2001. Expert Panel Review: Modeling Instruction in High School Physics. Washington, D.C.: Office of Educational Research and Improvement. Available: http://www.ed.gov/pffices/OERI/ORAD/KAD/expert_panel/math-science.html

National Science Curriculum for High Ability Learners

U.S. Department of Education Expert Panel. 2001. Expert Panel Review: National Science Curriculum for High Ability Learners. Washington, D.C.: Office of Educational Research and Improvement. Retrieved on 11/1/04 from http://www.ed.gov/offices/OERI/ORAD/KAD/expert_panel/highabilityproj.html.

VanTassel-Baska, J., Bass, G., Ries, R., Poland, D., and Avery, L.D. 1998. A National Study of Science Curriculum Effectiveness with High Ability Students. Gifted Child Quarterly, (42)4, 200–11

Physics Resources and Instructional Strategies for Motivating Students (PRISMS)

Hartman, D. (n.d.) Documentation of Structured Analysis for Selecting Scientifically- Based Research: Instructional Strategies and Programs. [Review of unpublished raw data: Unruh, R. Physics resources and instructional strategies for motivating students.]

Unruh, R., Countryman, L., and Cooney, T. 1992. The PRISMS Approach: A Spectrum of Enlightening Physics Activities. The Science Teacher, (59)5.

31 Science 2000/Science 2000+

California State University, Fresno. 1996. Instructional Technology Project: Research Report. Fresno, Calif.: Author.

U.S. Department of Education Expert Panel. 2001. Expert Panel Review: Science 2000. Washington, D.C.: Office of Educational Research and Improvement. Retrieved on 11/1/04 from http://www.ed.gov/offices/OERI/ORAD/KAD/expert_panel/ science2000.html.

Science and Technology Concepts for Middle Schools (STC/MS)

Yeh, S.S., and Pedulla, J.J. 2001. Evaluation of the Science and Technology Concepts for Middle Schools (STC/MS) Program, Phase II. Boston, Mass.: Center for the Study of Testing, Evaluation and Educational Policy, Boston College.

The Science Curriculum Improvement Study (SCIS)

Bredderman, T. 1983. Effects of Activity-Based Elementary Science on Student Outcomes: A Quantitative Synthesis. Review of Educational Research, 53(4), 499–518.

World Watcher/Learning about the Environment Curriculum (LATE)

Crawford, V., and Toyamsa, Y. 2002. World Watcher/Learning About the Environment Curriculum Final External Evaluation Report. Menlo Park, Calif.: SRI International. Retrieved on 10/20/04 from http://www.ctl.sri.com/publications/ downloads/Evaluation_of_LATE_Report1.pdf.

32 Professional Development Resources

Carpenter, T.P., Fennema, E., Peterson, P.L., Chiang, C.P., and Loef, M. 1989. Using Knowledge of Children's Mathematics Thinking in Classroom Teaching: An Experimental Study. American Education Research Journal, 26, 499–531.

Carpenter, T.P., Empson, S.B., Fennema, E., Franke, M.L., and Levi, L. 2000. Cognitively Guided Instruction: A Research-Based Teacher Professional Development Program for Elementary School Mathematics. Madison, Wisc.: National Center for Improving Student Learning and Achievement in Mathematics and Science, University of Wisconsin.

Cobb, P., Wood, T., Yackel, E., Nicholls, J., Wheatley, G., Trigatti, B., and Perlwitz, M. 1991. Assessment of a Problem-Centered Second-Grade Mathematics Project. Journal for Research in Mathematics Education, 22(1), 13–29.

Cohen, D.K., and Hill, H.C. 1998. State Policy and classroom Performance: Mathematics Reform in California (Policy Brief RB-23). University of Pennsylvania: Consortium for Policy Reform in Education. Retrieved 9/8/04 from http://www.cpre.org/Publications/rb23.pdf.

Good, T.L., and Grouws, D.A. 1979. The Missouri Mathematics Effectiveness Project: an Experimental Study in Fourth-Grade Classrooms. Journal of Educational Psychology, 71, 355–62.

Good, T.L., Grouws, D.A., and Ebmeier, H. 1983. Active Mathematics Teaching. New York, NY: Longman.

Hamilton, L.S., McCaffrey, D.F., and Stecher, B.M., Klein, S.P., Robyn, A., and Bugliari, D. 2003. Studying Large-Scale Reforms Of Instructional Practice: An Example from Mathematics and Science. Educational Evaluation and Policy Analysis, 25(1), 1–29.

Harlen, W. 2004. Evaluating Inquiry-Based Science Developments. Washington, D.C.: National Research Council. Retrieved 10/15/04 from http://www7.nationalacademies.org/bose/WHarlen_Inquiry_Mtg_Paper.pdf.

Kahle, J.B., and Rogg, S.R. 1996. A Pocket Panorama of the Landscape Study, 1995. Oxford, Ohio: Miami University

Kennedy, M. 1998. Form and Substance in In-Service Teacher Education (NISE Research Monograph No. 13). Madison, Wisc.: National Center for Improving Science Education, University of Wisconsin. Retrieved 9/15/04 from http://www.wcer.wisc.edu/NISE/Publications/Research_Monographs/vol13.pdf.

33 Lawrenz, F., and McCreath, H. 1988. Integrating Quantitative and Qualitative Evaluation Methods to Compare Two Teacher Inservice Training Programs. Journal of Research in Science Teaching, 25, 397–407.

Marek, E.A., and Methven, S.B. 1991. Effects of the Learning Cycle Upon Student and Classroom Teacher Performance. Journal of Research in Science Teaching, 28, 41–53.

Mason, D.A., and Good, T.L. 1993. Effects of Two-Group and Whole-Class Teaching on Regrouped Elementary Students' Mathematics Achievement. American Education Research Journal, 30, 328–60.

Otto, P.B., and Schuck, R.F. 1983. The Effect of a Teacher Questioning Strategy Training Program on Teaching Behavior, Student Achievement, and Retention. Journal of Research in Science Teaching, 20, 521–28.

Radford, D.L. 1998. Transferring Theory Into Practice: A Model for Professional Development for Science Education Reform. Journal of Research in Science Teaching, 35, 73–88.

Rose-Baele, J.S. 2003. Report of Fifth Grade Outcome Study, Science for All Students, 2001–2002 (Project Inquiry Report: No. 03-226). Arlington, Va.: National Science Foundation. Retrieved 8/26/04 from http://www.ccsdschools.com/administration/assessment/PIpage.html.

Rubin, R.L., and Norman, J.T. 1992. Systematic Modeling Versus the Learning Cycle: Comparative Effects of Integrated Science Process Skill Achievement. Journal of Research in Science Teaching, 29, 715–27.

Smith, E.L., Blakeslee, T.D., and Anderson, C.W. 1993. Teaching Strategies Associated with Conceptual Change Learning in Science. Journal of Research in Science Teaching, 30, 111–26.

Stallings, J., and Krasavage, E.M. 1986. Program Implementation and Student Achievement in a Four-Year Madeline Hunter Follow Through Project. The Elementary School Journal, 87(2), 117–38.

Stevens, R.J., and Slavin, R.E. 1995. The Cooperative Elementary School: Effects on Students' Achievement, Attitudes, and Social Relations. American Educational Research Journal, 32, 321–51.

34 Supovitz, J.A., and Turner, H.M. 2000. The Effects of Professional Development on Science Teaching Practices and Classroom Culture. Journal of Research in Science Teaching, 37, 963–80.

Villasenor, A., and Kepner, H.S. 1993. Arithmetic from a Problem-Solving Perspective: An Urban Implementation. Journal for Research in Mathematics Education, 24, 62–70.

Wood, T., and Sellers, P. 1996. Assessment of a Problem-Centered Mathematics Program: Third Grade. Journal for Research in Mathematics Education, 27, 337– 53.

35 Selected International References

Ainley, J.G. 1997. Australia. In David Robitaille (Ed.), National Contexts for Mathematics and Science Education (pp. 39–49). Vancouver, Canada: Pacific Educational Press.

Council of Ministers of Education, Canada. 2003. Attracting, Developing and Retaining Effective Teachers. (OECD Country Background Report) Paris, France: OECD.

Davis, C.J. 1997. England. In David Robitaille (Ed.), National Contexts for Mathematics and Science Education (pp. 119–129). Vancouver, Canada: Pacific Educational Press.

Dekkers, J., Highway, B., and de Laeter, J. 2001. Enrollment Trends in School Science Education in Australia. International Journal of Science Education, (23)5, 487– 500.

Ingvarson, L. 2003. Building a Learning Profession. Australian Council for Educational Research, Policy Brief (Issue 3, November). Retrieved 11/2/04 from http://www.acer.edu.au/publications/documents/PolicyBriefIssue3 BuildLearningprofession_000.pdf.

Jenkins, E.W. 2000. The Impact of the National Curriculum on Secondary School Science Teaching in England and Wales. International Journal of Science Education, (22)3, 325–36.

Lokan, J., and Greenwood, L. 2000 Mathematics Achievement at Lower Secondary Level in Australia. Studies in Educational Evaluation, 26, 9–26.

Program for International Student Assessment (PISA) data source: http://www.pisa.oecd.org/.

Ross, A. and Hutchings, M. 2003. Attracting, Developing and Retaining Effective Teachers in the United Kingdom of Great Britain and Northern Ireland. (OECD Country Background Report). Paris, France: OECD.

Schmidt, W.H., McKnight, C., Valverde, G.A., Houang, R.T., and Wiley, D.E. 1997. Many Visions, Many Aims. Volume 1: A Cross-National Investigation of Curricular Intentions in School Mathematics. Dordrecht, The Netherlands: Kluwer Academic.

Schmidt, W.H., McKnight, C., Valverde, G.A., Houang, R.T., and Wiley, D.E. 1997b. Many Visions, Many Aims Volume 2: A Cross-National Investigation of Curricular Intentions in School Science. Dordrecht, The Netherlands: Kluwer Academic.

36 Shilbeck, M., and Connell, H. 2003. Attracting, Developing and Retaining Effective Teachers. Australia Country Background Report. Paris, France: OECD.

Trends in International Mathematics and Science Study (TIMSS) data source: http://timss.bc.edu/.

Taylor, A.R. 1997. Canada. In David F. Robitaille (Ed.), National Contexts for Mathematics and Science Education. (pp. 70–81). Vancouver, Canada: Pacific Educational Press.

Valverde, G.A., and Schmidt, W.H. 2000. “Greater Expectations: Learning From Other Nations in the Quest for ‘World-Class Standards’ in U.S. School Mathematics and Science.” Journal of Curriculum Studies, (32)5, 651–87.

37 Appendix A: Detailed Findings of the Review of Math Curricula

This appendix provides a list of the mathematics curricula included in our review, together with a description of each curriculum and its related evaluation studies. Table 1 lists the mathematics curricula identified as having evaluation studies that met our criteria.

Table 1: Math Curricula with Studies Grades Curriculum Name Subject Matter Covered K12 Edison Schools Mathematics

K 8 Direct Instruction Mathematics

middle school Cognitive Tutor Mathematics *[sex, race/ethnicity]

middle school Connected Mathematics (CMP) Mathematics *[sex, race/ethnicity] middle school Integrated Mathematics, Science, and Technology (IMaST) Mathematics

middle school Mathematics in Context (MiC) Mathematics *[race/ethnicity] 6 8 MATHThematics Mathematics

middle school Prentice Hall: Tools for Success Mathematics

middle school Saxon Math: An Incremental Development Mathematics

7 12 Mathematics with Meaning Mathematics *[sex, race/ethnicity] 7 12 University of Chicago School Mathematics Project (UCSMP) Mathematics

9 12 College Preparatory Mathematics (CPM) Mathematics

9 12 Contemporary Mathematics in Context: A Unified Approach (Core-Plus Mathematics Mathematics Project CPMP)

9 12 Interactive Mathematics Program (IMP) Mathematics *[sex] 9 12 MATH Connections Mathematics *[sex, race/ethnicity] 9 12 Mathematics: Modeling Our World (MMOW/ARISE) Mathematics

9 12 Systemic Initiative for Montana Mathematics and Science (SIMMS) Mathematics

11-12 Advanced Placement (AP) Calculus Calculus

Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that their use in instruction elicits higher achievement/performance in sudents than other curricula to which they are compared on both standardized and/or state tests AND on curriculum developed tests. There are several curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. * An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets.

A - 1 A Description of the Mathematics Curricula and Evaluation Studies Found

In addition to a description of each curriculum, these overviews include the type of student achievement measure used, the number and direction of the results, and, if available, the size of any differences between groups. If the results were broken out by sex and/or race and ethnicity, this too is indicated.

The types of measures used in the impact studies are broken into five categories: • standardized achievement tests (i.e., PSAT, SAT, SAT-9, and Iowa Test of Basic Skills) • state-mandated achievement tests (i.e., FCAS, MEAP, and MCAS) • standards-based curriculum-driven measures (i.e., Balanced Assessment, Problem Situation Test) • teacher-based measures (i.e., GPA, school tests, and mathematics courses taken) • percent passing different mathematics courses

In the charts that follow within this appendix, results were tracked rather than the number of studies. Multiple grades and multiple measures were counted as multiple results and broken out as such. For example, a study that looked at the impact of different subsets of a curriculum for sixth, seventh, and eighth grades was counted as three results. Cohort studies that tracked students across multiple grades were counted as one result. Also counted as one result were findings from different sites within the same study using the same measure. If a majority of sites had changes favoring the tested curriculum, the result was indicated as positive. If the majority of sites did not differ from the comparison, the result was indicated as no change. If a majority of sites had changes favoring the comparison curriculum, the result was indicated as negative.

Note: Most impact studies for math curricula reported the statistical significance of their results. Only differences that have reached the conservative minimum acceptable statistical significance level of .05 were included in the results reported for each study. If differences are statistically significant, then there is another measure, called an effect size, which shows how big the difference is. In our description of the study results, we provide effect sizes where available, although very few studies reported these results. Effect sizes greater than .4 are considered large; between .2 and .4 are considered moderate; and less than .2 are considered small.

A - 2 Advanced Placement (AP) Calculus

The AP Calculus curriculum includes two courses: AP Calculus AB, which is comparable to one semester of college-level calculus, and AP Calculus BC, which is comparable to two semesters of college-level calculus. Both courses include elementary functions, limits and continuity, and differential and integral calculus, with Calculus BC covering these topics more extensively than Calculus AB. Prerequisites include knowledge of analytic geometry and elementary functions in addition to college preparatory algebra, geometry, and trigonometry. Enrolled students are expected to take the Advanced Placement examination in Calculus AB or BC.

Contact: College Board Headquarters 45 Columbus Avenue New York, NY 10023-6992 Tel: 212-713-8066 Web site: http://apcentral.collegeboard.com/article/0,3045,151-165-0-2178,00.html

Results: Five results were found from studies on the effect of AP Calculus courses. These studies compared college mathematics performance of students who took AP Calculus with other students who did not.

Type of measure Number of Results Effect size results Teacher-based 3 AP students scored higher No effect size measures/GPA in two results; there were reported no differences in one result. Pass rates/courses 2 AP students scored higher No effect size taken in one result, and there reported were no differences in one result.

Results were not reported by sex, race, or ethnicity.

A - 3 Cognitive Tutor

The Cognitive Tutor (CT), from Carnegie Learning, includes full curricula in Algebra I, Geometry, Algebra II, an Integrated Math Series, and Quantitative Literacy Through Algebra. Each curriculum combines software-based, individualized computer lessons with collaborative, real-world problem-solving activities. Students spend about 40 percent of their class time using the software and the remainder of their time engaged in classroom problem-solving activities.

Contact: Carnegie Learning, Inc. 1200 Penn Avenue Suite 150 Pittsburgh, PA 15222 Tel: 888-851-7094 E-mail: [email protected]

Results: Twenty-one results from studies on the effect of CT were found, all of which focused on Algebra I at the middle and high school level.

Type of measure Number of Results Effect size results Standardized 8 CT students scored higher Moderate effect achievement tests in all eight results. sizes in one result Statewide tests 5 CT students scored higher No effect size in three results; there reported were no differences in two results. Curriculum-driven, 7 CT students scored higher Large effect sizes in skill-specific tests in seven results. two results Passing rates 1 CT students had higher No effect size passing rates. reported

Two results looked at sex differences; in one result, CT boys’ scores were higher than those of comparison boys. There were no differences between boys in one result, and there were no differences for the girls’ scores in either result. Two results reported differences by race and ethnicity. In one result, African-American CT students had higher scores than comparison African-American students, and there were no differences for Hispanic students. In the second result, Hispanic CT students scored higher than comparison Hispanic students.

A - 4 College Preparatory Mathematics (CPM)

The CPM series offers a four-year integrated curriculum in which mathematics topics are revisited and built upon through the years. Problem-solving strategies are emphasized as a vehicle for learning mathematics, and student study teams are an integral part of the learning process. Based on the belief that concept mastery requires time, the curriculum spirals through practice of the main course concepts throughout each year and emphasizes students' supportive group work. The program sees the teacher's role as a guide.

Contact: CPM Business Office 1233 Noonan Drive Sacramento, CA 95822 Tel: 916-681-3611 Web site: http://www.cpm.org/

Results: Six results from studies on the effect of CPM were found, one of which focused on algebra and five focused on grades 9 through 11 math.

Type of measure Number of Results Effect size results Standardized 5 There were no differences No effect size achievement tests in all five results. reported Teacher-developed 1 There were no No effect size measure differences. reported

Results were not reported by sex, race, or ethnicity.

A - 5 Connected Mathematics (CMP)

The Connected Mathematics (CMP) curriculum is composed of eight models, each focusing on one important area of mathematics and emphasizing previously learned content. Connected Mathematics is designed to develop students’ knowledge and understanding of mathematics through attention to connections: between mathematical ideas and their applications in the world outside school; among the core ideas in mathematics; among the strands in a modern mathematics curriculum; and between the planned teaching-learning activities and the special aptitudes and interests of middle school students.

Contact: Pearson Education P.O. Box 2500 Lebanon, IN 46052-3009 Tel: 800-848-9500 Web site: http://www.phschool.com/math/cmp/

Results: Thirty-four results from comparison studies looking at the effect of CMP were found focusing on middle grade mathematics.

Type of measure Number of Results Effect size results Standardized 11 CMP students scored No effect size achievement tests higher for six results; they reported scored lower for one result; and there were no differences for four results. Statewide tests 16 CMP students scored No effect size higher for 14 results; reported there were no differences for two results. Curriculum-driven, 6 CMP students scored Large effect sizes skill-specific tests higher for all six results. were found for one result. Teacher-based 1 CMP students scored No effect size measure higher in one result. reported Seven results were broken out by race/ethnicity and two by sex. No significant sex differences were found. African-American CMP students were found to score higher than African-American comparison students in six results; while Hispanic CMP students scored higher than Hispanic comparison students in four results; in a fifth result, there were no differences. African-American and Hispanic CMP students showed greater gains than others in two results, while African Americans alone showed greater gains in one result. In one result, Native-American CMP student performance decreased.

A - 6 Contemporary Mathematics in Context: A Unified Approach (Core-Plus Mathematics Project CPMP)

Core-Plus Mathematics consists of a single core sequence for both college-bound and employment-bound students during the first three years of high school. A flexible fourth- year course can be used to prepare students for college mathematics.

Contact: Core-Plus Mathematics Project Department of Mathematics Western Michigan University Kalamazoo, MI 49008-5248 Tel: 866-407-CPMP (2767) E-mail: [email protected] Web site: http://www.wmich.edu/cpmp/

Results: Fifty-six results from comparison studies on the effect of CPMP were found focusing on high school math.

Type of measure Number of Results Effect size results Standardized 43 In 19 of the results, No effect size achievement tests CPMP students scored reported higher; in 24 results, there were no differences. Statewide tests 5 Students scored higher in No effect size all five results. reported Curriculum-driven, 2 Students scored higher in No effect size skill-specific tests both results. reported Teacher-based 6 CPMP students scored No effect size measure/GPA higher in one result, no reported differently in another, and lower in four results.

Results were not reported by sex, race, or ethnicity.

A - 7 Direct Instruction In Direct Instruction (DI), each program is fully scripted, from what the teacher says and anticipated student responses, to correctional procedures. Each skill is broken down into its component parts, and then each component of the skill is taught to mastery. Afterward, the skills are combined within a larger context where they may be utilized across settings, resulting in generalized fluency. The DI mathematics curriculum covers kindergarten through eighth grade, but focuses primarily on kindergarten through sixth grade. The mathematics curriculum covers 19 different topics ranging from addition and subtraction to money, mathematics study skills (graphs, charts, maps, and statistics), and geometry.

Contact: The McGraw-Hill Companies P.O. Box 182604 Columbus, OH 43272 Tel: 888-772-4543 Web site: http://www.sraonline.com/index.php/home/curriculumsolutions/di/connectingmath/114

Results: Nine results from comparison studies on the effect of DI, focusing on seventh and eighth grade math, were found.

Type of measure Number of Results Effect size results Standardized 6 DI students scored higher No effect size achievement tests for four results; they reported scored lower for two results. State test 3 In all three results, DI No effect size students scored higher. reported

Results were not reported by sex, race, or ethnicity.

A - 8 Edison Schools

Founded in 1992, Edison Schools is focused on raising student achievement through research-based school design, aligned assessment systems, interactive professional development, integrated use of technology, and other program features, including a longer school day and school year. The math curriculum for grades 6 through 8 includes applied arithmetic, prealgebra, and pregeometry, using a spiral curriculum approach to teach concepts and ideas. The mathematics in grades 9 through 10 provides three years of high school math in two years’ time, using an integrated application-based approach to algebra, geometry, and trigonometry with additional emphasis on probability, statistics, and discrete mathematics. Grades 11 and 12 incorporate advanced mathematical modeling, Calculus and Statistics, Algebra 2, and Precalculus, as well as AP Statistics and Calculus.

Contact: 521 Fifth Avenue, 11th Floor New York, NY 10175 Tel: 212-419-1600 Web site: http://www.edisonschools.com/

Results: Seventeen results from comparison studies on the effect of Edison Schools were found; fifteen focus on middle school math and two focus on high school math.

Type of measure Number of Results Effect size results Standardized 8 For four results, Edison No effect size achievement tests students scored higher; in reported the remaining four, there were no differences. State test 9 For two results, Edison No effect size students scored higher; reported for four results, Edison students scored lower; and for three results, there were no differences.

Results were not reported by sex, race, or ethnicity.

A - 9 Integrated Mathematics, Science, and Technology (IMaST)

The Integrated Mathematics, Science, and Technology program provides integrated sixth, seventh, and eighth grade curricula that promote hands-on learning for students and teamwork among teachers from different disciplines. IMaST emphasizes learning based on constructivist theory and active student participation involving a hands-on approach comprising a wide variety of activities.

Contact: Ronjon Publishing, Inc. 1001 S. Mayhill Rd. Denton, TX 76208 Tel: 800-262-3060 Web site: http://www.ilstu.edu/depts/cemast/programs/imast.shtml

Results: One study that looked at the effect of IMaST on seventh and eighth grade students was found.

Type of measure Number of Results Effect size results Standardized 1 IMaST students scored No effect size achievement tests higher for the one result. reported

Results were not reported by sex, race, or ethnicity.

A - 10 Interactive Mathematics Program (IMP)

This four year, problem-based curriculum incorporates traditional branches of mathematics (algebra, geometry, and trigonometry) with additional topics recommended by the NCTM Standards, such as statistics, probability, curve fitting, and matrix algebra. Students are encouraged to experiment, investigate, ask questions, make and test conjectures, reflect, and accurately communicate their ideas and conclusions. Although each unit has a specific mathematical focus, other topics are brought in as needed to solve the central problem. Ideas that are developed in one unit are usually revisited and deepened in one or more later units. Algebra and geometry are distributed throughout the four years.

Contact: Key Curriculum Press 1150 65th Street Emeryville, CA 94608 Tel: 800-995-MATH (6824) Web site: http://www.keypress.com/catalog/products/textbooks/Prod_IMP.html

Results: Twenty-two results from comparison studies were found on the effect of IMP, all focusing on high school. Type of measure Number of Results Effect size results Standardized 12 IMP students scored No effect size achievement tests higher for eight results; reported there were no differences for four results. Pass rates or courses 5 In all five results, IMP No effect size taken students had higher reported passing rates and/or were more likely to take more mathematics courses. Curriculum-driven, 3 IMP students scored Large effect sizes skill-specific tests higher for all three were found in the results. three results. Teacher-based 2 In both results, IMP No effect size measure students scored higher. reported

Results were not reported by race or ethnicity. The one result reported by sex found IMP girls were slightly more apt to continue three or more years in math than IMP boys, while the reverse was the case for comparison students.

A - 11 MATH Connections

MATH Connections (MC) is an integrated curriculum that blends ideas from traditionally separate mathematical fields (e.g., algebra, geometry, statistics, and discrete mathematics) in ways that blur the lines between them. This three-year curriculum replaces the traditional Algebra I, Geometry, Algebra II sequence and is designed for all students in grades 9, 10, and 11, with honors students beginning the curriculum in grade 8.

Contact: 750 Old Main Street Suite 303 Rocky Hill, CT 06067-1567 Tel: 860-721-7010 Web site: http://www.mathconnections.com

Results: Fifteen results from comparison studies, all of which focused on high school students, were found on the effect of MC.

Type of measure Number of Results Effect size results Standardized 6 MC students scored No effect size achievement tests higher for three results; reported for three results, there were no differences. State test 8 MC students scored There was a large higher for all eight effect size for three results. results. Curriculum-driven, 1 No difference was found. N/A skill-specific test

One result looked at sex, race, and ethnic differences and found no differences within MC students.

A - 12 Mathematics in Context (MiC)

Mathematics in Context is a four-year middle school curriculum (grades 5 through 8) that encourages students to discover mathematical concepts and skills through engaging problems and meaningful contexts. Each year includes lessons in the four strands (numbers, algebra, geometry and statistics, and probability) that are interwoven through 10 units. For example, sample algebra units include Patterns and Symbols (grade 5), Expressions and Formulas (grade 6), Ups and Downs (grade 7), and Graphing Equations (grade 8).

Contact: Holt, Rinehart, and Winston Attn: Ms. Web1 10801 N. MoPac Expressway Building 3 Austin, TX 78759 Tel: 800-HRW-9799 (800-479-9799) Web sites: http://www.hrw.com/math/mathincontext/ http://mic.britannica.com/mic/common/home.asp

Results: Twenty results from comparison studies, which focused on middle school students, were found on the effect of MiC.

Type of measure Number of Results Effect size results Standardized 17 In 15 results, MiC In two results, a achievement tests students scored higher; in majority of classes two results, there were no showed at least differences. moderate effect sizes. State test 3 MiC students scored No effect size higher for all three reported results.

One result was broken out by race. No statistically significant difference was found between African-American students using MiC and comparison students.

A - 13 Mathematics: Modeling Our World (MMOW/ARISE)

In Mathematics: Modeling Our World (MMOW), students are taught to use a variety of resources to solve problems and to choose resources that meet the needs of a particular situation. As in real life, MMOW’s problems do not necessarily have perfect solutions. MMOW works to strengthen the students’ ability to solve problems by setting goals and thinking strategically about how to achieve these goals, solving problems through trial and error and/or process of elimination, using technology like calculators and computers, and working together to solve semi-structured problems and communicating the solutions.

Contact: W.H. Freeman and Company 41 Madison Avenue New York, NY 10010 Tel: 800-446-8923 Web sites: http://www.whfreeman.com/highschool/contact_hs_rep.asp http://www.comap.com/highschool/projects/mmow/introduction.htm

Results Four results related to MMOW were found, one focusing on middle school students and three focusing on high school students.

Type of measure Number of Results Effect size results Standardized 4 For three results, MMOW No effect size achievement tests students scored higher; reported for one result, there were no differences.

Results were not reported by sex, race, or ethnicity.

A - 14 Mathematics with Meaning

Mathematics with Meaning (MwM) is not a complete curricular program; rather, it is a combination of professional development, instructional strategies, and carefully planned materials designed to alter the pedagogy and content of middle school and high school mathematics courses in order to improve student achievement. The program consists of instructional units that teachers may use on a supplementary basis or as their entire instructional program. MwM takes a student-centered approach based on exploratory learning and problem solving, focusing on developing conceptual understanding, connections, and communication with mathematical concepts through frequent group work and hands-on activities.

Contact: College Board Dept CBO P.O. Box 869010 Plano, TX 75074 Tel: 212-713-8260 Tel: 800-323-7155 E-mail: [email protected] Web site: http://www.collegeboard.com

Results: Eleven results from comparison studies on the effect of MwM were found; three focused on middle school math achievement and eight on high school math achievement.

Type of measure Number of Results Effect size results

Statewide tests 11 In six results, there were No effect reported no differences; in five results, MwM students scored higher.

Six results were broken out by sex, four by race. In the six results where sex differences were given, boys slightly outperformed girls. In four results where race/ethnic differences were given, achievement scores were significantly lower for African-American students than other students in both MwM and comparison groups.

A - 15 MATH Thematics

Middle Grades MATH Thematics (STEM) is a three-year curriculum designed for use in grades 6 through 8. Four unifying concepts—Proportional Reasoning, Multiple Representations, Patterns and Generalizations, and Modeling—are used across the three years with seven content strands: Number, Measurement, Geometry, Statistics, Probability, Algebra, and Discrete Mathematics.

Contact: McDougal Littell Customer Service Center A Houghton Mifflin Company 1900 S. Batavia Geneva IL 60134 Tel: 617-351-5326 Tel: 800-462-6595 Web sites: http://www.mcdougallittell.com/ http://www.classzone.com/math_middle.cfm

Results Six results from comparison studies were found on the effect of STEM.

Type of measure Number of Results Effect size results Standardized 2 In one result, STEM No effect size achievement tests students scored higher; in reported the second, there were no differences. Statewide tests 2 In one result, STEM No effect size students scored higher; in reported the second, there were no differences. Curriculum-driven, 2 In both results STEM No effect size skill-specific tests students scored higher. reported

Results were not reported by sex, race, or ethnicity.

A - 16 Prentice Hall: Tools for Success

In Prentice Hall Math (PHM) various mathematical strands (such as number sense, algebra, geometry, measurement, data analysis, and problem solving) are integrated throughout the series to ensure that students are prepared for subsequent mathematics courses at the high school level. Each lesson includes a Think and Discuss section that presents the new material along with questions to get students actively thinking about and discussing important concepts. Textbooks in this curriculum include Work Together activities that allow students to work in groups, often doing hands-on activities to reinforce math topics.

Contact: Pearson Education P.O. Box 2500 Lebanon, IN 46052-3009 Tel: 800-848-9500 Web site: http://www.phschool.com/math/

Results: Eleven results from comparison studies on the effect of PHM were found.

Type of measure Number of Results Effect size results Standardized 8 PHM students scored Moderate effect achievement tests higher in five results; in 3 sizes in one result results, there were no differences. Statewide tests 2 PHM students scored No effect size higher in both results. reported Curriculum-driven, 1 PHM students scored No effect size skill-specific tests higher in this result. reported

Results were not reported by sex, race, or ethnicity.

A - 17 Saxon Math: An Incremental Development

Saxon Math (SM) is a kindergarten through grade 12 curriculum that systematically distributes instruction, practice, and assessment throughout the academic year rather than concentrating concepts in a single unit or chapter. Each increment builds upon the foundation of earlier increments, to lead students toward a deeper understanding of mathematical concepts. Instruction of related concepts is spread throughout the grade level, ensuring that students have an opportunity to master each concept before they are introduced to the next one.

Contact: Saxon Publishers 2600 John Saxon Blvd. Norman, OK 73071 Tel: 800-284-7019 E-mail: [email protected]

Results: A total of eight results from comparison studies on the impact of SM were found, focusing on both middle and high school.

Type of measure Number of Results Effect size results Standardized 3 SM students scored No effect size achievement tests higher in all three results. reported Statewide tests 3 SM students scored No effect size higher in all three results. reported Curriculum-driven, 1 SM students scored No effect size skill-specific tests higher in this result. reported Teacher-based 1 SM students scored No effect size measure/GPA higher in this result. reported

Results were not reported by sex, race, or ethnicity.

A - 18 Systemic Initiative for Montana Mathematics and Science (SIMMS)

The SIMMS curriculum is divided into six levels, each consisting of one year of work. Level one is typically offered to ninth graders, followed by level two in grade 10. After completing level two, students may choose between levels three and four and then proceed to either level five or six in the subsequent year. The sequence for potential math and science majors is (1) level one, (2) level two, (3) level four, (5) level six. Levels one and two offer basic mathematical literacy.

Contact: Kendall/Hunt Publishing 4050 Westmark Drive P.O. Box 1840 Dubuque, IA 52004-1840 Tel: 800-542-6657 Web sites: http://www.simms-im.com http://www.montana.edu/~wwwsimms/

Results: Twelve results from comparison studies were found on the effect of SIMMS. Four results each focused on levels one and two, and two each focused on levels four and six.

Type of measure Number of Results Effect size results Standardized 6 No differences were found in the six No effect achievement tests results. size reported Curriculum-driven, 6 In one result, SIMMS students No effect skill-specific tests scored higher; no differences were size reported found in the other five.

Results were not reported by sex, race, or ethnicity.

A - 19 University of Chicago School Mathematics Project (UCSMP)

The UCSMP secondary curriculum consists of six courses: Transitional Mathematics; Algebra; Geometry; Advanced Algebra; Functions, Statistics, and Trigonometry; and Pre-Calculus and Discrete Mathematics. Transitional Mathematics, which was originally designed for average to above average seventh graders (but can be started earlier or later), weaves together three more or less equal strands of major content: applied arithmetic, prealgebra, and elementary geometry. Algebra, geometry, and some discrete mathematics are integrated into all courses, as are statistics and probability.

Contact: UCSMP 5835 South Kimbark Avenue Chicago, IL 60637 Tel: 773-702-1130 Web site: http://socialsciences.uchicago.edu/ucsmp/

Results: Fourteen results were found from comparison studies on the effect of UCSMP. The following UCSMP courses were covered: Transitional Mathematics (2), Algebra (2), Geometry (5), Advanced Algebra (4), and Pre-Calculus and Discrete Mathematics (1).

Type of measure Number of Results Effect size results Standardized 5 In all five results, No effect size achievement tests UCSMP students scored reported higher. Curriculum-driven, 8 In all eight results, No effect size skill-specific tests UCSMP students scored reported higher. Teacher-based 1 No differences were No effect size measure/GPA found. reported

Results were not reported by sex, race, or ethnicity.

A - 20 Appendix B: Detailed Findings of the Review of Science Curricula

This appendix provides a list of the science curricula included in our review, together with a description of each curriculum and its related evaluation studies. Table 2 below lists the science curricula that we identified as having evaluation studies that met our criteria.

Table 2: Science Curricula with Studies Grades Covered Curriculum Name Subject Matter K 12 Expeditionary Learning Outward Bound (ELOB) Whole School Reform *[LEP] K 8 Full Option Science System (FOSS) Multi-Science *[LEP] PK 8 Great Explorations in Math and Science (GEMS) Math & Multi-Science K 6 The Science Curriculum Improvement Study (SCIS) Multi-Science

28 National Science Curriculum for High Ability Learners Multi-Science

68 DESIGNS/DESIGNS II Physical Science *[sex] 68 Integrated Math, Science, and Technology (IMaST) Math, Science & Technology

6 8 Center for Learning Technologies in Urban Schools (LeTUS) Multi-Science *[sex] 68 Learning by Design (LBD) Multi-Science 68 Science 2000/Science 2000+ Multi-Science 6 8 Science and Technology Concepts for Middle Schools (STC/MS) Multi-Science 69 Event-Based Science (EBS) Earth Science

78 Constructing Ideas in Physical Science (CIPS) Physical Science *[sex] 79 Foundational Approaches in Science Teaching (FAST) Multi-Science

79 Multi-Science, Global Lab Curriculum (GLC) Environmental Focus 79 Issues, Evidence and You (IEY)/SEPUP Multi-Science 9 11 BSCS: An Inquiry Approach Multi-Science 9 12 High Schools that Work (HSTW) Whole School Reform 9 12 Modeling Instruction in High School Physics Physics *[sex] 912 World Watcher/Learning about the Environment (LATE) Environmental Science *[urban]

10-12 Physics Resources and Instructional Strategies for Motivating Students Physics (PRISMS)

Note: Shaded curricula are those for which we have found the strongest evidence of effectiveness, that is, quantitative evidence that 1) their use in instruction elicits higher achievement/performance in students than other curricula to which they are compared on both standardized and/or state tests AND on curriculum developed tests, or 2) they showed large effect sizes in terms of increasing student achievement. All science curricula listed in Table 2, however, have credible evaluations that show evidence of effectiveness. There are several curricula for which this evidence of effectiveness has not been collected but which might also qualify as effective should appropriate studies be conducted. An omission from this list of many curricula signifies merely that these curricula have not yet provided quantitative evidence of effectiveness that meets our criteria. * An asterisk marks curricula for which effectiveness data are provided for subgroups of students, indicated in brackets.

B - 1

A Description of the Science Curricula and Evaluation Studies Found

The attached descriptions of science curricula, which appear in alphabetical order, include the type of student achievement measure used, the number and direction of the results, and, if available, the size of any differences between groups. When known, effect sizes have been listed. Where results have been provided by race/ethnicity, sex, or other demographic characteristics, we have reported these. The absence of such notation means that no data were reported by subgroup. In the case of three curricula, descriptions of studies do not follow the normal format because results reported are not best described in that particular format.

Note: Almost all impact studies for science curricula reported the statistical significance of their results. Only differences that have reached the conservative minimum acceptable statistical significance level of .05 were included in the results reported for each study. If differences are statistically significant, then there is another measure, called an effect size, which shows how big the difference is. In our description of the study results, we provide effect sizes where available, although few studies reported these results. Effect sizes greater than .4 are considered large, between .2 and .4 are considered moderate, and less than .2 are considered small.

B - 2 Biological Sciences Curriculum Study (BSCS): An Inquiry Approach

This program introduces ninth-, tenth-, and eleventh- grade students to the core concepts in inquiry, the physical sciences, the life sciences, and the earth-space sciences as articulated in the National Science Education Standards. In addition, the curriculum engages students in integration across the disciplines in relevant contexts that explore the standards related to science in a personal and social perspective. This program provides high school students with an alternative to the traditional sequence of biology, chemistry, and physics. Included with this program is a professional development component designed to help teachers and school districts implement the materials.

Contact: Pamela Van Scotter Director The BSCS Center for Curriculum Development BSCS 5415, Mark Dabling Blvd. Colorado Springs, Colorado 80918 Web site: http://www.bscs.org/page.asp?id=curriculum_development|high_school_9- 12|An_Inquiry_Approach

Results: Most of the evaluative work on BSCS was of the materials, not student achievement; however, the results of a nationwide field test are summarized below.

Type of measure Number of Results Effect size studies Content test, 1 Average student gains at No effect sizes unknown if self- both ninth- and tenth- reported designed or state- grade levels were issued between 20 and 25 percent. Note: Statistical significance levels were not reported.

The field test was conducted across 10 states and included students in urban, suburban, and rural schools. However, data regarding specific demographic groups was not provided.

B - 3 Center for Learning Technologies in Urban Schools (LeTUS)

Center for Learning Technologies in Urban Schools (LeTUS), developed by researchers at the University of Michigan and Detroit public school teachers, includes project-based curriculum materials that build from district, state, and national standards to support the development of integrated science understanding for middle school students. The materials support students' science learning through engaging them in inquiry about real- world problems, providing them with multiple opportunities to work with concepts, and integrating the use of learning technologies in instruction. LeTUS is focused on learning about and developing a new machine to construct large buildings and bridges, an area that has been identified as of interest to young urban students.

Contact: Joseph Krajcik School of Education University of Michigan 610 East University Avenue, Rm. 4109 Ann Arbor, MI 48109-1259 E-mail: [email protected]

Results: Two evaluations of the LeTUS program were found, one that utilized skill-specific instruments and one that used statewide achievement test scores to measure effectiveness.

Type of measure Number of Results Effect size studies Curriculum-driven, 1 Significant content and Large effect size for skill-specific tests process gains that content achievement increased with program and more moderate revision and scale-up effects for process skills. Statewide tests 1 Students who completed Moderate effect size at least one LeTUS unit in for all three content seventh or eighth grade areas and both outperformed peers in all process areas. content and process Moderate effects in categories measured by increasing test the test. passing rates Program-designed 1 Sixth graders showed Large effect size for measures significant overall achievement gains improvement on pre- and pre- and post-test post-test measures over a four-year period.

The study looking at differences by sex indicated that participation in at least one LeTUS unit is associated with an apparent reduction in boy-girl achievement differences on statewide examinations.

B - 4 Constructing Ideas in Physical Science (CIPS)

Constructing Ideas in Physical Science (CIPS) is an inquiry-based, yearlong physical science course that attempts to engage seventh- or eighth - middle school students in constructing meaningful understanding of physical science concepts. The CIPS course is based on the themes of interactions and energy transfers between objects. CIPS has five units. Each unit consists of two or three cycles of activities designed to help students develop physics and chemistry concepts.

Contact: CIPS Project 6475 Alvarado Road, Suite 206 San Diego, CA 92120 Fax: 619-594-1581 E-mail: [email protected]

Publisher: It's About Time, Inc. Tel: 888-698-TIME (8463)

Results:

Type of measure Number of Results Effect size studies Multiple-choice 1 CIPS students scored Small content items higher than non-CIPS Multiple-choice comparisons after Small process items controlling for prior knowledge, student Open-ended content demographics, and Moderate items weeks of instruction in Physical Science.

CIPS participation did not appear to have closed either sex (male/female) or racial (white/Asian versus non-Asian minority) achievement gaps on either multiple-choice content or process questions or open-ended content items.

B - 5 DESIGNS/DESIGNS II

Project DESIGNS (Doable Engineering Science Investigations Geared for Non-science Students), developed by the Harvard-Smithsonian Center for Astrophysics, includes design-based activity modules for use in physical science and technology education courses in grades five through nine. The project's six topics cover chemistry, static forces, electricity and magnetism, potential and kinetic energy, energy transfer, and force, work, power, and torque. Designs II resulted from a follow-up effort to develop a full-year middle school (grades seven through nine) physical science course based on the modules. Project pedagogy was derived from the constructivist model of learning and takes into account students' personal theories. The project's goal was to open science concepts to students through activities involving the design, construction, and optimization of simple devices.

Contact: Program overview: Web site: http://cfa-www.harvard.edu/sed/resources/designsinfo.html

Publisher: Kendall/Hunt Web site: http://www.kendallhunt.com/index.cfm?PID=219&PGI=152

Results:

Type of measure Number of Results Effect studies size Self-designed test to 1 Designs II students gained over Large measure process skills control students. Self-designed tests to 1 (same as Students with initially lower scores Large measure conceptual above) gain more. knowledge

Analysis by sex for conceptual knowledge shows no significant difference in gains.

B - 6 Event-Based Science (EBS)

The Event-Based Science (EBS) series is a module-based program designed for students in grades six through nine, with a focus on current events. The series has 18 modules designed to last four to six weeks, each focusing on different themes and concepts across the domains of earth, life, and physical sciences. The modules may be sequenced over all middle school grade levels and combined with other instructional materials in order to build a comprehensive middle school science program. One or two modules typically are used in a year, with teachers selecting particular units based on the district's science standards, the local curriculum program, the interests of the student population, and their own background knowledge in specific topics. EBS is not a “stand alone” curriculum. By design, teachers and students supplement each module with additional data about a specific event from various resources included as part of the materials or suggested by the program.

Contact: Russell G. Wright Montgomery County Public Schools 850 Hungerford Drive Rockville, MD 20850 Tel: 800-327-7252 Fax: 301-279-3153 E-mail: [email protected] Web site: http://www.mcps.k12.md.us/departments/eventscience/

Results: One interim study of the impact of EBS was found. This study measures the first three years of the EBS project. The report of the final findings from the six-year project is not yet available.

Type of measure Number of Results Effect size studies Curriculum-driven, In two of the three tested Small skills-specific, years, EBS students multiple-choice test outperformed the control group, after controlling for prior science performance. Science attitudes EBS students displayed Small survey 1 more positive attitudes about science than the control group. Task-based EBS students outperformed Moderate performance the control group, assessment rubric controlling for prior science performance.

B - 7 Expeditionary Learning Outward Bound (ELOB)

As a whole-school reform effort, Expeditionary Learning Outward Bound (ELOB) for kindergarten through twelfth grade organizes curriculum, instruction, assessment, school culture, and school structures around producing high quality student work in learning expeditions. These expeditions are long-term, in-depth investigations of themes or topics designed to engage students both in and beyond the classroom through projects, fieldwork, and service. Learning expeditions are designed with clear learning goals aligned with district and state standards. Ongoing assessment is woven throughout each learning expedition.

Contact: Linda Collins Outward Bound USA 100 Mystery Point Road Garrison, NY 10524 E-mail: [email protected] Web site: http://www.elob.org/

Results: Only one study of many found for ELOB looked at science achievement. Two schools are included in this study, but only one includes data for science.

Type of measure Number of Results Effect size studies Standardized tests 1 ELOB students showed steady No effect size gains in science. reported

One of the schools in the study experienced an increase of about 22 percent in immigrant students (who were limited English proficient over the five years accounted for in the evaluation. The school showed consistent gains in all subject areas. This same school also has a high number of students qualifying for free and reduced-price lunch. The evaluation concludes that ELOB, in this instance, is “particularly successful with a challenging, normally low-achieving population.”

B - 8 Foundational Approaches in Science Teaching (FAST)

Foundational Approaches in Science Teaching (FAST) is a three-volume, comprehensive curriculum program based on a constructivist philosophy of learning in which students construct their own knowledge through experiential, hands-on learning. Investigations help students build on existing knowledge and reinforce conceptual understanding throughout their work. The continual reinforcement and return to concepts allows students to achieve a deep understanding of the material and to arrive at that understanding at different points in the curriculum. FAST puts the teacher in the role of “director of research.” The program emphasizes an instructional strategy that is based on the teacher's question development practices and other techniques that encourage students to think critically. It is heavily laboratory-based, with most concepts taught through laboratory experiences in which students develop skills in measurement and lab procedures.

Contact: The University of Hawaii The Curriculum Research and Development Group, Science Section 1776 University Avenue UHS Building 2, Room 202 Honolulu, HI 96822-2463 Tel: 808-956-6918 Fax: 808-956-4933 E-mail: [email protected] Web site: http://www.hawaii.edu/crdg/FAST.pdf

Results:

Type of Measure Number Results Effect size of studies Laboratory skills 2 No effect size FAST students outperformed Process skills 1 reported non-FAST comparison group. Knowledge skills 1 Statewide test 1 In one of two years tested, No effect size scores FAST students scored reported significantly higher than the state average, after controlling for student background. Standardized test 2 FAST students outperformed No effect size scores non-FAST students in one reported study. In the second, both FAST and non-FAST groups scored above national norms.

B - 9 Full Option Science System (FOSS)

The Full Option Science System (FOSS) was developed by the Lawrence Hall of Science, University of California, Berkeley. Funded by the National Science Foundation (NSF), FOSS combines science content and process with a goal of increasing scientific literacy for students and instructional efficiency for teachers. The curriculum is organized into topical units, called courses, under each of three strands: Earth and Space Science, Life Science, and Physical Science and Technology. Each course is an in-depth unit requiring 9 to 12 weeks of instruction. The units have approximately 10 investigations, each with three to seven parts. The system advocates that students should learn important scientific concepts and develop the ability to think well if they are engaged in situations in which they actively construct ideas through their own explorations, applications, and analyses.

Contact: FOSS Project Lawrence Hall of Science, University of California Berkeley, CA 94720 Phone: 510-642-8941 Fax: 510 642-7387 Email: [email protected] Web sites: http://www.lhsfoss.org/ http://www.delta-education.com/science/foss/index.shtml

Results: Two studies of FOSS materials met our criteria, though in each of these, use of FOSS was included as only one part of a larger curriculum or professional development intervention.

A 2003 evaluation of the NSF-sponsored Project Inquiry, a broad professional development intervention including the adoption and use of FOSS materials, revealed that fifth-grade students of teachers trained in kit use performed significantly higher on both multiple-choice and constructed/open-ended response assessments than did the control group. Self-reported hours of teacher professional development were also positively associated with science achievement. The authors conclude that professional development is associated with better implementation of inquiry-based instruction and greater science topic coverage, which is in turn associated with science achievement.

A 2002 study of the use of multiple kit- and inquiry-based science materials (including FOSS) in fourth and sixth-grade classes for English language learners revealed a positive relationship between years in the science program and standardized science test scores. This relationship remained after controlling for the students’ increasing English language proficiency.

B - 10 Global Lab Curriculum (GLC)

The Global Lab Curriculum (GLC) was a four-year project to create a science course emphasizing student collaborative inquiry. Organized around six units, GLC culminates in the design and conduct of original student investigations. The structure purposefully provides substantial guidance to students in initial investigations and gradually peels away the support to allow students to exercise acquired skills in designing and conducting their own collaborative experiments. GLC is also designed to capitalize on the Internet as both a communication and motivational tool that helps establish a cross- cultural science community.

Contact: Harold McWilliams TERC Center for Earth and Space Science Education 2067 Massachusetts Ave. Cambridge, MA 02140 Tel: 617-547-0430 E-mail: [email protected] Web site: http://CESSE.terc.edu

Results:

Type of measure Number of Results Effect size studies Self-designed tests 1 Students with low levels of prior No effect knowledge benefit more from GLC size reported than those with medium or high levels of prior knowledge.

B - 11 Great Explorations in Mathematics and Science (GEMS)

Great Explorations in Math and Science (GEMS) is a supplemental enrichment program for students from preschool through eighth-grade. GEMS provides teachers with more than 70 teacher's guides, support documents, and pedagogical handbooks; professional development opportunities; an active web site; and a national support network of GEMS leaders and associates and over 45 regional sites. GEMS uses generally accessible materials that integrate science and mathematics. The program's units, presented as flexible enhancements or in curriculum sequence, are designed to help all teachers reach all students and feature clear, step-by-step teacher instructions. In addition to the specific standards-based learning goals and content, the program emphasizes cooperative learning and problem solving, literature and language arts connections, and real-world relevance. GEMS units feature an inquiry-based, guided-discovery, student-centered approach to learning. An assessment component is in place for the entire series.

Contact: Jacqueline Barber Lawrence Hall of Science University of California 1 Centennial Drive Berkeley, CA 94720-5200 Tel: 510-642-7771 Fax: 510-643-0309 Contact e-mail: jbarber@ berkeley.edu Program e-mail: gems@ berkeley.edu Web site: http://www.lhs.berkeley.edu/GEMS/

Results: Several studies of the GEMS units, which serve kindergarten through eighth grade, have been produced. Only one of these met our criteria and focused on the fourth through eighth-grade astronomy unit Earth, Moon, and Stars.

Type of measure Number of Results Effect size studies Content-specific, 3 Students receiving No effect sizes skills-driven test GEMS instruction reported had significantly higher pre–post gains than control groups without GEMS instruction.

B - 12 High Schools that Work (HSTW)

High Schools that Work (HSTW) is a whole-school, research- and assessment-based reform that offers a framework of goals and key practices for improving the academic, technical, and intellectual achievement of high school students. HSTW blends traditional college-preparatory content with quality technical and vocational studies. HSTW provides technical assistance and staff development focused on techniques and strategies such as teamwork, applied learning, and project-based instruction. The HSTW assessment is based on the National Assessment of Educational Progress (NAEP). The developer does not offer specific subject area programs, but consultants provide workshops customized to fit an individual school's needs.

Contact: Gene Bottoms, Senior Vice President Southern Regional Education Board Tel: 404-875-9211, extension 249 E-mail: [email protected] Web site: http://www.sreb.org/programs/hstw/hstwindex.asp

Results: Two studies were conducted using the same measure.

Type of measure Number of Results Effect size studies HSTW assessment 1 Gains in achievement increase with No effect size the number of students in a school reported completing the HSTW curriculum. HSTW assessment 1 Over time, more students met No effect size achievement goals and completed reported program.

B - 13 Integrated Mathematics, Science, and Technology (IMaST)

Integrated Mathematics, Science, and Technology Curriculum (IMaST) was developed by the Center for Mathematics, Science, and Technology at Illinois State University. IMaST is a standards-based, integrated curriculum for grades six through eight. IMaST integrates technology, science, and mathematics and includes connections to the language arts and social sciences, as well as readings that profile typical careers related to the curriculum content. The curriculum is based on the constructivist learning theory that allows concept development to take place in a structured venue.

Contact: Center for Mathematics, Science, and Technology Illinois State University Campus Box 5960 Normal, IL 61790-5960 Tel: 309-438-3089 Fax: 309-438-3592 E-mail: [email protected] Web site: http://www.ilstu.edu/depts/cemast/programs/imast.shtml

Results:

Type of measure Number of Results Effect size studies Trends in 1 IMaST students No effect size International outperform traditional reported Mathematics and peers, especially Science Study regarding science (TIMMS) sub-test processes.

B - 14 Issues, Evidence and You (IEY)/ The Science Education for Public Understanding Program (SEPUP)

The University of California, Berkeley Lawrence Hall of Science developed the SEPUP/IEY curriculum for middle school and junior high use with support from the National Science Foundation. Issues, Evidence and You (IEY) focuses on environmental issues in a social context. The program builds upon earlier SEPUP modules, is designed to address students’ increasing ability to think abstractly, and builds upon students’ need for peer interaction and support. The developer intended the curriculum to serve as the physical science component of an integrated science program (physical, life, and earth science) or as a year-long physical science program. The course consists of 65 activities or investigations presented in a conceptual sequence. The instructional times of the activities vary from one to three class periods. The curriculum can accompany Science and Life Issues (SALI), another SEPUP program now in the piloting phase. The program also integrates student assessment into the curriculum, providing teachers with a basis for understanding gaps in student knowledge of core scientific concepts and a plan for addressing such gaps as they are identified.

Contact: Web site: http://www.sepup.com/index.htm Publisher LAB-AIDS Tel: 800-381-8003

Results: An assessment of the pilot implementation of IEY was conducted as an experimental study using pre- and post-tests of student knowledge for two groups: one taught using IEY and another using the traditional science curriculum. The study spanned 15 multi- school sites in 12 states, covering 26 classrooms and a total of 830 students. Student achievement was measured by the assessments from the IEY program, which consisted of multiple choice and short answer items.

Type of measure Number of Results Effect size studies Curriculum-driven, 1 IEY students Moderate skill-specific tests experienced an increase in ability to present an evidence-based line of reasoning, while comparison classes did not experience significant improvement in this skill.

B - 15 Learning by Design (LBD)

Developed by Georgia Tech's EduTech Institute, Learning by Design (LBD) is an approach to science learning in which middle school students learn as a result of collaboratively engaging in design activities and reflecting appropriately on their experiences. Design problem-solving is the scaffolding of LBD; interventions combine teacher facilitation, paper-and-pencil design diaries and other paper prompts, and software tools and prompts. LBD has units in both physical and earth science.

Contact: The EduTech Institute Georgia Institute of Technology 801 Atlantic Drive Atlanta, Georgia, 30332-0280 Tel: 404-894-3807 Web site: http://www.cc.gatech.edu/edutech/projects/lbdview.html

Results:

Type of measure Number of Results Effect size studies Self-designed test 1 LBD gains in content learning No effect incorporating some higher than comparison students. size reported standardized test Typical LBD students do as well or items (NAEP, better than honors-level non-LBD TIMSS) students. Performance 2 Typical LBD students score higher No effect assessments than typical companion group size reported students in applying collaborative science skills and practices. Typical LBD students’ performance was on par with non-LBD honors students, and LBD honors students outperformed non-LBD honors students.

B - 16 Modeling Instruction in High School Physics

Modeling Instruction in High School Physics is grounded in the thesis that scientific activity centers on modeling: the construction, validation, and application of conceptual models to understand and organize the physical world. The program uses computer models and modeling to develop the content and pedagogical knowledge of high school physics teachers and train them to be leaders in science teaching reform and technology infusion. The program relies heavily on professional development workshops to equip teachers with a teaching methodology. Teachers are trained to develop student abilities to make sense of physical experience, understand scientific claims, articulate coherent opinions of their own, and evaluate evidence in support of justified belief. For example, students analyze systems using graphical models, mathematical models, and pictorial diagrams called system schema.

Contact: David Hestenes Director, Modeling Instruction Program Department of Physics and Astronomy Arizona State University P.O. Box 871504 Tempe, AZ 85287-1504 Tel: (480) 965-6277 E-mail: [email protected] Web Site: http://modeling.la.asu.edu/modeling-HS.html

Results: Four evaluations of the impact of Modeling Instruction on student achievement were found.

Type of measure Number of Results Effect size studies Tests of alternate methods 2 Average pre–post test gains in No effect size of physics instruction Modeling Instruction reported. (Comparison of average classrooms were double those classroom scores between in traditional classrooms, and teachers using Modeling Instruction, traditional 10 percentage points higher methods, and reform than those in reform-method methods) classrooms. Tests of alternate methods 2 Post-test scores after a teacher No effect size of physics instruction is trained in Modeling reported. (Comparison of average Instruction were between 6 classroom scores between and 10 percentage points teachers pre-Modeling Instruction training and the higher than their classroom same teachers post-training) averages before training. Data from three studies show that male students consistently outperform female students.

B - 17 National Science Curriculum for High Ability Learners

The National Science Curriculum for High Ability Learners Project is a supplemental program that has been implemented across grades two through eight with a broad group of students within the average-to-gifted range of ability. The curriculum units employ problem-based learning for engaging students in the study of the concept of systems, specific science content, and the scientific research process. Students engage in a scientific research process that leads them to create their own experiments and design their own solutions to each unit's central problem. The units encourage in-depth study, and content areas cover a breadth of scientific subject matter drawn from the physical, life, and earth sciences. Each unit constitutes approximately 30 hours of instruction, with students typically receiving two units within an academic year. Major components of the program include a curriculum framework that contains goals and learning outcomes linked to individual lesson plans; embedded and post assessments that focus on science content, concept, and process learning; 25 lesson plans that address these goals and outcomes with relatively equal emphasis on each of the goals; and a real-world problem that serves as the catalyst for subsequent learning in the unit.

Contact: Joyce VanTassel-Baska Center for Gifted Education College of William and Mary 427 Scotland St. Williamsburg, VA 23185 Tel: 757-221-2362 Fax: 757-221-2184 E-mail: [email protected] Web Site: http://cfge.wm.edu

Results:

Type of measure Number of Results Effect size studies Open-ended 1 Students in National Science High assessment to test Curriculum (NSC) classrooms scored gifted science better on one unit tested when students compared to students in non-NSC classrooms. Open-ended 4 Same as above High assessment to test gifted science students

B - 18 Physics Resources and Instructional Strategies for Motivating Students (PRISMS)

The goal of Physics Resources and Instructional Strategies for Motivating Students (PRISMS) is to provide learning activities to promote understanding of physics principles in the context of experiences relating to the daily lives of secondary school students. PRISMS includes a guide with over 130 activities in the form of student instructions and teacher notes with background information on the activities. The program’s resources include several videotapes from which students make observations and take data, and recommended software and interfacing for schools that have access to microcomputers. A complete student evaluation and testing program is included in a three- to four-diskette set. The instructional strategies blend exploratory activities, concept development, and application activities to stimulate problem-solving skills and the understanding of major physics concepts. The guide can be integrated with the use of any physics textbook and is designed to be individualized to meet the needs of each teacher. The guide also contains activities dealing with scientific, technological, and social issues as well as career information. The new version is called PRISMS PLUS.

Contact: Roy D. Unruh, Director Physics Department, Room 303 University of Northern Iowa Cedar Falls, IA 50614-0150 Tel: 319-273-2380 Fax: 319-273-7136 E-mail: [email protected] Web Site: http://www.prisms.uni.edu/

Results:

Type of measure Number of Results Effect size studies State assessment 2 PRISMS students have higher No effect achievement gains than size reported comparison students. Program-designed 1 PRISMS students’ gains in No effect measures reasoning and problem-solving size reported skills were greater than those of comparison students.

B - 19 Science 2000/Science 2000+

Science 2000+ (previously known as Science 2000) is a multimedia, multiyear science curriculum for high schools that takes an integrated, thematic approach to the earth, life, and physical sciences. At each grade level, the yearlong course includes four nine-week units, connected by central themes and a storyline—a narrative that sets a real-world context for the science content. Each unit poses problems related to real-life scientific and social issues. Students address these problems by drawing information from CD-ROM- based resources (text, images, video, simulations) supplemented by laser disc, web references, and manipulative kits. Multimedia resources (text, images, and video) are cross-referenced and linked within the CD.

Contact: Ellen M. Nelson Decision Development Corporation 2303 Camino Ramon, Suite 220 San Ramon, CA 94583-1389 Tel: 800-835-4332 or 925-830-8896 Fax: 925-830-0830 E-mail: [email protected] Web site: http://www.ddc2000.com

Results:

Type of measure Number of Results Effect size studies Self-designed test 1 All Science 2000+ No effect size instruments students showed gains in reported content knowledge from pre- to post-test. Study showed the curriculum to be effective in increasing the content knowledge of all students, regardless of gender, ethnicity, or language classification.

B - 20 Science and Technology Concepts for Middle Schools (STC/MS)

Science and Technology Concepts for Middle Schools (STC/MS) is a modular program composed of 24 units. There are four units for each grade level, one each in the following strands: life science, earth science, physical science, and technology. Each STC unit generally has 16 lessons with hands-on investigations. Teachers can use the four modules to comprise the science curriculum for the entire school year or use one or two individual units as supplements to other curriculum pieces. Eight modules for grades seven and eight are currently under development. When completed, STC/MS will include eight units for science in grades seven and eight. The instructional units will be balanced among life, earth, physical sciences, and technological design. The components are designed to be offered as two one-year courses (one unit from each of the scientific strands) or as four single semester courses.

Contact: National Science Resources Center 901 D Street SW, Suite 704B Washington, DC 20560-0952 Web site: http://www.nsrconline.org/curriculum_resources/middle_school.html

Publisher: Carolina Biological Tel: 800-227-1150 Web site: http://www.carolina.com

Results: In a 2001 study of four of the eight STC/MS modules, a post-test-only design was used to compare the performance between groups receiving STC/MS instruction and those who received traditional instruction. The impact of the curriculum on student achievement in each particular content area was measured by multiple-choice and short-answer tests developed to measure concepts specific to each unit, including TIMSS and NAEP. When possible, test items were taken from previously existing assessments so that national and international comparisons were possible. In all four STC/MS curriculum units, students demonstrated significantly higher performance than control groups and national/international comparison groups. The quasi-experimental evaluation design, however, makes it difficult to control for prior knowledge or instruction of students in comparison groups.

B - 21 The Science Curriculum Improvement Study (SCIS)

The Science Curriculum Improvement Study (SCIS) was developed at the Lawrence Hall of Science at the University of California between 1962 and 1974 for use in grades kindergarten through six. The goal of the program is the development of scientific literacy, defined as a combination of basic knowledge concerning the natural environment, investigating ability, and curiosity. The program consists of 12 units, one life and one physical science unit at each elementary grade level. About 10 major concepts are developed each year. The concepts are interrelated and are intended to provide a conceptual framework for the child's thinking. Opportunities are provided for developing science processes as well. The general instructional pattern is free exploration of new materials, the introduction of a new concept, and the application of the new concept in a range of new situations.

Contact: Note: Science Curriculum Improvement Study (SCIS) is now called SCIS 3+

Publisher: Delta Education Tel: 800-258-1302 Web Site: http://www.delta-education.com/scisgallery.aspx?collection=N&menuID=11

Results: A 1983 meta-analysis of 57 controlled studies of SCIS and two other activity-based science programs (Elementary Science Study and Science—A Process Approach) draws conclusions about the programs across process, content, and affective outcomes. Seventy percent of the studies were dissertations, and a conservative estimate of the combined students tested is 13,000 in more than 900 classrooms. Seventy-nine percent of the studies had a quasi-experimental design. Forty-eight percent of the studies tested effects after more than one year of program use.

The overall effect of these three programs on all outcome areas was positive, though not dramatically so; thirty-two percent of comparison studies had statistically significant results favoring the treatment group, while six percent favored the nontreatment group. These results support an overall conclusion of a positive program effect. The mean effect size for all studies, with all outcomes weighted equally, was .35, or a 14 percentile point improvement over non-activity-based instruction. The effects on measures of science process, intelligence, and creativity were positive. Small positive effects were observed in attitudes towards science. Contrary to a common worry about activity-based programs, content achievement was not negatively affected.

A second meta-analysis conducted in 1986 largely confirmed these results, emphasizing in particular the differences in attitude toward science and process skills (17 and 19 percentile point gains, respectively) among students who were taught with activity-based modules.

B - 22 World Watcher/Learning about the Environment Curriculum (LATE)

The World Watcher/Learning about the Environment Curriculum (LATE) is a yearlong, inquiry-based, technology-supported environmental science curriculum for high school developed at Northwestern University. It is based on the Learning-for-Use model of learning that conceptualizes content and process learning as complementary and mutually facilitating, rather than at odds. LATE incorporates the use of scientific visualizations and is centered on three key issues: population growth and resource availability, electricity generation and energy demand, and managing water resources for agriculture and human consumption.

Contact: Daniel C. Edelson Northwestern University Learning Sciences Program Annenberg Hall 332 Evanston, Ill 60208-2610 E-mail: [email protected] Web site: http://www.worldwatcher.northwestern.edu/curriculum.htm

Results:

Type of measure Number of Results Effect size studies Self-designed test 1 Gains at all grade levels and all Large (urban), populations (urban, suburban). moderate Biggest differences in tenth grade. (suburban)

Urban students showed higher gains (5 points) than suburban students (3.5 points). Researchers caution not to interpret this result as greater effectiveness of the curriculum for the urban group. It is possible that a subgroup of a sample that scores below average on a test tends to do better on retests.

B - 23 Appendix C: Matrix Summarizing the Key Elements of Professional Development Studies Found

The matrix that appears on the following page provides the key characteristics of the professional development (PD) studies that met our criteria. These characteristics include subject matter, grade span, source of participants, form/distribution of in-service time, total in-service contact hours, study duration, and effect sizes/results. The studies have also been organized according to categories developed by Kennedy (1998) to classify types of professional development models according to the content that they provide to teachers:

Group 1: PD models that specify a set of teaching behaviors that apply generically to all school subjects.

Group 2: PD models that prescribe a set of teaching behaviors that, while seemingly generic, are presented as applying to one particular school subject, such as mathematics or science.

Group 3: PD models that provide general guidance on both the curriculum and the pedagogy for teaching a particular subject, justifying the recommended practices with references to knowledge about how students learn a particular subject.

Group 4: PD models that provide knowledge about how students learn a particular subject matter but that do not specify the practices that should be used to teach that particular subject.

In the Kennedy study, as the models moved from focusing on changing teacher behavior to changing teacher knowledge and from more prescriptive to less, their effectiveness in terms of improving student achievement increased as well.

Key Elements of Professional Development Studies Citation Subject matter Grade span of Source of participants Form and distribution of Total in-service Study duration Effect size/results context participating in-service time contact hours 1 in months 1 students Group 1: Focus on Teaching Behaviors Applying Generically to All School Subjects

Stallings & Krasavage* Math 2-4 Schoolwide projects Distributed workshops 16 Basic skills: small-moderate (1986) Stevens & Slavin* (1995) Math K-6 Schoolwide projects Distributed workshops 8 Basic skills and problem solving: small

Group 2: Focus on Teaching Behaviors Applying to a Particular Subject

Good, Grouws, & Math 4-12 Individual volunteers 2 @ 1.5 3 4 Basic skills and problem solving: Ebmeier* (1983) small Good & Grouws* (1979) Math 4 Individual volunteers 2 @ 1.5 3 4

Mason & Good* (1993) Math 4-6 Individual volunteers 3 @ 1.5 4.5 5 Basic skills and problem solving: small-moderate Villasenor & Kepner Math 1 Individual volunteers Summer workshop + 3 @ 25 6 Basic skills and problem solving: large (1993) 2 duirng year Lawrenz & McCreath* Science 1-8 Individual volunteers University course (15 @ 45 8 Problem solving: moderate-large (1998) 3) Marek & Methven* Science K-5 Individual volunteers 4-week Summer Institute 100 8 Problem solving: moderate-large (1991) Otto & Schuck* (1983) Science 8 Individual volunteers 5 @ variable 16 2.5 Problem solving: large

Radford (1998) Science 4-10 Individual volunteers 7-weeks of summer 8 Problem solving: small-moderate. training, 5 day-long Science attitude: moderate. workshops during year Rubin & Norman* (1992) Science 6-9 Individual volunteers University course (10 @ 30 3 Problem solving: large 3)

Group 3: Focus on Curriculum or Pedagogy Justified by How Students Learn

Cobb et al.* (1991) Math 2 Individual volunteers 1-week Summer Institute 150 8 Basic skills: small. Problem solving: + Distributed moderate-large

Hestenes (2000) Science (physics) High school Individual volunteers 2 years of 4-week 16 Significant gains in teacher Summer Institute knowledge, significant gains in student post-test performance. No effect size reported.

Rivet & Krajcik (2004) Science 6 Individual volunteers 1-week Summer Institute varied by year of 16-64 Basic skills & problem solving: + Distributed implementation moderate-large Satchwell & Loepp (2002) Integrated math, 6-8 Schoolwide pilot testing 1-week Summer 50+ 16 Significant gains over control group in science, and inservice + 4 on-site both math & science basic skills & technology consults problem solving. No effect sizes reported.

Smith et al. (1993) Science 7 Individual volunteers 2 @ 4 hours 8 4 1/2 Significant difference for students of teachers with both PD & Curriculum treatments and Curriculum-only treatments. PD-only treatments not as effective. No effect sizes reported.

Wood & Sellers* (1996) Math 2-3 Individual volunteers 1-week Summer Institute 150 16 Basic skills: small-moderate. Problem + Distributed solving: moderate-large

Group 4: Focus on How Students Learn and How to Assess Student Learning

Carpenter et al.* (1989) Math 1 Individual volunteers 4-week Summer Institute 80 Basic skills & problem solving: moderate to large Note: Format and categories of this table and summaries of articles marked with an asterisk were taken from Kennedy (1998).

1. Some of these estimates of contact time were estimated from general descriptions of programs. For estimates of program durations, a school year was assumed to be roughly 8 months, a semester 4 1/2 months, and two school years 16 months.

C - 1 Appendix D: Matrix of All Curricula Identified by Grade Level, Student Achievement Studies, and Inclusion in Whole School Reform

Mathematics Curricula

Student Achievement Whole School Targeted Grade Studies with Reform with # Mathematics Curricula Levels Comparisons Curriculum 1 A+ny where Learning System Middle School no n/a

2 Academic Systems Interactive Math Curriculum Middle School no n/a

3 Accelerated Schools K–12 n/a no

4 Advanced Placement (AP) Calculus 11–12 yes n/a yes/no (schools can 5 America's Choice K–12 n/a opt for another curriculum) 6 Applications and Connection 6–8 no n/a

7 ATLAS Communities Pre-K–12 n/a no

8 Bob Jones University Math Series Middle School no n/a

9 Center for Effective Schools K–12 n/a no

10 Coalition of Essential Schools K–12 n/a no

11 Cognitive Tutor Middle School yes n/a

12 College Board Pacesetter 9–12 no n/a

13 College Preparatory Mathematics (CPM) 9–12 yes n/a

14 Community for Learning K–12 n/a no

15 Compass Learning Middle School no n/a

16 Co-nect K–12 n/a no

17 Connected Mathematics (CMP) Middle School yes n/a

18 Core Knowledge K–8 no yes Contemporary Mathematics in Context: A 19 Unified Approach (Core-Plus Mathematics 9–12 yes n/a Project CPMP) 20 Destination Math Middle School no n/a

21 Different Ways of Knowing Pre-K–8 no yes

D - 1 Student Achievement Whole School Targeted Grade Studies with Reform with # Mathematics Curricula Levels Comparisons Curriculum 22 Direct Instruction K–8 yes yes

23 Dolciani (textbook) Middle School no n/a

24 Edison Schools K–12 yes yes

25 Expeditionary Learning Outward Bound (ELOB) K–12 no no

26 First Things First K–12 n/a no

Glencoe Main Middleschool Textbook–Math 27 Middle School no n/a Applications and Connections

28 Hands-on Equations Middle School no n/a

29 Harcourt Math Middle School no n/a

30 Heath Mathematics Connections Middle School no n/a

31 Heath Passport Middle School no n/a

32 High Schools that Work (HSTW) 9–12 no yes

33 Holt Middle School Math Middle School no n/a no (only 34 I Can Learn (ICL) Education Systems Middle School unpublished n/a manuscript) Integrated Mathematics, Science, and 35 Middle School yes n/a Technology (IMaST)

36 Integrated Thematic Instruction K–12 n/a no

37 Interactive Mathematics Program (IMP) 9–12 yes n/a

38 International Baccalaureate Program K–12 no yes

39 Key Math Teach and Practice Middle School no n/a

40 Lamar CISO Algebra Program Middle School no n/a

41 Larson Series 9–12 no n/a

42 Larson Series Middle School no n/a

Learning by Design: Integrating and Enhancing 43 the Middle School Math, Science and Middle School no n/a Technology Curricula

D - 2 Student Achievement Whole School Targeted Grade Studies with Reform with # Mathematics Curricula Levels Comparisons Curriculum 44 Learning Network (The) K–8 n/a no

45 Lightspan Math Program Middle School no n/a

46 Math Advantage Middle School no n/a

47 MATH Connections 9–12 yes n/a

48 Math Passport Series Middle School no n/a

49 Mathematics: Applications and Connections Middle School no n/a

50 Mathematics in Context (MiC) Middle School yes n/a

Mathematics: Modeling Our World 51 9–12 yes n/a (MMOW/ARISE)

52 Mathematics Plus Middle School no n/a

53 Mathematics with Meaning 7–12 yes n/a

54 Mathscape Middle School no n/a

55 MathScape: Seeing and Thinking Mathematics Middle School no n/a

56 MicroSociety K–8 no no

57 MATHThematics 6–8 yes n/a

58 Middle School Math Middle School no n/a

59 Middle Start 5–9 no yes

Middle-school Mathematics through 60 Middle School no n/a Applications Project (MMAP)

61 Modern Red Schoolhouse K–12 n/a no

62 More Effective Schools K–12 n/a no

63 Onward to Excellence K–12 n/a no

64 Partnership for Access to Higher Mathematics Middle School no n/a

65 Prentice Hall: Tools for Success Middle School yes n/a

D - 3 Student Achievement Whole School Targeted Grade Studies with Reform with # Mathematics Curricula Levels Comparisons Curriculum 66 Project lead the way Middle School no n/a

67 Project lead the way 9–12 no n/a

Purdue Problem Centered Mathematics 68 Middle School no n/a Curriculum

69 Quantum Learning K–12 n/a no

70 QuESt K–12 n/a no

71 Real Math basal mathematics program Middle School no n/a

72 Saxon Math: An Incremental Development Middle School yes n/a

73 School Development Program K–12 n/a no

74 School Renaissance K–12 no no Scott Foresman's Math Diagnostic & 75 Middle School no n/a Interventions System SimCalc: Cognitive Foundations for a 76 Middle School no n/a Multiplicative Structures Curriculum

77 Singapore Mathematics Middle School no n/a

78 Strength in Numbers Math Program Middle School no n/a

79 Successmaker Middle School no n/a

Summer Success: Math Afterschool Achievers. 80 Middle School no n/a Math Club Everyday Counts Series Systemic Initiative for Montana Mathematics 81 9–12 yes n/a and Science (SIMMS) Talent Development High School with Career 82 9–12 no no Academies

83 Talent Development Middle School 4–9 no no no (only 84 The Expert Mathematician Middle School unpublished n/a manuscript) 85 Transition Mathematics Middle School no n/a

86 Turning Points 6–8 n/a no

University of Chicago School Mathematics 87 7–12 yes n/a Project (UCSMP)

D - 4 Student Achievement Whole School Targeted Grade Studies with Reform with # Mathematics Curricula Levels Comparisons Curriculum 88 Urban Learning Centers Pre-K–12 n/a no

89 Ventures Initiative and Focus System K–12 no no

D - 5 Science Curricula

Student Achievement Whole School Targeted Grade Studies with Reform with # Science Curricula Levels Comparisons1 Curriculum 1 Active Physics High School NE/NR no yes (America's 2 America's Choice K–12 NC Choice) An Integrated Biology/Biotechnology High School 3 9–12 NE/NR no Curriculum

4 ARIES: Astronomy-Based Physical Science 3–8 NE/NR no

5 TERC Astrobiology High School NC no

6 Astronomy Village Middle School NE/NR no

7 Biology Middle School NE/NR no

8 Biology: A Community Context 9–12 NE/NR no

9 Biology: Principles and Explorations High School NE/NR no

10 Biology: The Dynamics of Life High School NE/NR no

11 Biology: Visualizing Life High School NE/NR no

12 BSCS Biology: A Human Approach High School NE/NR no

13 BSCS Biology: A Molecular Approach 9–11 NE/NR no

14 BSCS Biology: An Ecological Approach High School NE/NR no

15 BSCS: An Inquiry Approach 9–11 yes no

16 BSCS: Middle School Science and Technology Middle School NE/NR no

17 CHEM 2: Chemicals, Health, Environment, and Me Elementary School NE/NR no

18 ChemCom: Chemistry in the Community 9–12 NE/NR no

19 ChemDiscovery/ChemQuest High School NC no

20 Constructing Ideas in Physical Science (CIPS) 7–8 yes no

1 Under Student Achievement Studies with Comparisons, yes signifies the curriculum had a study that met our criteria, NC signifies that the study did not meet our criteria for inclusion, and NE/NR signifies that no evaluation was found, no evaluation exists, or we were not able to obtain a copy of an evaluation.

D - 6 Student Achievement Whole School Targeted Grade Studies with Reform with # Science Curricula Levels Comparisons Curriculum yes (Core 21 Core Knowledge K–12 NC Knowledge)

22 Design It! Engineering in Afterschool Programs Elementary School NE/NR no

23 DESIGNS/DESIGNS II 6–8 yes no

Developmental Approaches in Science, Health and Elementary and 24 NC no Technology (DASH) Middle School yes (Different 25 Different Ways of Knowing K–12 NC Ways of Knowing) Elementary and yes (Direct 26 Direct Instruction NC Middle School Instruction)

27 Earth Sciences and Community High School NE/NR no

28 Elementary Science Study Elementary School NC no

29 Event-Based Science (EBS) 6–9 yes no yes (Expeditionary 30 Expeditionary Learning Outward Bound (ELOB) K–12 yes Learning Outward Bound) 31 Exploring Earth High School NC no

32 Exploring Life High School NE/NR no

33 Foundation Science Middle School NE/NR no

34 Foundational Approaches in Science Teaching (FAST) 7–9 yes no

Foundations and Challenges to Encourage Technology- 35 Middle School NE/NR no Based Science (FACETS)

36 Full Option Science System for Middle School (FOSS) K–8 yes no

37 GeoKits 5–9 NE/NR no

38 Glencoe Life Science Middle School NE/NR no

39 Glencoe Physical Science High School NE/NR no

40 Global Lab Curriculum (GLC) 7–9 yes no

41 Great Explorations in Mathematics and Science (GEMS) Pre-K–8 yes no

D - 7 Student Achievement Whole School Targeted Grade Studies with Reform with # Science Curricula Levels Comparisons Curriculum 42 Hands-on Physics High School NE/NR no

43 Health Biology High School NE/NR no

yes (High Schools 44 High Schools that Work (HSTW) 9–12 yes that Work) Insights in Biology: An Introductory High School Science 45 9–12 NE/NR no Program Integrated Mathematics, Science, and Technology 46 6–8 yes no Curriculum (IMaST)

47 Integrated Science 6–8 NE/NR no

48 Investigating Earth Systems (IES) 5–8 NE/NR no

Issues, Evidence, and You (IEY)/The Science Education for 49 7–9 yes no Public Understanding Program (SEPUP)

50 Learning by Design (LBD) 6–8 yes no

Center for Learning Technologies in Urban Schools 51 6–8 yes no (LeTUS)

52 Living by Chemistry High School NE/NR no

53 Macmillan/McGraw-Hill Science Elementary School NE/NR no

54 Matter and Molecules Middle School NE/NR no

55 Minds-On Physics High School NC no

56 Modeling Instruction in High School Physics 9–12 yes no

57 Models in Technology and Science (MITS) 5–8 NE/NR no

58 Modern Biology High School NE/NR no

59 National Science Curriculum for High Ability Learners 2–8 yes no

Physics Resources and Instructional Strategies for 60 10–12 yes no Motivating Students (PRISMS)

61 Prentice Hall Exploring Life Science Elementary School NE/NR no

62 Prentice Hall Exploring Physical Science Middle School NE/NR no

Middle and High 63 Prime Science NE/NR no School

D - 8 Student Achievement Whole School Targeted Grade Studies with Reform with # Science Curricula Levels Comparisons Curriculum 64 Project Physics High School NC no

65 Science 2000/Science 2000+ 6–8 yes no

66 Science and Life Issues/SEPUP Middle School NE/NR no

67 Science and Sustainability/SEPUP Middle School NE/NR no

Science and Technology Concepts for Middle Schools 68 6–8 yes no (STC/MS)

69 Science Curriculum Improvement Study (SCIS) K–6 yes no

70 Science in a Technical World High School NC no

Science Insights (Exploring Living Things, Exploring Matter 71 Middle School NE/NR no and Energy)

72 Science Interactions Middle School NE/NR no

73 Science that Counts in the Workplace High School NE/NR no

74 Science: A Process Approach Elementary School NC no

75 SciencePlus: Science and Technology Middle School NE/NR no

yes (Talent 76 Talent Development Middle School Middle School NE/NR Development Middle School)

77 The Changing Global Environment High School NE/NR no

78 Ventures Initiative and Focus System K–12 NE/NR no

79 Voyages through Time High School NE/NR no

World Watcher/Learning about the Environment Curriculum 80 9–12 yes no (LATE)

D - 9