comm-calif2.qxp 6/12/97 4:27 PM Page 817 The Math Wars California Battles It Out over Mathematics Education Reform (Part II) Note: The first part of this article appeared in the June/July issue of the Notices, pages 695–702. The Numbers Battle examinations but also group assignments, long- “Show us the data, show us that this program term projects, and portfolios of work done over works,” demands Michael McKeown, the head of a period of time. Some also favor the idea of judg- Mathematically Correct, the San Diego-based ing the efficacy of curricular programs by criteria group critical of mathematics education reform. other than standardized tests, such as the num- McKeown makes his living in the data-driven ber and type of mathematics courses students field of biology, as a researcher at the Salk In- subsequently take. stitute and an adjunct faculty member at UC Such are the views in that portion of the math San Diego. One of the things he finds most vex- wars devoted to testing. The scramble to find ing about the reform is that it has been imple- numbers to support certain viewpoints has led mented without having been subject to large- to some questionable uses of data. For example, scale, well-designed studies to show that student the anti-reform group HOLD (Honest Open Log- achievement rises when the reform materials ical Debate on mathematics reform) recently cir- are used. He points out that what convinced culated an e-mail message containing data on the everyone that there is a problem in mathemat- Elementary Level Mathematics Examination, ics education was low scores on tests like the Na- which is administered in the California State tional Assessment of Educational Progress. To University system to place students in appro- show that the reform materials are an improve- priate mathematics courses. The data were stun- ment, he asserts, one had better show that stu- ning: in 1989, 28.2% of students failed; in 1992, dent performance on those tests improves with 45.0% failed; and in 1994, 54.0% failed. The e-mail the new materials. message blamed mathematics education reform For their part most reformers acknowledge in California for this poor showing. What the the importance of showing that their students message neglected to say was that the exami- can perform well by traditional measures. How- nation was changed in 1992 to include not just ever, they are wary of overemphasis on perfor- two but three years of high school mathemat- mance on standardized tests because, they say, ics. According to Scott Farrand of California they are aiming for a far deeper understanding State University, Sacramento, who serves on a of mathematics than can be gauged by such Cal State committee working on remediation, tests. Indeed, some argue that it was standard- the change in the test was the “primary cause” ized testing that drove the traditional curricu- of the rise in failures. There are other causes, lum to sacrifice understanding in favor of the but he believes it is simplistic to blame reform. narrow skills of arithmetic computation and As Farrand puts it, “The greater issue is that symbol manipulation. In this spirit, most reform the high school curriculum serves too many programs use a variety of methods of assessing masters, aims to cover too much, none of it with student achievement, including not only timed enough depth.” AUGUST 1997 NOTICES OF THE AMS 817 comm-calif2.qxp 6/12/97 4:27 PM Page 818 This example shows how seemingly clear-cut Mathematics Program (IMP), the data that exist statistics can mask a much more complicated pic- may not be sufficiently conclusive to change the ture. Indeed, one of the reasons there is a dearth minds of the critics of these programs. When of data on the reform is that it is very difficult MathLand was put on the textbook adoption list to design studies of educational programs well in California, there were no reliable data show- enough to produce meaningful statistics. Asked ing how it influenced student achievement. Ac- what kind of evidence would satisfy him that the cording to MathLand’s national mathematics reform was working, McKeown reels off a list of consultant, Dennis Estrada, for three years teach- variables that would need to be controlled in a ers extensively field-tested the program with well-designed study. For example, one must con- students and helped to shape and fine-tune all trol for the fact that experimental programs are of the lessons. Estrada explains that in classroom sometimes tested only on students who are al- field testing, authors and editorial team mem- ready doing well rather than on students who are bers work closely with teachers and students. still struggling with the old program. Pretests and Achievement data collected in such an environ- posttests have to take into account the prior ment would not be reliable, he says. achievement of the students. Often it is the bet- Now that MathLand has been out for a cou- ter and more enthusiastic teachers who volun- ple of years, a study has been done using stan- teer to test new programs, and they may also dardized test scores for 30,000 students in make greater efforts if they know that their per- grades one through six in eight school districts formance is being monitored. The sample of in California.2 The districts supplied scores for students and teachers must be large enough to 1995—the year before MathLand was imple- test whether the program can be used in differ- mented in those districts—and for 1996. The ent settings, and a variety of programs should study is based on “normal curve equivalent” be included so that results can be compared. And (NCE) scores, which are a standard way of pre- then there is the “Hawthorne effect”, which sug- senting educational data. The actual NCE scores gests that a change in any variable will cause a are not presented. Instead, the study calculates temporary rise in achievement. the differences in NCEs between the two testing McKeown has one example of an educational points, noting that if the NCE is the same from study that aimed for scientific rigor: Project Fol- one year to the next, then “one year’s growth” low-Through. Sponsored by the U.S. Department has occurred. The study formed 23 cohort of Education, Project Follow-Through was begun groups of students from the same district and in 1967 as part of Lyndon Johnson’s “War on at the same grade level, representing all 30,000 Poverty” and received funding through 1995, students. On tests of applications of math- for a total of about $1 billion. During its main ematics, 13 groups achieved the normal one phase from 1968 to 1976, 10,000 children in 120 year’s growth, 5 were above normal, and 5 were communities were studied. The main objective below normal. On tests of mathematical com- of Project Follow-Through was to come up with putation, data were available for only 8 cohort effective ways to teach disadvantaged children. groups, and all of these achieved the normal Twelve different approaches were included in the one year’s worth of progress. The reason that so study, and many of them utilized ideas in use few cohort groups had computation scores was in reform classrooms today, such as construc- that half of the districts in the study do not test tivism and group learning. One of the approaches for computation. This is exactly the area in which was called “direct instruction” and was based on MathLand has been most strenuously attacked, the ideas of Siegfried Englemann, who devel- so these data are unlikely to quell MathLand’s oped finely honed materials to be used by teach- critics. ers in more or less the traditional mode of in- MathLand has had its share of controversies struction. Mathematics test scores pointed to over test scores. After the program was intro- “direct instruction” as being far and away the duced in San Francisco in the fall of 1995, Math- most effective of the approaches in the study. Land took some heat when the school district’s Before the evaluation of Project Follow-Through mathematics test results showed a decline the was released, questions were raised about the following spring. An article in the San Francisco methodology and some of the findings, and the Examiner reported that schools superintendent project’s results were buried under the ensuing controversy. Thus the largest educational study 1A group at the University of Oregon has assembled a ever undertaken in the U.S. was never used to great deal of information on Project Follow-Through at shape educational policy.1 the Web site http://darkwing.uoregon.edu/ Needless to say, none of the reform programs ~adiep/ft/151toc.htm. has been part of a study as extensive as Project 2A press release about the study is available on the Math- Follow-Through. For the two programs discussed Land Web site at http://192.216.191.114/ in this article, MathLand and the Interactive assessinmath.html. 818 NOTICES OF THE AMS VOLUME 44, NUMBER 7 comm-calif2.qxp 6/12/97 4:27 PM Page 819 Bill Rojas believed the drop in scores was due One aspect of the WCER evaluation that has to MathLand. “There’s math and math substi- left some leery of its conclusions is the fact that tutes,” he was quoted as saying. “Let’s get real all results must be cleared by the directors of math in there.” In fact, it is unlikely that Math- IMP before they are released.