Rohrer, D., & Taylor, K. (2006). The effects of overlearning and distributed practice on the retention of mathematics knowledge. Applied Cognitive Psychology, 20, 1209–1224.

The Effects of Overlearning and Distributed Practice on the Retention of Mathematics Knowledge

Doug Rohrer and Kelli Taylor University of South Florida

In two experiments, 216 college students learned a mathematical procedure and returned for a test either one or four weeks later. In Experiment 1, performance on the four-week test was virtually doubled when students distributed 10 practice problems across two sessions instead of massing the same 10 problems in one session. This finding suggests that the benefits of distributed practice extend to abstract mathematics problems and not just rote memory cognitive tasks. In Experiment 2, students solved 3 or 9 practice problems in a single session, but this manipulation had no effect on either the one-week or four-week test. This result is at odds with the virtually unchallenged support for the strategy of continuing practice beyond the point of mastery in order to boost long- term retention. The results of both experiments suggest that the organization of practice problems in most mathematics textbooks is one that minimizes long-term retention.

Perhaps no mental ability is more a student first masters a skill and then important than our capacity to learn, but the immediately continues to practice the same skill. benefits of learning are lost once the material is For example, after a student has studied a set of forgotten. Such forgetting is particularly 20 vocabulary words until each definition has common for knowledge acquired in school, and been correctly recalled once, any immediate much of this material is lost within days or further study of these definitions entails an weeks of learning. Thus, the identification of overlearning strategy. Overlearning is learning strategies that extend retention would particularly common in mathematics education prove beneficial to students and any others who because many mathematics assignments include wish to retain information for meaningfully long a dozen or more problems of the same type. A periods of time. Toward this aim, the two distributed or spaced practice strategy requires experiments presented here examined how the that a given amount of practice be divided across retention of a moderately abstract mathematics multiple sessions and not massed into just one procedure was affected by variations in either the session. For example, once a mathematics total amount of practice or the scheduling of this procedure has been taught to students, the practice. corresponding practice problems can be massed Specifically, the two learning strategies into one assignment or distributed across two or assessed here are known as overlearning and more assignments. As detailed in the general distributed practice. By an overlearning strategy, discussion, the practice problems in most mathematics textbooks are arranged so that Address correspondence to Doug Rohrer, Department of Psychology, students rely on overlearning and massed PCD4118G, University of South Florida, Tampa, FL 33620, USA. E- mail: [email protected]. We thank Christina Coghlan, Tammy practice. Logan, Kristina Martinez, Erica Porch, and Courtnay Rosene for The strategies of distributed practice and their assistance with data collection. This research was supported by two grants from the Institute of Education Sciences, U.S. Department overlearning are not complementary, and the two of Education, to Hal Pashler and Doug Rohrer (R305H020061, strategies cannot be compared directly. Instead, R305H040108). The opinions expressed are those of the authors and do not necessarily represent the views of the Institute of Education distributed practice is the complement of massed Sciences. practice, and the comparison of these two Mathematics Learning 2 strategies requires that the total amount of instance, in the Driskell et al. meta-analysis practice be held constant. For example, one described above, only 7 of the 51 comparisons group of students might divide 10 problems relied on a retention interval of more than one across two sessions while another group solves week, and the longest was 28 days. Moreover, all 10 problems in the same session (as in the largest effect sizes were observed for Experiment 1). By contrast, assessing the retention intervals lasting less than one hour, and benefits of overlearning requires a manipulation Driskell et al. astutely observed that benefits of of the total amount of practice given within a overlearning declined sharply with retention single session. Thus, one group might be interval. assigned three problems while another is The possibility that the benefits of assigned nine problems (as in Experiment 2). overlearning may dissipate with time is also Thus, because overlearning and distributed supported by several overlearning experiments practice are orthogonal and not complementary, including an explicit manipulation of retention it is logically possible that neither, both, or just interval. For example, in Experiment 1 of one of these strategies could benefit long-term Reynolds and Glaser (1964), some students retention. Naturally, both strategies have been studied biology three times as much as others, the focus of numerous previous studies, but the and the high studiers recalled 100% more than following review of the research literature the low studiers after 2 days but just 7% more reveals caveats, gaps, and inconsistencies with after 19 days. A similar decline in the test score regard to the benefits of each strategy for benefits of overlearning was observed in two conceptual mathematics tasks. recent experiments reported by Rohrer, Taylor, Overlearning Pashler, Wixted, and Cepeda (2005). An overlearning experiment requires a Finally, it appears that the benefits of manipulation of the total amount of practice overlearning are especially unclear in within a single session, so that one condition mathematics learning because, to our knowledge, includes more practice than another. Numerous every previously published overlearning experiments have shown that this increase in experiment relied solely on verbal memory tasks. practice can raise subsequent test performance Moreover, virtually all of these tasks required (e.g., Bromage & Mayer, 1986; Earhard, Fried, only rote memory. Thus, the results of these & Carlson, 1972; Gilbert, 1957; Kratochwill, experiments may not generalize to abstract Demuth, & Conzemius, 1977; Krueger, 1929; mathematical tasks. This gap in the literature is Postman, 1962; Rose, 1992). This benefit of surprising in light of the heavy reliance on overlearning is also supported by a meta-analysis overlearning by students in mathematics courses, by Driskell, Willis, and Cooper (1992), who as further detailed in the general discussion. In examined 51 comparisons of overlearning versus summary, because of the uncertainty surrounding learning-to-criterion in experiments using the long-term benefits of overlearning and the cognitive tasks and found a moderately large apparent absence of overlearning experiments effect of overlearning on a subsequent test (d = using mathematics tasks, it is unclear whether .75). By these data, it is not surprising that overlearning is efficient or even effective when overlearning is a widely advocated learning long-term retention is the aim. strategy (e.g., Fitts, 1965; Foriska, 1993; Hall, Distributed Practice 1989; Jahnke & Nowaczyk, 1998). When practice is distributed or spaced, a Yet a closer review of the empirical given amount of practice is divided across literature reveals that the benefits of overlearning multiple sessions and not massed into one on subsequent retention may not be long lasting. session. The duration of time between learning This is because most previous overlearning sessions is the inter-session interval (ISI). For experiments have employed a relatively brief example, if 10 math problems are divided across retention interval (RI), which is the duration two sessions separated by one week, the ISI between the learning session and the test. For equals one week. By contrast, massed practice Mathematics Learning 3 entails an ISI of zero. When practice is Modigliani (1985) observed a spacing effect with distributed, the retention interval refers to the young children who were asked to memorize five duration between the test and the most recent multiplication facts (e.g., 8 x 5 = 40). While such learning session. For example, if a concept is facts are certainly useful, the present study studied on Monday and Thursday and tested on focuses on mathematical tasks that require more Friday, the RI equals one day. (Incidentally, than rote memory. while the benefits of distributing practice across A second reason to question the benefits sessions is the focus of the present paper, one of distributed practice for non-rote mathematics can also distribute practice within a session when learning is given by the design of three previous multiple presentations are separated by unrelated mathematics learning experiments that are often tasks, e.g., Greene, 1989; Toppino, 1991). cited as evidence of a spacing effect with a non- Distributed practice often yields greater rote mathematics task. In these experiments, the test scores than massed practice, and this finding retention interval was shorter for Spacers than is known as the spacing effect (e.g., Baddeley & for Massers, and this confound undoubtedly Longman, 1978; Cull, 2000; Bjork, 1979, 1988; benefited the Spacers. For instance, in Grote Bloom & Shuell, 1981; Carpenter & DeLosh, (1995), the Massers learned only on Day 1 while 2005; Dempster, 1989; Fishman, Keller, & the Spacers’ learning continued from Day 1 Atkinson, 1968; Seabrook, Brown, & Solity, through Day 22. Yet every student was tested on 2005). At very brief retention intervals, however, Day 36, which produced a 35-day RI for the spaced practice may be no better or even worse Massers and a 14-day RI for the Spacers. This than massed practice (e.g., Bloom & Shuell, undoubtedly benefited the spacers. The same 1981; Glenberg & Lehman, 1980; Krug, Davis, confound between ISI and RI occurred in the two & Glover, 1990). By this result, students are mathematics learning experiments reported by behaving optimally when they mass their Gay (1973). We are not aware of any previously learning into one session just prior to an exam if published, non-confounded experiment that they do not need the information after the exam. examines how the distribution of practice affects At longer retention intervals, though, the benefits the retention of a non-rote mathematics task. of spacing are often sizeable, and many There are, however, non-experimental researchers have therefore argued that students studies that have found benefits of distributed should rely more heavily on distributed practice practice of mathematics. Most notably, perhaps, in order to extend retention (Bahrick, Bahrick, Bahrick and Hall (1991) assessed the retention of Bahrick, & Bahrick, 1993; Bjork, 1979; Bloom algebra and geometry by people who had taken & Shuell, 1981; Dempster, 1989; Schmidt & these courses between 1 and 50 years earlier. A Bjork, 1992; Seabrook, Brown, & Solity, 2005). regression analysis showed that retention was Even at longer retention intervals, positively predicted by the number of courses though, the benefits of spacing are more requiring the same material. For example, much equivocal for tasks that require abstract thinking of the material learned in an algebra course rather than only rote memory. In a recent meta- reappears in an advanced algebra course, and the analysis of 112 comparisons of distributed and completion of both courses therefore provides massed practice, Donovan and Radosevich distributed practice of this overlapping material. (1999) found that the size of the spacing effect Thus, the results suggest that distributed practice declined sharply as task complexity increased may be beneficial in mathematics learning. This from low (e.g., rotary pursuit) to average (e.g., possibility is assessed here with a controlled word list recall) to high (e.g., puzzle). By this experiment, albeit with retention intervals that finding, the benefits of spaced practice may be are measured in weeks rather than years. muted for mathematical tasks that require more Task than rote memory. As for mathematical tasks that In the experiments reported here, college do rely solely on rote memory, a spacing effect students learned to calculate the number of has been demonstrated. For example, Rea and unique orderings (or permutations) of a letter Mathematics Learning 4 sequence with at least one repeated letter. For Experiment 1 example, the sequence abbbcc has 60 The first experiment examined the benefit permutations, including abbcbc, abcbcb, bbacbc, of distributing a given number of practice and so forth. The solution is given by a formula problems across two sessions rather than that is presented and illustrated in the Appendix. massing the same practice problems into one While the number of permutations can also be session. As shown in Figure 1A, the Spacers obtained by listing each possible permutation, attempted 5 problems in each of two sessions this alternative approach fails when the number separated by one week, whereas the Massers of permutations is large and the problem must be attempted the same 10 problems in session two. completed in a relatively brief amount of time Each group received a tutorial immediately (as in the present study). No student saw a given before their first practice problem, and the letter sequence more than once. A total of 17 students were tested one or four weeks after their different sequences were used in the two final practice problem. experiments, and these are listed in the Method Appendix. Participants. All three sessions were Base Rate Survey completed by 116 undergraduates at the We assessed the base rate knowledge of University of South Florida. The sample this task for our experiment participants by included 95 women and 21 men. An additional giving a brief test to a sample of students drawn 39 students completed the first session but failed from the participant pool used in Experiments 1 to show for either the second or third session. and 2. We expected that few if any of the None of the students participated in Experiment students would be able to perform the task 2. because we have never encountered this Design. There were two between-subjects particular kind of permutation problem in an variables: Strategy (Space or Mass) and undergraduate level textbook. Retention Interval (1 or 4 weeks). Thus, each Method student was randomly assigned to one of four Participants. The sample included 50 groups: Spacers with 1-week RI, Spacers with 4- undergraduates at the University of South week RI, Massers with 1-week RI, and Massers Florida. These included 43 women and 7 men. with 4-week RI. None participated in Experiments 1 or 2. Procedure. The students attended three Procedure. After a brief demographic sessions. At the beginning of the first session, survey, students were given three minutes to each student was randomly assigned to one of solve the following three problems: aabbbbb, the four conditions listed above. Students were aaabbbb, and abccccc. These sequences yield not told what tasks awaited them in future answers of 21, 35, and 42, respectively. sessions. It is not known whether some students Results and Discussion practiced the procedure outside of the Not surprisingly, none of the students experimental sessions, although there was no correctly answered any of the three problems. extrinsic reward for test performance. If any self Many of them attempted to solve the problems review did occur, we know of no reason why its by listing every permutation, but none exhibited prevalence would vary among Spacers and knowledge of the necessary formula. Thus, this Massers. mathematics procedure appears to be unknown All students completed a tutorial, two to the participant pool used in Experiments 1 and practice sets, and a test. Students read the tutorial 2. Furthermore, even if some participants in immediately before their first practice set. The these experiments did possess any relevant tutorial included two pages of instruction (3 min) knowledge before the experiment, the use of and written solutions to Problem 11 (3 min) and random assignment ensured that their presence Problem 12 (2 min) of the Appendix. The first would not be a confounding variable. practice set included Problems 1 – 5 of the Appendix, in that order, and the second practice Mathematics Learning 5 set included Problems 6 – 10 of the Appendix, in (SE = 1.6%) while the Spacers averaged only 2 that order. Each problem appeared in a booklet 85% (SE = 3.2%), F (1, 108) = 6.79, p < .05, ηp with only one problem per page. Students were = .06. This difference was due to very poor allotted 45 s to solve each problem. Immediately performance by a subset of Spacers who after each attempt, students were shown the apparently forgot the procedure during the one solution for 15 s before immediately beginning week inter-session interval. Thus, the one-week the next problem. ISI introduced a disadvantage for the Spacers, The first two sessions were separated by but this worked against the spacing effect rather one week. The Spacers completed the first five- than for it. problem practice set in session one and the Test. The mean percentage accuracy for second five-problem practice set in session two. the five test problems is shown in Figure 1B. As The Massers completed both practice sets in illustrated, the Spacers and Massers were not session two, without any delay between the two reliably different at the 1-week RI, but the sets. Spacers sharply outscored the Massers at the 4- One or four weeks after the second week RI. This parity at one week caused the session, every student returned for a test. The test main effect of Strategy (Space vs. Mass) to fall included the five test problems listed in the short of statistical significance, F (1, 106) = 3.67, 2 Appendix in the order shown. The five test p = .06, ηp = .03. Not surprisingly, the main problems were presented simultaneously, and effect of RI was reliable, F (1, 106) = 12.92, p < 2 students were allotted five minutes to solve all .001, ηp = .11. The reliance of the spacing effect five problems. No feedback was given during the on retention interval was evidenced by an test. interaction between ISI and RI, F (1, 106) = Results 2 7.21, p < .01, ηp = .06. This pattern was further Learning. The tutorial was sufficiently confirmed by Tukey tests showing that the effective, as evidenced by performance on the difference between Spacers and Massers was five problems given to every student reliable at the 4-week RI (p < .05) but not the 1- immediately after the tutorial. Of the 116 week RI. These post hoc tests also showed that students, 65 scored a perfect five, 27 scored four, the difference between the one- and four-week 12 scored three, 6 scored two, 3 scored one, and test scores was significant for the Massers (p < 3 scored zero. All further analyses excluded the .05) but not the Spacers. students who correctly answered zero (n = 3) or Discussion one (n = 3) of these five problems. It appears that For the longer retention interval of four the three students with scores of one were weeks, the distribution of 10 practice problems guessing, as four of the five correct answers were across two sessions was far more useful than the multiples of ten. Notably, this inclusion criterion massing of all 10 problems in the same session. of at least two correct is based solely on Thus, these data provide an instance of the performance during the first session because the spacing effect for a non-rote mathematics additional reliance on performance during the learning task in a non-confounded experiment. second practice set (which was delayed for the At the one-week retention interval, there was no Spacers) would have confounded the experiment. reliable difference between the two strategies Naturally, the Massers and Spacers performed among students who were tested only one week equivalently on the first practice set because the after learning. This result is consistent with procedures for these groups did not diverge until previous findings demonstrating no spacing after the first practice set was complete. effect or even massing superiority at sufficiently Specifically, the Massers averaged 88% (SE = short retention intervals, as described in the 2.3%) and the Spacers averaged 87% (SE = introduction. Despite this ambiguity after one 2.5%), F < 1. week, though, the spacing superiority after four However, for the second set of five weeks suggests that long-term retention, which is practice problems, the Massers averaged 94% Mathematics Learning 6 the focus of the present paper, is better achieved that each of the nine problems was presented by distributing practice problems across sessions. equally often. This was done to equate the Experiment 2 selection of practice problems given to Lo and The second experiment assessed the Hi Massers. In addition, the nine problems given effect of overlearning on retention by varying the to the Hi Massers were presented in one of three number of practice problems within a single different orders so that the first three problems session. The Hi Massers attempted 9 practice corresponded to the only three problems given to problems, whereas the Lo Massers attempted the same number of Lo Massers. The other only 3 practice problems (as detailed in Figure aspects of the procedure, including the five- 1C). Thus, the Hi Massers relied heavily on problem test, were the same as those in overlearning. Because this increase in the Experiment 1. number of practice problems produced a Results concomitant increase in time devoted to practice, Learning. The tutorial was again this manipulation is effectively a manipulation of sufficient to produce learning, as demonstrated total practice time. Students were tested either by students’ performance on the three practice one or four weeks later. As detailed in the problems completed immediately after the introduction, many researchers have found tutorial. Specifically, 65 of the 100 students benefits of overlearning on a subsequent test, but correctly answered all three problems, 23 scored the bulk of these experiments relied on relatively two, 10 scored one, and 2 scored zero. As in brief retention intervals. Experiment 1, students with scores of zero or Method one were excluded from further analysis. Participants. All three sessions were As expected, there was no reliable completed by 100 undergraduates at the difference between Hi and Lo Massers on the University of South Florida. The sample first three problems because these two groups included 83 women and 17 men, and none underwent the same procedure until after these participated in Experiment 1. An additional 17 three problems were completed. Specifically, the students completed the first session but failed to Hi Massers averaged 90% (SE = 2.3%) and the show for the second session. Lo Massers averaged 88% (SE = 2.3%), F < 1. Design. We manipulated two between- For the additional six practice problems given subjects variables: Practice Amount (Hi or Lo) only to the Hi Massers, accuracy averaged 95% and Retention Interval (1 or 4 weeks). Thus, each (SE = 1.3%). student was randomly assigned to one of four Test. As shown in Figure 1D, there was groups: Hi Massers with 1-week RI, Hi Massers virtually no difference between the Hi and Lo with 4-week RI, Lo Massers with 1-week RI, and Massers on either the one- or four-week test. Lo Massers with 4-week RI. Consequently, an analysis of variance revealed Procedure. Each student attended two no main effect of Practice Amount (F < 1) and sessions, separated by one or four weeks. At the no interaction between Practice Amount and beginning of the first session, each student was Retention Interval (F < 1). Not surprisingly, the randomly assigned to one of the four conditions main effect of retention interval was significant, 2 listed above. Each student then observed a F (1, 84) = 33.16, p < .001, ηp = .28. tutorial consisting of screen projections that Discussion included the complete solutions to Problems 10, The increase in the number of practice 11, and 12 of the Appendix, in that order. problems given during a single learning session Immediately after this tutorial, all students began had virtually no effect on subsequent test scores the practice problems. Each Hi Masser was given at either retention interval. This null effect of nine problems (which were Problems 1 through overlearning is not well explained by a lapse in 9 of the Appendix), and each Lo Masser was attention by the Hi Massers during their assigned three of these nine problems. The three additional six practice problems because these problems assigned to each Lo Masser varied, so Mathematics Learning 7 problems were solved with 95% accuracy. Thus, proceed well beyond that minimally necessary as fully described in the general discussion, these for an immediate, correct first reproduction” (p. results provide no support for the oft cited claim 181). Fitts concluded that, “The importance of that overlearning boosts long-term retention and continuing practice beyond the point in time therefore cast doubt on the utility of mathematics where some (often arbitrary) criterion is reached assignments that include many problems of the cannot be overemphasized” (p. 195). And Hall same type. wrote, “The overlearning effect would appear to General Discussion have considerable practical value since continued The two experiments assessed the practice on material already learned to a point of benefits of distributed practice and overlearning mastery can take place with a minimum of effort, on subsequent test performance. In Experiment and yet will prevent significant losses in 1, distributing 10 practice problems across two retention” (p. 328). In contrast to these sessions instead of massing all 10 problems in conclusions, the results of Experiment 2 revealed the same session had no effect on one-week test no effect of overlearning on retention. scores but virtually doubled four-week test Conceptually, the minimal effect of scores. In Experiment 2, increasing the number overlearning on retention can be interpreted as an of problems solved in a single session from three instance of diminishing returns. That is, with to nine had virtually no effect on test scores on each additional amount of practice devoted to a either the one- or four-week test. In brief, the single concept, there is an ever smaller increase extra effort devoted to additional problems in test performance. Thus, after the initial produced no observable benefit, whereas the exposure to a concept, the first one or two distribution of a given number of number of practice problems might yield a large increase in practice problems produced benefits without any a subsequent test score. Yet each additional extra effort. practice problem provides an ever smaller gain The results of previous experiments have until, ultimately, any additional practice within provided little support for distributed practice the same session will yield very little gain. with non-rote mathematics tasks. As detailed in It should be noted, though, that a small the introduction, three previously published amount of overlearning may be useful if experimental findings that are cited as instances overlearning is strictly defined as any practice of a spacing effect with a non-rote mathematics beyond one correct problem. By this definition, task are, in fact, confounded in favour of the overlearning occurs even when a student spacing effect. The results of Experiment 1, correctly solves only two or three problems, however, suggest that the superiority of which means that even the Lo Massers in distributed practice over massed practice extends Experiment 2 relied on a small amount of to these more abstract cognitive tasks. overlearning. Consequently, the results of Consequently, we concur with those authors who Experiment 2 do not support the extreme view have urged greater reliance on distributed that students should be assigned only one practice as a means of boosting long-term problem of each type in a given session. Instead, retention (Bahrick & Hall, 1991; Baddeley & the results suggest that students who correctly Longman, 1978; Bjork, 1979, 1988; Dempster, solve several problems of the same kind have 1989; Reynolds & Glaser, 1964; Schmidt & little to gain by working more problems of the Bjork, 1992: Seabrook, Brown, & Solity, 2005). same type within the same session. After these With regard to overlearning, however, the first problems are solved correctly, students present results strongly conflict with the could devote the remainder of the practice numerous claims about its utility as a learning session to problems drawn from previous lessons strategy (e.g., Fitts, 1965; Foriska, 1993; Hall, in order to reap the benefits of distributed 1989; Jahnke & Nowaczyk, 1998). Indeed, the practice. strategy of overlearning is widely advocated. As A final caveat concerns two important Jahnke and Nowaczyk advised, “Practice should limitations on the extent to which the present Mathematics Learning 8 finding will generalize. First, all of the number of practice problems, but only a few of participants in the present experiments were these problems relate to the immediately college students, and it is not known whether the preceding lesson. Additional problems of the results of both experiments would have been same type then appear perhaps once or twice in observed with much younger students. However, each of the next dozen or so assignments and we suspect that young children would provide once again after every fifth or tenth assignment qualitatively similar results because such thereafter. In brief, the number of practice parallels have been observed in previous problems relating to a given topic is no greater distributed practice experiments (e.g., Seabrook, than that of typical mathematics textbooks, but Brown, & Solity, 2005; Toppino, 1991). Second, the temporal distribution of these problems is because the experiments reported here relied on a increased dramatically. test that required students to solve problems While the distributed practice format identical to those presented during the practice should improve retention, it might also prove session (albeit with different numerical values), more challenging than the massed practice it is unknown whether the benefit of distributed format. With a massed practice format, students practice or the futility of overlearning would who have solved the first few problems have have occurred if the test had required students to little difficulty with the remaining problems of apply their previous learning in novel ways (i.e., the same type. Indeed, they merely need to assessed what is known as transfer). repeat the procedure. A distributed practice The Organization of Practice Problems in format, however, ensures that each practice set Mathematics Textbooks includes many different challenges. Thus, the Many mathematics textbooks rely on a challenge and long-term returns of a distributed format that fosters both overlearning and massed practice format provide an example of what practice. In these textbooks, virtually all of the Bjork and his associates have called a “desirable problems for a given topic appear in the difficulty” (Christina & Bjork, 1991; Schmidt & assignment that immediately follows the lesson Bjork, 1992). Despite this difficulty, however, on that topic. This format fosters overlearning some students might find the mixture of because each assignment includes many problems more interesting than a group of problems of the same kind. This format also similar problems. fosters massed practice because further problems A distributed practice format is used in of the same kind are rarely included in the Saxon series of mathematics textbooks (e.g., subsequent assignments. As an illustration, we Saxon, 1997). While numerous non-controlled examined every problem in the most recent studies have compared Saxon textbooks to other editions of four textbooks in pre-algebra textbooks, we are not aware of any published, mathematics or introductory algebra that are very controlled experiments with these textbooks. popular in the United States. The proportion of However, there may be little information to be the problems within each assignment that gained from an experiment in which students are corresponded to the immediately preceding randomly assigned to a condition that uses a lesson, when averaged across assignments, Saxon or non-Saxon textbook because the equalled between 75% and 92% for the four numerous differences between two such books. Thus, the format of these practice sets textbooks would confound the experiment. For facilitates overlearning and massing. example, if such an experiment did show Fortunately, there is an alternative format superior retention for users of the Saxon that minimizes overlearning and massed practice textbook, it is logically possible that this benefit while emphasizing distributed practice, and it was the result of textbook features other than the does not require an increase in the number of distributed practice format. (Neither author has assignments or the number of problems per had any affiliation with Saxon Publishers; assignment. With this distributed-practice however, the first author is a former mathematics format, each lesson is followed by the usual Mathematics Learning 9 teacher who has used both Saxon and non-Saxon program record the date of each problem mathematics textbooks in the classroom.) attempt. Perhaps a more informative experiment In addition to its effect on retention, a would compare two groups of students who distributed-practice format can also facilitate underwent instruction programs that differed learning because it allows students ample time to only in the temporal distribution of practice master a particular skill. For instance, if a student problems. For example, a class of students could is unable to solve the problems of a given type in be divided randomly into two groups, with each a single lesson, a distributed-practice format will group participating in the same class activities. provide further opportunities throughout the Every student would also receive a packet that year. included the same lessons in the same order. The Conclusion selection of practice problems would also be The results of Experiment 1 suggest that identical, but the practice problems of each type the retention of mathematics is markedly would be distributed or massed. improved when a given number of practice Textbook publishers could adopt a problems relating to a topic are distributed across distributed-practice format with little trouble or multiple assignments and not massed into one cost. They would merely rearrange the practice assignment. Moreover, this benefit of distributed problems in the next edition of their textbooks, practice can be realized without increasing the regardless of whether the lessons are changed as number of practice problems included in a well. Oddly, practice problems typically receive practice set typical of most mathematics relatively little attention from publishers and textbooks. Specifically, rather than require textbook authors, and the practice problems are students to work far more than just a few often written by sub-contracted writers. Yet the problems of the same kind in the same session, practice sets are as least as important as the which had no effect in Experiment 2, each lessons. In fact, as many mathematics teachers practice set could instead include problems will attest, a majority of their students never read relating to the most recent topic as well as the lessons and instead devote all of their problems relating to previous topics. This individual effort to the practice sets. distributed practice format could be easily In addition to the implications for adopted by the authors of textbooks and CAI textbook design, the benefits of distributed software. practice are equally applicable to the design of Any resulting boost in students’ the algorithms used in computer-aided mathematics retention might greatly improve the instruction (CAI). Unlike textbooks, the mathematics achievement, and there is little programs can provide individualized training and doubt that the mathematics skills of most error-contingent feedback, and an increasing students need improving. In one recent report on number of educators and agencies have urged mathematics achievement, less than one third of greater reliance on such technologies (e.g., a sample of U.S. students received a rating of “at Department for Education and Skills, United or above proficient” (Wirt et al., 2004). Such Kingdom, 2003). Yet virtually all currently reports often lead people to conclude that available CAI programs are designed to foster students are not learning, but it may be that many learning rather than retention. Of course, such mathematical skills and concepts are learned but programs could be easily adapted to incorporate later forgotten. The prevalence of such forgetting distributed rather than massed practice, and may partly reflect the widespread reliance on students’ compliance to a distributed practice practice schedules that proved to be the worst schedule could be verified by ensuring that the strategies in the experiments reported here. Mathematics Learning 2 References Baddeley, A. D., & Longman, D. J.A. (1978). Dempster, F. N. (1989). Spacing effects and The influence of length and frequency of their implications for theory and practice. training sessions on the rate of learning to Educational Psychology Review, 1, 309-330. type. Ergonomics, 21, 627-635. Department for Education and Skills, United Bahrick, H. P., Bahrick, L.E., Bahrick, A.S., & Kindgom (2003). Fulfilling the potential. Bahrick, P. E. (1993). Maintenance of (DfES publication 0265 / 2003). DfES foreign-language vocabulary and the spacing Publications. effect. Psychological Science, 4, 316-321. Donovan, J. J., & Radosevich, D. J. (1999). A Bahrick, H. P., & Hall, L. K. (1991). Lifetime meta-analytic review of the distribution of maintenance of high school mathematics practice effect: Now you see it, now you content. Journal of Experimental don't. Journal of Applied Psychology, 84, Psychology: General, 120, 20-33. 795-805. Bjork, R. A. (1979). Information-processing Driskell, J.E., Willis, R. P., & Copper, C. (1992). analysis of college teaching. Educational Effect of overlearning on retention. Journal Psychologist, 14, 15-23. of Applied Psychology, 77, 615-622. Bjork, R. A. (1988). Retrieval practice and the Earhard, B., Fried, C., & Carlson, G. (1972). maintenance of knowledge. In M.M. Interference, overlearning, and anticipation Gruneberg, P.E., Morris, & R.N. Sykes time. Journal of Experimental Psychology, (Eds.). Practical aspects of memory II 94, 345-347. (pp. 391-401). London: Wiley. Fishman, E. J., Keller, L., & Atkinson, R. C. Bloom, K.C., & Shuell, T.J. (1981). Effects of (1968). Massed versus distributed practice in massed and distributed practice on the computerized spelling. Journal of learning and retention of second- Educational Psychology, 59, 290-296. language vocabulary. Journal of Fitts, P. M. (1965). Factors in complex skill Educational Research, 74, 245-248. training. In R. Glaser (Ed.), Training Bromage, B. K., & Mayer, R. E. (1986). research and education (pp. 177-197). New Quantitative and qualitative effects of York: Wiley. repetition on learning from technical Foriska, T. J. (1993). What every educator texts. Journal of Educational Psychology, should know about learning. Schools in the 78, 271-278. Middle, 3, 39-44. Carpenter, S.K., & DeLosh, E.L. (2005). Gay, L. R. (1973). Temporal position of reviews Application of the testing and the spacing and its effect on the retention of effects to name learning. Applied mathematical rules. Journal of Educational Cognitive Psychology, 19, 619-636. DOI: Psychology, 64, 171-182. 10.1002/acp.1101 Gilbert, T. F. (1957). Overlearning and the Christina, R.W., & Bjork, R.A. (1991). retention of meaningful prose. Journal of Optimizing long-term retention and General Psychology, 56, 281-289. transfer. In D. Druckman & R.A. Bjork Glenberg, A. M., & Lehmann, T. S. (1980). (Eds.), In the mind’s eye: Enhancing Spacing repetitions over 1 week. Memory & human performance (pp. 23-56). Cognition, 8, 528-538. Washington DC: National Academy Greene, R. L. (1989). Spacing effects in Press. memory: evidence for a two-process account. Cull, W. L. (2000). Untangling the benefits of Journal of Experimental Psychology: multiple study opportunities and repeated Learning, Memory and Cognition, 15, 371- testing for cued recall. Applied Cognitive 377. Psychology, 14, 215-235. Mathematics Learning 2 Grote, M. G. (1995). Distributed versus massed Rohrer, D., Taylor, K., Pashler, H., Wixted, J. T, practice in high school physics. School & Cepeda, N. J. (2005). The effect of Science and Mathematics, 95, 97-101. overlearning on long-term retention. Applied Hall, J. F. (1989). Learning and memory, 2nd Ed. Cognitive Psychology, 19, 361-374. DOI: Boston: Allyn & Bacon. 10.1002/acp.1083 Jahnke, J.C., & Nowaczyk, R. H. (1998). Rose, R. J. (1992). Degree of learning, Cognition. Upper Saddle River, NJ: Prentice interpolated tests, and rate of forgetting. Hall. Memory & Cognition, 20, 621-632. Kratochwill, T., Demuth, D.M., & Conzemius, Saxon, J. (1997). Algebra I (3rd Ed.). Norman, W.C. (1977). The effects of overlearning on OK: Saxon Publishers. preschool children's retention of sight Schmidt, R. A., & Bjork, R. A. (1992). New vocabulary words. Reading Improvement, 14, conceptualizations of practice: Common 223-228. principles in three paradigms suggest new Krueger, W. C. F. (1929). The effect of concepts for training. Psychological Science, overlearning on retention. Journal of 3, 207-217. Experimental Psychology, 12, 71-78. Seabrook, R., Brown, G.D.A., & Solity, J.E. Krug, D., Davis, T. B., & Glover, J. A. (1990). (2005). Distributed and massed practice: Massed versus distributed repeated reading – From laboratory to classroom. Applied a case of forgetting helping recall. Journal of Cognitive Psychology, 19, 107-122. DOI: Educational Psychology, 82, 366-371. 10.1002/acp.1066 Postman, L. (1962). Retention as a function of Toppino, T. C. (1991). The spacing effect in degree of overlearning. Science, 135, 666- young children’s free recall: Support for 667. automatic-process explanations. Memory & Rea, C. P., & Modigliani, V. (1985). The effect Cognition, 19, 159-167. of expanded versus massed practice on the Wirt, J., Choy, S., Rooney, P., Provasnik, S., retention of multiplication facts and spelling Sen, A., & Tobin, R. (2004). The Condition lists. Human Learning: Journal of Practical of Education 2004 (NCES 2004-077). U.S. Research & Applications, 4, 11-18. Department of Education, National Center Reynolds, J. H., & Glaser, R. (1964). Effects of for Education Statistics. Washington, DC: repetition and spaced review upon retention U.S. Government Printing Office. of a complex learning task. Journal of Educational Psychology, 55, 297-308. Mathematics Learning 2 Appendix

Students were taught to calculate the number of unique orderings (or permutations) of a letter sequence with at least one repeated letter (e.g., abbbcc). For n items and k unique items, the number of permutations equals n! / (n1! n2! … nk!), where ni = number of repetitions of item i. For example, abbbcc includes six letters (n = 6) and three unique letters (k = 3), and the letters a, b, and c appear 1, 3, and 2 times, respectively (n1 = 1, n2 = 3, n2 = 2). Thus, by the formula, the number of permutations equals

6! / (1! 3! 2!)

= (6Χ5Χ4Χ3Χ2Χ1) / [(1)Χ(3Χ2Χ1)Χ(2Χ1)]

= (6Χ5Χ4) / (2)

= 60.

Tutorial and Learning Session (see the procedures of each experiment for details) 1. abccc 20 2. abcccc 30 3. aabbbbb 21 4. abbcc 30 5. aaabbb 20 6. aabbbb 15 7. aabb 6 8. abbccc 60 9. abccccc 42 10. aabbbbbb 28 11. abbcccc 105 12. abcccccc 56

Test abcc 12 aabbb 10 aabbcc 90 aaabbbb 35 aaaabbbb 70

Experiment 1 A Learning Procedure B Test Results 100% 1 week apart 9 75% Spacers 70% session one session two cy 64% ra Spacers 5 problems 5 problems Accu Massers 32% Massers − 10 problems

0% 14 Retention Interval (weeks)

Experiment 2

C Learning Procedure D Test Results 100%

session one 69%

cy 67% a Hi Massers Hi Massers 9 problems

Lo Massers 3 problems Accur Lo Massers 28% 27%

0% 14 Retention Interval (weeks)

Figure 1. Learning Procedure and Test Results for Experiments 1 and 2. Accuracy represents the mean percentage correct. Error bars reflect plus or minus one standard error.

Figure 1