Argument Maps Improve Critical Thinking

Teaching Philosophy, 27:2, June 2004 95 Argument Maps Improve Critical Thinking CHARLES TWARDY Monash University I’ve been skeptical about claims for various approaches to teaching critical thinking, including those for argument maps coming from nearby University of Melbourne. Indeed, confident in our skepticism, we at Monash Philosophy accepted a challenge to compare our methods with theirs on pre- and post-test gains on the California Critical Thinking Skills Test (CCTST) developed by Peter Facione (1990, 1992). The Monash students did a bit better than theirs on the pre-test, raising our hopes. But when Melbourne University’s post-test results showed far higher performance gains, I thought their method worth a closer look. In short, I think computer-based argument mapping is the key to Melbourne’s success, and that it should feature centrally in any critical thinking course. In fact, it is useful quite beyond critical thinking courses. I will describe the method, my experiences with it, and the results. Introduction Being soundly trounced on improvement scores only convinced us that something over there was working. I thought the approach was fine, and argument maps were certainly helpful, but suspected that most of the gains were due to founder effect: the Melbourne teacher (Tim van Gelder) developed the Reason! approach to critical thinking, the Reason!Able software,1 and the course materials used at Melbourne. In addition, he had been the sole teacher since they adopted this approach, so it was very likely that a lot of the result was because of his founder’s enthusiasm, a well-known effect in education. Tim also wanted to see how much was founder effect. So we replaced Tim with a relatively inexperienced postdoc from an outside university. Me. Tim got more time to develop materials and do research, and Monash got a “spy” to learn more about the Reason! method and why it worked so well. © Teaching Philosophy, 2004. All rights reserved. 0145-5788 pp. 95–116 96 CHARLES TWARDY The quick answer is that with my novice teaching skills thrown for the first time into an unfamiliar curriculum, student improvement on the CCTST was still 90 percent of Tim’s previous gains, and about three times the gross gain typical of other endeavors. The students improved noticeably despite what I must candidly regard as a poorly taught first-half of the subject. I’ll report on my experiences teaching the Reason! method, including common student difficulties and misconceptions, and benefits and drawbacks of the course materials and the Reason!Able software. I’ll start with some results, to show that there really is a difference, and move on to discuss the method in detail. However, as a quick introduction, I should say that the Reason! method consists mainly of intensive practice on computer-supported argument mapping in a highly structured setting. Numbers and Results This section highlights some of the results presented in Donohue et al. (2002). With a first-round entrance cutoff at the 93rd percentile of entrance score for students in Arts, Melbourne University is roughly equivalent to an elite private university in the United States. (There are no private secular universities in Australia.) It takes only the top 5–7 percent. Monash University, by comparison, has a first-round cutoff of about 88 for Arts. I lectured in a first-year introduction to critical thinking with about 135 students. I ran one of that class’s nine tutorials (“discussion sec- tions” in the U.S.). Mine had twelve students; Tim had two tutorials and the remainder were taken by philosophy graduate or honors students. I also ran a tutorial in a third-year History and Philosophy of Science class that used the Reason!Able software and argument maps to guide discussions. All tutorials used a computer-equipped classroom with an interactive whiteboard. In the first-year introduction, we measured the average gain (improvement) in performance on the CCTST. At the beginning of the semester, we randomly gave half the students one version of the CCTST, and half another version, and at the end of the semester, gave them the other version. Raw gains would apply only to this test, but we want a common measure we can use across tests. On common standardized effect size measures the average gain in terms of the variability in the scores: divide the average gain by that variability. The usual measure of variability is the standard deviation. The standard deviation is calculated so that a central slice of the bell-shaped curve one standard deviation to either side of the mean contains about 68 percent of the scores. Two standard deviations contain about 95 percent, and three standard deviations contain about 99 percent. So a ARGUMENT MAPS IMPROVE CRITICAL THINKING 97 tall, narrow bell will have a small standard deviation, while a wide flat bell will have a large standard deviation, but no matter the width of the bell, one standard deviation will capture 68 percent of the scores. If we divide the gain by the standard deviation in raw test scores, we get a standardized effect size called Cohen’s d (Cohen, 1988). It measures the number of standard deviations’ improvement, and so gives a measure that takes inherent variability into account.2 The result was that my first-year class had a gain of about 0.72 standard deviations, down slightly from Tim’s average of about 0.8. On the other hand, an effect size of 0.7 compares favorably with the 0.3 typical of other endeavors, and the 0.5 in a critical-thinking subject at McMaster University. (van Gelder, 2001; Donohue et al. 2002; Hitchcock, 2003) More results are shown in Figure 1. Because the confidence intervals do not overlap the line at 0.34, we know that the Reason! results are clearly above the expected gain for one whole year in university, and so represent a real improvement.3 The other subjects are not clearly adding anything to this baseline “maturation” effect. Indeed, our Monash CT subject is up with the rest, despite being only six weeks of lecture time in the elapsed twelve weeks. 1 0.8 0.6 0.4 Average gain for ﬁ rst year of university 0.2 Estimated gain for ﬁ rst semester of university 0 Fullerton Monash Monash Fullerton Melb McMaster Reason Reason Reason Phil CT Phil CTs HPS LEMUR 2002 2001 2000 1990 2001 2001 1990 2001 2001 Figure 1. Effect sizes in standard deviations (Cohen’s d) on the CCTST for various subjects. 98 CHARLES TWARDY To put the numbers in further perspective, consider general educational testing results. A full three years of undergraduate education has an effect size of about 0.55 on the CCTST, with about 0.34 occurring in the first year. (Pascarella and Terenzini, forthcoming). Or consider the SAT college entrance exam, which has had a standard deviation of about 115 in recent years. (Educational Statistics, 1997) Therefore an effect size of 0.8 would correspond to an 90-point increase on SAT score. In contrast, major (and expensive) test-prep companies claim to provide a 120–140-point average increase but ac- cording to a College Board study4 they actually provide an average increase of only 29 points on the verbal and 40 points on the math. (Powers and Rock, 1998) And there is no reason to believe SAT coach- ing transfers to anything other than the SAT (because it often consists largely of test-taking strategy specifically for the SAT, plus some crash review), while there is good reason to believe that Reason!-based critical thinking does transfer (perhaps even to verbal SAT scores, because they rely in part on logical thinking and analysis similar to what is found on the CCTST). But anyone can get large effects by teaching to the test. However, a CCTST question is typically something like: When Mark is late for dinner, both Susan and Andrew eat out. Which of the following must be true? 1. Mark is late, Susan eats out, Andrew eats out 2. Mark is not late, Susan eats in, Andrew eats in In contrast, as we will see below, an argument mapping question is typically a passage of text and a request to produce and evaluate a map. For comparison, interactive methods of physics education such as those of Eric Mazur (Mazur, 1997) get a stronger effect. In a survey of 62 introductory physics courses (6,500 students) using the standard Force Concepts Inventory pre/post test, Richard Hake (Hake, 1998, 1999) found that interactive instruction yielded a gain of 2.18(!!) standard deviations, versus 0.88 for traditional methods.5 On the other hand, they are teaching not just a skill but domain knowledge, and in class they frequently discuss conceptual questions like those which would appear on the FCI.6 Limitations of the Empirical Study The previous section aims to show that the Reason! method is much more effective than other methods at teaching critical thinking. There are two main avenues of criticism: results on the CCTST may not be the right measure, and there may be problems with this kind of study design and analysis. ARGUMENT MAPS IMPROVE CRITICAL THINKING 99 The CCTST is not perfect, but there is good reason to think it is relevant. Nevertheless, it is good to know that in the years where other tests were also given, independent assessments of written essays simi- larly favored the Reason! approach, as did the Watson-Glaser Critical Thinking Appraisal (Donohue et al., 2002).

Load more