TSOXXX10.1177/0092055X14527909Teaching SociologyDelucchi 527909research-article2014 Teaching Sociology 2014, Vol. 42(3) 231 –239 Measuring Student Learning © American Sociological Association 2014 DOI: 10.1177/0092055X14527909 in Social Statistics: A Pretest- ts.sagepub.com Posttest Study of Knowledge Gain Michael Delucchi1 Abstract This study used a pretest-posttest design to measure student learning in undergraduate statistics. Data were derived from 185 students enrolled in six different sections of a social statistics course taught over a seven-year period by the same sociology instructor. The pretest-posttest instrument reveals statistically significant gains in knowledge for each course section and all sections combined. The results demonstrate that pretests can establish students’ prior knowledge at the beginning of the semester, while posttests measure learning at the end of the course. Namely, pretest-posttest knowledge gain was influenced more by the content and presentation of the social statistics course than by students’ statistical ability and/ or test-taking skills prior to the class. Recommendations on the use of pretest-posttest data to inform pedagogy in social statistics are included in the discussion. Keywords direct assessment, student learning, knowledge gain, social statistics The baccalaureate degree curricula of most under- (DeCesare 2007; Lomax and Moosavi 2002), and a graduate sociology and other social science pro- belief that learning was greater than students could grams require one or more courses in statistics. have achieved without the instructional innovation Despite the importance of quantitative data analysis (Auster 2000; Wybraniec and Wilmoth 1999; in these disciplines, students with limited mathemat- Yamarik 2007). ics backgrounds and anxiety over statistics present a Upon close review, many of these studies pro- challenge to professors teaching these courses vide little or no (direct assessment) empirical evi- (Bandalos, Finney, and Geske 2003; Blalock 1987; dence that students’ statistics skills and knowledge Cerrito 1999; Forte 1995; Garfield and Chance (i.e., learning) actually increased because of the 2000; Macheski et al. 2008; Wilder 2010). Not sur- teaching strategy. Assessment is subjective and fre- prisingly, faculty assigned to teach statistics search quently relies on the perceptions of students or fac- for approaches that improve students’ statistical ulty (Fisher-Giorlando 1992; Lomax and Moosavi skills. Numerous classroom techniques (e.g., small- 2002; Marson 2007; Schacht and Stewart 1992). group work, collaborative testing, humor, computer- While not without some merit, comments based on assisted instruction, active learning, etc.) have been described in college statistics courses (Delucchi 1University of Hawaii, Kapolei, USA 2006; Helmericks 1993; Schacht and Stewart 1990; Corresponding Author: Schumm et al. 2002; Strangfeld 2013). Faculty Michael Delucchi, Division of Social Science, University using these practices report greater student satisfac- of Hawaii, 91-1001 Farrington Highway, Kapolei, HI tion with the course (Fischer 1996; Perkins and Saris 96707, USA. 2001; Potter 1995; Stork 2003), reduction of anxiety Email: [email protected] Downloaded from tso.sagepub.com at ASA - American Sociological Association on July 2, 2014 232 Teaching Sociology 42(3) informal impressions, and even quantitative mea- (McCormick 2001). The institution is coeduca- sures of course satisfaction (e.g., student evalua- tional (68 percent women; 32 percent men), ethni- tions of teaching or alumni surveys), do not directly cally diverse (59 percent ethnic minorities), and signify student learning. As indicators of perceived comprised predominantly of students of nontradi- knowledge (rather than actual knowledge), these tional age (65 percent are 25 years of age or older). indirect methods of assessment are limited because Eighty-two percent of students are employed (40 assumptions must be made about what such self- percent working more than 31 hours per week), and reports mean (Price and Randall 2008). all students commute to the campus. Students’ academic performance, such as exami- nation scores and course grades, as a proxy for learn- ing (Borresen 1990; Delucchi 2007; Perkins and Course Description Saris 2001; Smith 2003; Yamarik 2007) also does Statistical Analysis is an undergraduate course not represent direct evidence of learning (Baker taught in the Division of Social Sciences. The 1985; Chin 2002; Garfield and Chance 2000; Lucal course introduces students to descriptive and infer- et al. 2003; Wagenaar 2002; Weiss 2002). Why not? ential statistics. Completion of College Algebra (or Learning is different from performance. Learning is a higher-level mathematics course) with a grade of increased knowledge: that is, the difference between “C” or better is the prerequisite. Statistical Analysis what students know at the beginning of the semester is one of two methods courses required for all compared with the end of the semester. Performance social science majors (e.g., anthropology, econom- is demonstrating mastery: for example, accurate ics, political science, psychology, and sociology) at computation of a statistic or a correct answer to a the university. In addition, the course fulfills a core multiple-choice question. This distinction between requirement for professional studies majors (e.g., learning and performance is important since stu- business/accounting and public administration). As dents enter courses with unequal knowledge, skills, a result, approximately 68 percent of the students and educational experiences. For instance, a student enrolled in Statistical Analysis are social science may begin the semester knowing little, learn a great majors and 32 percent come from professional deal, perform adequately, and receive average studies. grades, or a student may enter a course knowing a great deal, learn a small amount, perform very well, and earn high grades (Neuman 1989). Consequently, Sample pretests are necessary to establish prior knowledge, Student data were derived from enrollment lists and posttests are requisite to measure learning. and class records for six sections of Statistical Direct assessment of student learning in under- Analysis that I taught over a seven-year period. graduate statistics courses, based on pretests and Complete information was obtained for 185 of the posttests, is rare (Bridges et al. 1998; Price and 214 students enrolled in the course at the beginning Randall 2008). An objective of this study is to eval- of each semester, an 86 percent response rate. The uate the effect of course completion on students’ class met for 75 minutes, twice a week, during a statistical knowledge. Therefore, I use a pretest- 16-week semester. The course consisted primarily posttest design to measure students’ prior course of lectures on descriptive and inferential statistics knowledge and to measure learning at the end of that paralleled chapters in the text and readings in a the semester. This method of assessment is pro- booklet about the Statistical Package for the Social ceeded by recommendations on the use of pretest- Sciences (SPSS) (Stangor 2000). posttest data to inform pedagogy in undergraduate Requirements for the course included Examination social statistics. 1 (15 percent), Examination 2 (20 percent), a final examination (35 percent), two small group projects worth 10 percent each, and 12 quizzes weighted a DATA AND METHODS combined 10 percent. Textbook homework and computer exercises were assigned but not collected Institutional Context on a regular basis.1 This approach was intended to The study was conducted at a small (approximately encourage students to be proactive and self-diag- 1,500 students) state-supported baccalaureate nostic with regard to course content. Students degree–granting university in the United States. reporting difficulty completing these assignments The “Carnegie classification” describes the institu- were invited to seek my assistance prior to the due tion as a Baccalaureate College–Liberal Arts date. While the text (most recently Levin and Fox Downloaded from tso.sagepub.com at ASA - American Sociological Association on July 2, 2014 Delucchi 233 2011) and SPSS booklet (Stangor 2000) changed as semester. These examinations required students to new editions became available, the instructor, lec- perform computations and interpret data. During tures, homework assignments, quizzes, group proj- each 75-minute test, students worked indepen- ects, examinations, and grading criterion were dently but were permitted to use calculators, text- essentially constant across the six sections of the books, lecture notes, quizzes, and homework course. assignments. The three examinations were scored on a 0- to 100-point scale. Approximately once a week during the final 10 Pretest-Posttest Instrument to 15 minutes of class, students completed a quiz. To assess students’ statistical knowledge, a com- Each quiz involved computations and interpreta- prehensive multiple-choice examination was tions similar to (but less rigorous than) those on administered at the second class meeting during the examinations.4 Students could use calculators, text- first week of the semester.2 This pretest contained books, lecture notes, and their homework but were 30 questions on descriptive and inferential statis- required to complete quizzes independently. The tics derived from “typical” computational and first four quizzes covered descriptive statistics