Behavior Research Methods, Instruments, & Computers 1985, 17(2),352-358 An evaluation of statistical software for research and instruction DARRELL L. BUTLER Ball State University, Muncie, Indiana and DOUGLAS B. EAMON University of Houston-Clear Lake, Houston, Texas A variety of microcomputer statistics packages were evaluated. The packages were compared on a number of dimensions, including error handling, documentation, statistical capability, and accuracy. Results indicated that there are some very good packages available both for instruc­ tion and for analyzing research data. In general, the microcomputer packages were easier to learn and to use than were mainframe packages. Furthermore, output of mainframe packages was found to be less accurate than output of some of the microcomputer packages. Many psychologists use statistical programs to analyze ware packages available on the VAX computers at Ball research data or to teach statistics, and many are interested State University: BMDP, SPSSx, SCSS, and MINITAB. in using computer software for these purposes (e.g., Versions of these programs are widely available and are Butler, 1984; Butler & Kring, 1984). The present paper used here as points ofreference to judge the strengths and provides some program descriptions, benchmarks, and weaknesses of the microcomputer packages. Although evaluations of a wide variety of marketed statistical pro­ there are many programs distributed by individuals (e.g., grams that may be useful in research and/or instruction. see Academic Computer Center of Gettysburg College, This review differs in several respects from other re­ 1984, and Eamon, 1983), none are included in the present cent reviews of statistical programs (e. g., Carpenter, review. This choice was made because these programs Deloria, & Morganstein, 1984; Eamon, 1983; Francis, are typically not well documented and sometimes contain 1983; also see Wilson, 1983). First, the software evalua­ bugs (Cline, 1972). tion was done by undergraduates as well as by a profes­ There were 11 reviewers-a professional academic psy­ sional psychologist. Students were not used in other evalu­ chologist and 10 undergraduates. The students were ob­ ations of programs. Second, this review is more recent tained by advertising in psychology courses. All students and includes some changes in reported software as well had completed a lO-week course in elementary statistics as some new programs. Finally, the present evaluation with a passing grade. Some effort was made to ensure places more emphasis on the apparent audience of a pro­ that students differed substantially in overall grade-point gram. Other reviews typically have been directed to average (GPA) and in statistical sophistication. Each un­ professional users only. dergraduate was paid $3.35 per hour for about 10 h of work per week during the fall quarter. PROCEDURE A student worked with one software package each week (approximately) in our supervised microcomputer labora­ Software vendors were contacted in June and again in tory. However, the student was given no extra help from August or September 1984, and were asked to supply faculty and had to rely on manuals and other available review copies ofthe most recent versions oftheir statistical students to solve problems. This procedure provided a sen­ programs. When appropriate, vendors were asked to sup­ sitive measure of ease of program use, an acknowledged ply versions that could run on an Apple II series com­ problem when educators use mainframe statistical pack­ puter, but they were also informed that we would review ages such as SPSS (see Thisted, 1979). programs for other machines, such as the mM PC. A list All students were given the same assignment for all pro­ ofvendors who responded is given in Table 1, along with grams. They were supplied with two groups of data (A program names, prices, and addresses. One other source and B), each containing nine data sets. The first data set of software was included in this evaluation, the major soft- in Group A involved the integers 1 through 9. The other data sets in Group A were created by algebraically alter­ ing the first data set along the lines suggested by Longley Send requests for reprints to Darrell L. Butler, Department of Psy­ (1967). Longley suggested that accuracy of programs be chology, Ball State University, Muncie, IN 47306. tested by adding 100, 1,000, ... to each datum and Copyright 1985 Psychonomic Society, Inc. 352 STATISTICAL SOFTWARE 353 Table 1 Statpac ($400) Walonick Associates Software Products, Prices, and Vendors 5625 Girard Avenue South Program (Price) Vendor and Address Minneapolis, MN 55419 (612)866-9022 Apple II Programs Trajectories ($495)d DBI Software A-Stat ($200) Rosen Grandon Associates v4.0 One Energy Place v83.1 7807 Whittier Street 5805 East Pickard Road Tampa, FL 33617 Mt. Pleasant, MI 48858 (813)985-4911 (800)221-3791 Daisy ($200) Rainbow Computing, Inc. 'First disk on campus/additional disks on campus. bFirst workbook v2.2 8811 Amigo Avenue and disk on campus/each additionalworkbook anddisk. <Price reponed Northridge, CA 91324 here is for ANOVA only. There are 6 packages; each package is $85­ (213)349-0300 and (800)423-5441 $95, or one complete set can be obtainedfor $450. dFomu!rly SPS. ELF ($200) The Winchendon Group v5.0 3907 Lakota Road P.O.Box 10339 Alexandria, VA 22310 (703)960-2587 HSD ANOVA II ($150) Human Systems Dynamics rechecking program output. For our purposes, a slightly vl.J 9010 Reseda Boulevard, Suite 222 tougher standard was used. Eight other data sets were Northridge, CA 91324 created by adding 90,900,9,000, ... , or 900,000,000 (213)993-8536 and (800)451-3030 to each number in the original data set. The data sets in Keystat ($50/$IO)a Oakleaf Systems P.O. Box 472 Group B were slight variations of the nine sets in Decorah, IA 52107 Group A; one number in a Group A data set was changed Peachtree Statistics ($30) Peachtree Software to create an equivalent Group B data set. These changes 3445 Peachtree Road N.E. made the moments of the data sets in Group B different Atlanta, GA 30326 from the data sets in Group A. (800)554-8900 Student evaluators were asked to run independent Speedstat Softcorp International statistical group t tests between the first data set in (Vol. I, $249) 229 Huber Village Boulevard (Vol. 2, $299) Westerville, OH 43081 Group A (the integers 1-9) and each of the other data sets. (800)543-1350 Then they were asked to run one-way, independent group STAN ($250) Statistical Consultants analyses ofvariance (ANOVA). They were to run a sim­ vl.2 462 East High Street ple two-group analysis (using 1-9 vs. 91-99), and then Lexington, KY 40508 increase the complexity of the analysis by adding in the (606)252-3890 data sets having more digits per number. As a final step Statmaster (S2oo/S24)b Teri L. Bragg in formal evaluation, students were asked to run correla­ College Division Little, Brown and Company tions between the original data set and all the others. Stu­ 34 Beacon Street dents were also encouraged to try to solve problems in Boston, MA 02106 their statistics books and to try out other capabilities of SYSTAT ($495) SYSTAT, Inc. the programs. For example, several ofthe programs con­ v 1.3 603 Main Street tained chi-square statistics, and students were encouraged Evanston, IL 60202 (312)864-5670 to try some of these routines. Stat. and Prob. ($95)C COMPress Each student met with the first author ofthis paper once P.O. Box 102 each week to discuss the strengths and weaknesses ofthe . Wentworth, NH 03282 program used during the previous week. In addition, the (603)764-5831 students supplied computer-printed and handwritten USTATS rsioo: Wm. C. Brown Publishers records of their interactions with the programs. These 5712 Odana Road Madison, WI 53719 were studied (usually in the students' presence) to deter­ (608)273-0040 mine the ways in which the programs handled mistakes by the students, to determine whether vocabulary IBM PC Programs problems were creating difficulties for the students, and MICROSTAT ($325) Ecosoft, Inc. v4.1 6413 North College Avenue to determine the accuracy of the programs' output. Indianapolis, IN 46220 In addition to these student tests, the first author ran (317)255-6476 a number of checks on the flexibility and strength of the Number Cruncher ($200) Statistical Systems editors and analysis routines. These checks varied among v4.1 865 East 400 North programs because there were large differences in abili­ Kaysville, UT 84037 ties of the software. Also, the manuals were studied in (801)546-0445 detail, and an attempt was made to compute''readability NWA Statpak ($495) Northwest Analytical, Inc. v3.1 520 N.W. Dacis indexes" (e.g., Kincaid, Fishburne, Rogers, & Chissom, Portland, OR 97209 1975). Unfortunately, such indexes do not work well on (503)224-7727 manuals such as these. 354 BUTLER AND EAMON TYPES OF PROGRAMS ions about the programs are summarized after a cross­ comparison of the features of the programs. From the point of view of the user, there are three different apparent audiences of statistical programs. At GENERAL PROGRAM FEATURES one extreme are professional users who need programs that can analyze a substantial amount of data in a com­ Many of the basic program features are summarized plex experimental design, the kind of data many in Table 2. Nearly all of these programs are evolving. researchers produce in laboratories. Mainframe statistics Each time a new version is released, problems may be packages such as SPSS (and recent updates such as SCSS), eliminated, error handling may improve, new operating BMDP, MINITAB, and SAS have been used for this pur­ systems may be implemented, or hardware requirements pose.