Journal of Theoretical and Philosophical Psychology © 2011 American Psychological Association 2011, Vol. 31, No. 1, 25–43 1068-8471/11/$12.00 DOI: 10.1037/a0021575 Methods Courses and Texts in Psychology: “Textbook Science” and “Tourist Brochures”

Russell E. Costa and Charles P. Shimp University of Utah

Two studies examined the possibility that instruction in psychological methodology is committed to a philosophy of science, logical positivism, that is not adequately acknowledged and is empirically problematic. Study 1 suggested that psychology departments had more courses in methodology than corresponding physics depart- ments, and psychology departments were far more likely to offer an introductory course in general methodology. Study 2 suggested that psychology had more introductory general methods textbooks than did physics. Both studies suggested psychology still presents itself as the kind of science that was defined by logical positivism. Study 2 suggested psychology does this only implicitly in its texts and without acknowledging problematic features of that philosophy. Psychology methods texts commonly justified the importance of the scientific method by describing human cognitive fallibilities without cautioning that these same fallibilities could be seen alternatively as showing the problematic nature of the positivistic conception of scientific methodology. In several regards, texts conformed to Fleck’s notion of “textbook science” and to Kuhn’s description of science texts as “tourist brochures.” It was cautioned that an implicit appeal by psychology to logical positivism contradicts its claim that its methods and data are objective, transparent, and little influenced by philosophy, logic divorced from data, or authority.

Keywords: psychological methods, implicit logical positivism, methods textbooks

C. S. Peirce’s (1877/1934) 19th century ar- inconsistency, and we describe consequences of guments for the superiority of scientific knowl- this inconsistency for the instruction in, and edge over other forms of knowledge are still practice of, psychological science. used in psychology today in instruction on the Peirce may have been aware that his argu- scientific method. He argued that the scientific ment in favor of the scientific method was based method generates a special, objective form of on tenacity, authority, philosophy, and logic knowledge unlike that generated by other meth- when he wrote: “The test of whether I am truly ods of fixing belief such as through sheer rep- following the method is not an immediate ap- etition (that he called “tenacity”), by appealing peal to my feelings and purposes, but, on the to authority, or by philosophical or logical a contrary, itself involves the application of the priori reasoning without empirical support. Pei- method” (Hartshorne & Weiss, 1877/1934, p. rce’s advocacy of the scientific method was, 245). Thus, he implied that understanding the however, inconsistent in the sense that he advo- scientific method required applying it to itself. cated its use to establish belief yet he advocated In his advocacy of this reflexive use of the for belief in the method itself instead through scientific method, Peirce was ahead of his time; tenacity, authority, philosophy, and logic. We neither he nor any of his contemporaries imple- suggest in this paper that instruction in psycho- mented his recommendation to apply the scien- logical methods can still today reveal this same tific method to study science itself. More recently, however, such a metascience has emerged. There are now research programs Russell E. Costa and Charles P. Shimp, Department of using the empirical methods of anthropology, Psychology, University of Utah. history, psychology, and sociology to inform Correspondence concerning this article should be addressed to Charles P. Shimp, Department of Psychology, University of our understanding of science, of how science Utah, 380 South 1530 East Room 502, Salt Lake City, UT practice is learned, and of the scientific method 84112-0251. E-mail: [email protected] (e.g., see Diener, 2009; Faust, 1984; Feist,

25 26 COSTA AND SHIMP

2006, 2008; Gholson, Shadish, Neimeyer, & the human condition, with all of its fallibilities, Houts, 1989; Gopnik & Schulz, 2007; Gorman, from the scientific method in order to give sci- 1992; Gorman, Tweney, Gooding, & Kincan- entific knowledge a privileged status over other non, 2005; Guba & Lincoln, 2005; Klahr & forms of knowledge. Logical positivists argued Nigam, 2004; Kuhn, Amsel, & O’Loughlin, on behalf of their view of science by appealing 1988; Mahoney, 1976; Mendelsohn, Weingart, to logic as developed by Whitehead and Russell & Whitley, 1977; Merton, 1938/1970; Price, (1910), by resorting to authority (Carnap, 1936, 1961; Ravetz, 1971; Rozin, 2009; Shimp, 2001; 1937), and by tenaciously repeating their views Simon, 1986; Slife, Reber, & Richardson, 2004; (Bridgman, 1927; Pratt, 1939; Stevens, 1935). Smith, Harre, & Langenhove, 1995). Empirical They did not appeal to empirical science itself work of these types is facilitating a new meta- to justify their position on the grounds that science more than a hundred years after philoso- scientific knowledge changes and is therefore an phers like Peirce observed, if only in an incon- unreliable foundation for a stable science. In sistent and nonempirical way, the need for this manner, logical positivism elevated discus- science to be directed at itself. For example, one sions about the scientific method above empir- founder of this new metascience used archival ical science itself, and thereby delayed empiri- methods to describe growth patterns in scien- cal investigations of the scientific method. tific publications and resulting implications for The logical positivism of the 1920s, 1930s, the practice of science (Price, 1961), and an- and 1940s developed methods by which it was other used archival methods to examine how proposed to discriminate scientific from nonsci- science could be viewed as a cultural practice entific knowledge (Bridgman, 1927; Carnap, (Merton, 1938/1970). More recently, Winston 1936, 1937; Feigl, 1970; Hempel, 1966; for (2004) has used archival methods to empirically discussion, see also Benham & Shimp, 2004; examine the language psychology textbooks use Feyerabend & Maxwell, 1966; Laudan, 1990). to describe research methods and experimental These methods included operational definitions, design. The present paper uses archival methods hypothesis testing, and means by which one to examine how contemporary instruction in could identify causal relations and the cumula- general psychological methodology depends on tive growth of reliable knowledge. Operational- the very methods of fixing a person’s belief, ism (Bridgman, 1927) was adopted, for exam- including tenacity, authority, philosophy, and ple, as a way to transform ambiguous, intuitive, logic that Peirce (1877/1934) suggested were natural language terms such as “length,” inferior to the scientific method. “speed,” or “weight” into empirical, objective, unambiguous, standardized methods by which Logical Positivism and Psychology the concepts were defined and measured. An operational definition was claimed to have the Now that this curiously long delay between virtue of expunging subjective qualities such as the rise of the social and behavioral sciences in perception, judgment, and memory from scien- the late 19th century and their application to the tific concepts. In this sense, operationalism cap- study of science has ended, explanations have tured an idea latent in the name “scientific arisen to explain it, with the dominant explana- method” itself; it was a special method that tion focusing on a once-prominent philosophy defined and legitimized science (Mertens, of science, logical positivism (Feist, 2006; 2005). While logical positivism’s view of sci- Koch, 1992a; Kuhn, 1996; Toulmin & Leary, ence was not identical to Peirce’s, it was similar 1992). In this paper, we describe a critical way in the sense that it based its claim that science in which that philosophy’s impact on psychol- provided a special and superior form of knowl- ogy is far from over. To describe this persis- edge through its use of a special kind of method: tence of logical positivism and its contemporary the scientific method. effects, it is necessary first to summarize its From its earliest laboratory efforts, psychol- basic tenets, its historical development, and ogy explicitly claimed to model its methods on philosophical and practical criticisms of it. Its the methods of the physical sciences, on the tenets involved an ahistorical, cumulative- grounds that doing so would facilitate its be- growth image of science, the purpose of which coming a legitimate science (e.g., Fechner, was to remove to as large a degree as possible 1860/1966; Watson, 1913). Thus, when logical INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 27 positivism appeared, it was welcomed by many become problematic; it is unenviable also be- psychologists, especially experimental psychol- cause it shares the same inconsistency as did ogists who viewed their research as putting psy- Peirce’s argument by justifying empirical meth- chology on a sound, rigorous empirical footing ods by nonempirical means. and who were looking for what they felt was a This brief review of logical positivism and its compatible philosophical foundation (e.g., Es- historical development suggests that contempo- tes, 1959; Hull, 1943; Pratt, 1939; Stevens, rary psychology may still be in an “unenviable 1935). Features of logical positivism therefore position” in which it is beholden to a former quickly came to govern how many psycholo- philosophy that grounded empirical science in gists evaluated the extent to which any psycho- nonempirical philosophical arguments, and is logical research program was scientific (for dis- isolated from contemporary philosophy of sci- cussion, see Capaldi & Proctor, 1999; Cohen, ence. A concrete example of this unenviable 1981; Koch, 1992a; Lacey, 2005; Laudan, position is the current status of an operational 1990; Pratt, 1939; Skinner, 1938; but see also definition. Bridgman (1927) originally pro- Skinner, 1945; Smith, 1986; Toulmin & Leary, moted this idea as a way to turn an abstract, 1992). ambiguous concept into an empirical method At about the same time many psychologists and thereby presumably to give it objective were accepting the philosophy of logical posi- meaning. Bridgman (1959) later qualified and tivism, it became the object of increasing criti- softened his original view, perhaps in part be- cism from within philosophy itself. Criticism cause of rising criticisms of logical positivism, was aimed at its claims about the logic of hy- and he specifically criticized how behavioral pothesis testing and its verification theory of psychology had adopted it (see also Koch, meaning. Quine (1951) especially challenged its 1999b). We show below that students of psy- claim that individual theoretical assumptions chological methods are often introduced even could be empirically tested; he suggested in- today to Bridgman’s original meaning of oper- stead that any test can examine any one assump- ational definition as part of instructional efforts tion only within a broader theoretical context, to legitimize psychology’s status as an objective so that a test of any one assumption is actually science grounded in the scientific method. a test of the broader theoretical context within Logical positivism’s outdated tenets persist, which that assumption has meaning. Criticism we suggest, because psychology adopted its was also aimed at its assumption that observa- ahistorical picture of science. Why should psy- tion is objective on the grounds that observation chology, if it has adopted such a view of sci- is theory laden (Hanson, 1958). Moreover, his- ence, historically analyze its relation to philos- torians and philosophers of science criticized ophy? The view we empirically examine here is logical positivism because it failed to accurately that logical positivism’s impact on psychology describe how science has actually been con- persists in part because psychology curricula ducted (Feyerabend, 1975; Hanson, 1958; and methods textbooks advocate it, but in a way Kuhn, 1993, 1996) thereby perhaps implying that discourages awareness of that advocacy, let that it was a vision of science that could not be alone historical and philosophical analysis of its actually implemented. Nonetheless, psychology origins. Our position is not uncommon among still found itself committed to this flawed phi- philosophers, historians, and sociologists of sci- losophy of science, leaving Koch (1999a; pp. ence. For example, Fuller (1997, p. 80) wrote 55–56) to note: “Psychology is thus in the un- that: enviable position of standing on philosophical While it is now widely conceded that the most perva- foundations which began to be vacated by phi- sive accounts of the history of science—those found in losophy almost as soon as the former had bor- science textbooks and science popularizations—are rowed them.” Similarly, Meehl (1970, p. 403) misleading to the point of being self-serving, few have wrote how “. . .many psychologists” were “es- stepped back to wonder what this might mean for an pousing a philosophical position that is some 30 activity that advertises itself as unique in its quest for some ultimate, singular and unified truth. years out of date.” This position is not just unenviable because its tenets involving opera- If Fuller’s analysis is correct, then at the tional definitions, the logic of hypothesis test- undergraduate instructional level, academic ing, and the objectivity of observation have psychology might often ironically argue on be- 28 COSTA AND SHIMP half of a scientific psychology different from to protect its basic assumptions from critical pseudo science and folk psychology but might theoretical objection or empirical examination. do so by using unscientific methods, including, According to these two perspectives, which to use Peirce’s (1877/1934) terms, authority (in we find to be in uncomfortably close agreement the form of textbooks and instructors), logic (in with more of our own academic experiences the description of methods such as “double- than we would wish were the case, instruction blind” methods that are said to logically prevent in psychological methods tends to present, certain specific forms of confounds), and tenac- knowingly or unknowingly, the charms of sci- ity (in the form of repeated exposure in psycho- entific psychology without its empirical and logical curricula to descriptions of scientific conceptual ambiguities, without the arbitrari- methodology and its virtues). According to this ness of its definitions, and without mention of view, psychology curricula and textbooks its unresolved theoretical problems. On this might give students a self contradictory pic- view, a young science like psychology, strug- ture of psychological science; they might ad- gling to present itself as a legitimate science, vocate one set of methods compatible with might especially wish to suppress that some of logical positivism while justifying them via its critical assumptions stem from a problematic another, incompatible set, without acknowl- philosophy of science. This suppression may edging or explaining the contradiction. This serve to sustain logical positivism’s unexam- possibility seems to pose a threat to teaching ined influence and by doing so may hinder scientific standards of objectivity, transpar- efforts to improve methodological practices in ency, and empirical validation. scientific psychology.

Two Studies of “Textbook Science” in “Textbook Science” and “Tourist Psychology Brochures” Undergraduate psychology curricula in the Similar possibilities have been described be- United States commonly include courses in fore. Kuhn (1996, pp. 1) suggested that the general methodology to teach students how to conventional image of science is derived in conduct psychological experiments, how to in- large measure from “the textbooks from which terpret the resulting data, how to discriminate each new scientific generation learns to practice science from pseudo science, and how to apply its trade.” He called their aim “pedagogic and the scientific method. These courses are often persuasive,” and he likened them to tourist bro- core courses required for a major in psychology. chures, which are well known to present their On the basis of our brief summary of the tenets subject in glowing terms without mentioning of logical positivism, of how it was adopted in any undesirable social, economic, or political psychology, and of how it persisted in psychol- problems. Tourist brochures describe and mar- ogy after it was largely abandoned in philoso- ket a vacation destination from a particular per- phy, and after reviewing the notions of textbook spective to highlight only its charms. Kuhn’s science and texts as tourist brochures, we spec- view of science texts implied they similarly aim ulated that psychology curricula and textbooks to persuasively argue the charms of a science would show a commitment to logical positiv- and to suppress conflicting information. In a ism, but without explicit philosophical analysis similar vein, Fleck (1935/1979) used the term and without empirical support. That is, we con- “textbook science” in his description of how jectured that crucial courses and texts in psy- textbooks present a vision of a science that has chology might portray psychology in Peirce’s been conceptually tidied up, made more coher- inconsistent manner; curricula and introductory ent, and generally made more palatable than the texts on research methods might promote logi- complex empirical literatures that the texts pur- cal positivism’s vision of the scientific method, port to describe really are. Both Fleck and Kuhn and might do so by unscientific and unacknowl- suggested that texts present a scientific disci- edged means. pline in the context of an ahistorical, cumulative An implicit advocacy of a philosophy merits growth interpretation of science to justify ignor- empirical investigation because if such advo- ing the discipline’s complex development and cacy is pervasive, then students are taught a INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 29 vision of empirical science that rests upon em- Others have expressed similar opinions about pirically unverified philosophical assumptions. psychology and its wish to emulate physics by Accordingly, we conducted two studies using conforming to the scientific standards of phys- archival research methods that examined the ics (e.g., Bayer, 2004; Koch, 1992a). possibility that psychology curricula and text- books have an unacknowledged commitment to Study 1: Methods Courses logical positivism. We hypothesized that in- structional curricula and methods texts in psy- To emulate physics, psychology would have chology would differ from those in the natural to have a vision of what physics is. If that vision sciences, specifically physics, because psychol- were similar to how Fuller, Koch, and others ogy, an example of a “soft science” might make described it, psychology might attribute to claims about its scientific legitimacy based on physics a greater commitment to logical posi- the standards logical positivism set for scientific tivism than physics actually displays. Specifi- methodology. The American Psychological As- cally, psychology might display a greater ten- sociation (APA) recently developed its first stra- dency to adopt the language and terminology of tegic plan, with one of its three “top priorities” logical positivism and might assign greater in- being “increasing recognition of psychology as structional resources to features of it than phys- a science” (Anderson, 2009, pp. 9). In the phi- ics does. The chief feature we examined in losophy of logical positivism, methodology is Study 1 was the use of positivistic terminology in undergraduate course titles, specifically, ter- assumed to be not only separable from theory minology involving general “methodology.” but to have priority over theory. Thus, if logical We asked if psychology placed greater weight positivism affects how psychology sees itself, on instruction in general methodology, which then it might use methodology to justify its according to the tenets of logical positivism was position as a science. Advocacy for psychology the key to achieving reliable, factual, objective as a science by APA, by curricula, and by texts scientific knowledge that was not contaminated by all might conform to logical positivism’s view theoretical speculation or by dependence on spe- of science and argue psychology is a science cific theoretical assumptions. Thus, Study 1 exam- because its methods are scientific (also see ined whether psychology offered more methods Mertens, 2005; Proctor & Capaldi, 2006). We courses than physics did and more specifically, expected that in contrast, physics, perhaps whether psychology offered more general, intro- the archetypal example of a “hard science,” ductory level methods courses. would have less need to look beyond itself to convince other scientists and the public to ac- Method cept it as a science, would have less need to accept logical positivism’s sharp distinction be- Sample. We surveyed course catalogs in tween objective method and speculative theory, psychology and physics departments of the and would show less interest in promoting its top 25 national universities and top 25 liberal methodology as the key to its scientific legiti- arts colleges as rated in 2009 by U.S. News & macy. Indeed, one prominent philosopher/ World Report (USNWR; U.S. News & World sociologist of science wrote (Fuller, 2003, p. Report, 2009), a popular ranking system often 46): utilized by prospective students and cited by the universities and colleges themselves for public- Perhaps I am focusing unfairly on the lack of impact ity purposes. We hasten to acknowledge that that the positivists have had on the physics agenda. any ranking of the top universities is controver- Subtle readers of the history of science know that positivism was always a “made for export” philosophy. sial and depends on the evaluative criteria. In other words, positivists have wanted to spread what However, the rankings of USNWR have the they took to be the secret of physics’ scientific success virtues for present purposes of being widely to the more backward disciplines. . . To imagine what disseminated, of conforming to well articulated positivism looks like to a physicist, consider how the standards published in the magazine, and of zeal of a missionary or an imperialist appears to an enlightened believer or citizen at home: what causes being generated independently from the present the domestic market to cringe may just impress the study so that selection bias on our part is un- overseas market. likely. Furthermore, the colleges and universi- 30 COSTA AND SHIMP ties in the sample may be assumed to be among ods of developmental psychology, methods of the most nationally and internationally influen- clinical psychology, etc.), as we were interested tial in terms of their instructional impact on only in how psychology and physics allocated future academics who themselves will be in- curricular space to the teaching of general, in- volved in curriculum design. Table 1 lists the troductory level methodology. The number of colleges and universities included in the sample. departments that offered introductory level gen- (Note that 52 colleges and universities actually eral methods courses was then tallied for each were included because of ties in the 25th rank in discipline. the USNWR survey.) We defined a psychology department as one with “psychology,” “psycho- Results logical,” “behavior,” or “behavioral” in its name. Thus, departments named “psychological Overall, 5,295 courses across 52 colleges and science” or “psychology and brain science” universities were identified in our sample: 3,044 were included, but those with titles such as in psychology and 2,251 in physics. We found “cognitive science” or “neural sciences” were that 145 out of 3,044 (4.8%) psychology not included because they did not have “psy- courses and 69 out of 2,251 (3.1%) physics chology” in their name. Forty-nine of the 52 courses were methods courses by our definition. institutions had qualifying psychology depart- Thus, the percentage of psychology courses that ments. All 52 institutions had qualifying phys- were methods courses was about 50% greater ics departments. than the corresponding percentage in physics. Procedure. For each institution included in Most importantly, investigation of the course the sample, we examined the publicly available, catalogs revealed that 23 of the 49 psychology online version of the general course catalog for departments offered an introductory level gen- the 2009–2010 academic year. First, we tallied eral methods course while only 2 of 52 physics the overall number of psychology and physics departments did. Therefore, about 47% of the courses at each institution. Courses were de- psychology departments offered introductory fined as any listing that had an independent general methods courses while only about 4% course number. Multiple sections of a course of physics departments did. We feel this result (e.g., PSYCH 101a and PSYCH 101b or identifies a fundamental difference between PSYCH 101–1 and PSYCH 101–2) or affiliated psychology and physics departments in their laboratory sections of a course (e.g., PHYSICS respective commitments to instruction in intro- 101 and PHYSICS 101 Lab) were not counted ductory level general methodology. as separate courses. The number of methods courses in each department was then tallied, Discussion with a “methods” course defined as any listing that had “method” or “methodology” in its The results of Study 1 showed that psychol- name. Words such as apparatus, techniques, ogy offered more methods courses than physics laboratory, procedures, instrumentation, exper- and in that sense showed a greater commitment imental design, or research (without the word than physics to logical positivism’s emphasis on “method”), were not counted as methods methodology. More specifically, psychology courses. We then compared the frequency of was more committed to the concept of general methods courses in physics and in psychology methodology that is free of constraints by spe- departments in the sampled liberal arts colleges cific theory or content; that is, to method that is and universities. not “theory laden” (Hanson, 1958). Therefore, Second, we examined those courses that met we interpret Study 1 to show that psychology the above criteria to determine how many de- has in this sense a stronger commitment to partments offered general, introductory level logical positivism than physics does, because it methods courses. Advanced courses (with “ad- was logical positivism that argued for theory- vanced” or “graduate” in their names) were free, perhaps even universally applicable, meth- excluded from this analysis, as were courses for odology. Both psychology and physics had specific applications and/or content (e.g., statis- methods courses that applied to specific areas of tical or mathematical methods, theoretical research, but psychology had a far greater num- methods, methods for quantum physics, meth- ber that were designed to be generally applica- INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 31

Table 1 Institutions in the Sample for Study 1 and Course Data Total # # Psych. Had general Total # # Physics Had general intro. psych. “methods” intro. psych. physics “methods” physics methods Institution courses courses methods course? courses courses course? Amherst College 35 0 — 19 1 — Bowdoin College 30 0 — 25 3 — Brown Univ. 65 4 Yes 48 1 — Bryn Mawr College 28 2 — 22 1 — California Inst. of Technology 15 0 — 50 2 — Carlton College 48 7 — 35 0 — Carnegie Mellon Univ. 46 3 — 47 1 — Claremont McKenna College 41 2 Yes 18 1 — Colby College 34 2 Yes 22 0 — Colgate Univ. 37 3 Yes 20 1 — Colorado College 20 0 — 20 0 — Columbia Univ. 100 1 — 30 1 — Cornell Univ. 117 2 Yes 80 1 — Dartmouth College 52 1 Yes 63 3 — Davidson College 33 0 — 24 1 — Duke Univ. 125 8 Yes 47 1 — Emory Univ. 58 1 — 38 1 — Georgetown Univ. 41 2 — 55 3 — Grinnell College 22 1 Yes 18 0 — Hamilton College 44 2 Yes 28 0 — Harvard Univ. 124 4 Yes 94 0 — — 1 31ءءءءHarvey Mudd College Haverford College 36 1 — 32 0 — Johns Hopkins Univ. 84 2 — 98 4 — — 0 94 ء ء ءءMassachusetts Inst. of Technology Middlebury College 42 1 Yes 22 0 — Mount Holyoke College 39 2 Yes 19 2 — Northwestern Univ. 42 2 Yes 22 0 — Oberlin College 37 7 Yes 25 0 — Pomona College 32 1 Yes 22 0 — Princeton Univ. 38 2 — 36 2 — Rice University 104 7 Yes 67 3 — Scripps College 35 1 — 23 1 — Smith College 45 3 — 23 2 — Stanford Univ. 140 9 Yes 100 1 — Swarthmore College 45 1 — 38 1 — United States Military Academy 51 1 Yes 11 0 — — 2 24 ء ءء †ءUnited States Naval Academy Univ. of California - Berkeley 118 2 — 82 0 — Univ. of California - Los Angeles 280 17 Yes 127 4 — Univ. of Chicago 29 2 Yes 37 2 — Univ. of Pennsylvania 107 1 — 68 1 — Univ. of Notre Dame 124 3 Yes 63 5 — Univ. of Virginia 52 2 Yes 43 0 — Vanderbilt Univ. 35 1 — 32 1 Yes Vassar College 39 6 — 21 2 — Washington and Lee Univ.† 43 0 — 28 2 — Washington Univ. in St. Louis 105 5 Yes 109 8 Yes Wellesley College 47 8 — 24 0 — Wesleyan Univ. 84 8 — 66 0 — Williams College 38 0 — 23 1 — Yale Univ. 58 5 — 38 2 — Totals 3044 145 23/49 2251 69 2/52 † indicates no online course catalog available; course data extracted from departmental website. — indicates no course .indicates no qualifying Psychology department ء .satisfied the column heading 32 COSTA AND SHIMP ble. These data empirically supported the an electronic search of the University of Utah’s metatheoretical views expressed by Bayer J. Willard Marriott Library general catalog. We (2004), Fuller (2003), Koch (1992a), and oth- conducted the search on April 6, 2009 and lim- ers, and encouraged us to examine further the ited it to entries from 1990 to the present. It nature of this commitment by psychology to should be noted that newer editions of sampled logical positivism’s notion of methodology. We texts may have been published since our search. next examined the textbooks written for these The University of Utah library electronic general, introductory courses in methodology. search algorithms yielded nearly identical total numbers of methods texts for psychology and Study 2: Methods Texts physics: 238 in psychology and 235 in physics. We next narrowed the search to include only Study 1 showed that psychology emphasizes introductory level, general methods textbooks methodology in its curricula, especially a gen- by examining the title of each entry. Those texts eral, theory-free methodology, more than phys- having “handbook” in their title and those that ics. We speculated that psychology does so to were edited compilations of contributions on conform to a vision of the scientific method that specialized or advanced topics were excluded it inherited largely from logical positivism. because they were not written for beginning Study 2 further examined this commitment to students. Guides for teachers of methods classes logical positivism by evaluating the textbooks were also excluded. If a book had more than one used in these introductory general methods edition, we used only the latest. courses. This examination is important because For psychology, if a title contained the words textbooks (along with instructors) are authori- “method,” “methods,” or “methodology,” and ties that pass the knowledge base of a discipline “psychology,” “psychological,” “behavior,” or on to undergraduate students. We asked “behavioral,” it was accepted for further scru- whether psychology methods textbooks tended tiny. Specialized books with “clinical,” “devel- to use concepts and terminology from logical opmental,” “experimental,” “social psychol- positivism and whether any such tendency ogy,” and so on in their titles were excluded; would be greater than in physics, where the only books aimed at a general introductory stu- residual impact of logical positivism seemed dent audience were included. Specialized books smaller, based on opinions of Fuller, Koch, and that were exclusively either entirely quantitative others, and the empirical results of Study 1. or qualitative in nature were excluded but if a Should these texts present methodology from title included both, it was judged to be suffi- the perspective of a problematic philosophy of ciently general to be included, provided it met science, logical positivism, without acknowl- the above criteria. In summary, we included edging doing so, they would exemplify Fleck’s introductory texts in general psychological re- (1935/1979) notion of “textbook science” that search methods and excluded specialized and does not reveal the philosophical commitments more advanced texts. or assumptions that the texts make, and of Ten textbooks from the total of the 238 li- Kuhn’s notion of science texts as “tourist bro- brary search results that met these above criteria chures” that do not acknowledge the implicit were found. We considered the possibility that marketing aspect of their material. Textbooks the Web search might not identify all texts might tend to suppress the contradiction that satisfying our criteria and therefore examined psychological science derives from the use of a the stacks in the Marriott Library where the 10 scientific method that itself does not derive identified by the search were located and from, or is not justified by, empirical results. found 9 more texts that met our criteria as Therefore, we also examined whether a com- described above and which upon examination mitment to logical positivism might be only proved to be conventional introductory general implicit. psychological methods textbooks. As with the electronic search, if multiple editions of a text Method were found in the library’s stacks, we only sampled the most recent available edition. We Sample. We used the search terms “meth- were not able to ascertain, after consultation ods of psychology” and “methods of physics” in with relevant library staff, why the library’s INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 33 search algorithms did not identify these addi- ics texts whose titles involved only “statistical,” tional texts. Table 2 displays the 19 psychology “mathematical,” or “computational” methods. methods textbooks in our sample. Similarly, specialized texts in methods of mag- For physics, we used criteria analogous to netism, nuclear physics, theoretical physics, those described above for psychology. If a title quantum physics, and so on were excluded be- had “method,” “methods,” or “methodology,” cause they were not general, introductory text- and “physics,” or “physical science” it was ac- books. No physics text satisfying the above cepted for further scrutiny. Just as we excluded criteria was identified via the Web search. psychology texts whose titles involved only Moreover, as in the case of psychology texts, “statistical methods,” rather than general re- we examined the stacks for physics texts that search methods, we excluded specialized phys- would satisfy our criteria and found no text

Table 2 Textbooks in the Sample for Study 2 Location # Author(s)/editor(s) Year Title Publisher identified 1 Breakwell, G. M., 2006 Research methods in psychology. Sage Web search Hammond, S., Fife- Schaw, C., & Smith, J. A. 2 Cherulnik, P. D. 2001 Methods for behavioral research: a Sage Web search systematic approach. 3 Coolican, H. 2004 Research methods and statistics in Hodder & Stoughton Stacks psychology. 4 Coolican, H. 2006 Introduction to research methods in Hodder Education Stacks psychology. 5 Cozby, P. C. 2009 Methods in behavioral research. McGraw Hill Web search 6 Davis, S. F. & Smith, 2005 An introduction to statistics and research Pearson Stacks R. A. methods: Becoming a psychological detective. 7 Dyer, C. 2006 Research in psychology: A practical Blackwell Web search guide to methods and statistics. 8 Elmes, D. G., Kantowitz, 1999 Research methods in psychology. Brooks/Cole Stacks B. H., & Roediger, H. L. 9 Evans, A. N. & Rooney, 2008 Methods in psychological research. Sage Web search B. J. 10 Goodwin, C. J. 2008 Research in psychology: Methods and Wiley Web search design. 11 Haslam, S. A. & 1998 Doing psychology: An introduction to Sage Stacks McGarty, C. research methodology and statistics. 12 Heiman, G. W. 1995 Research methods in psychology. Houghton Mifflin Web search 13 Langdridge, D. 2004 Introduction to research methods and Pearson Stacks data analysis in psychology. 14 Leary, M. R. 2004 Introduction to behavioral research Pearson Stacks methods. 15 Liebert, R. M. & Liebert, 1995 Science and behavior: An introduction to Prentice Hall Web search L. L. methods of psychological research. 16 Mook, D. G. 2001 Psychological research: The ideas Norton Stacks behind the methods. 17 Ray, W. J. 2000 Methods toward a science of behavior Wadsworth Web search and experience. 18 Sarafino, E. P. 2005 Research methods: Using processes and Pearson Web search procedures of science to understand behavior. 19 Shaughnessy, J. J. & 1994 Research methods in psychology. McGraw-Hill Stacks Zechmeister, E. B. 34 COSTA AND SHIMP written after our cutpoint date (1990) that had a failed to explicitly state why it demonstrated title that met our criteria. This absence of any human fallibility. That is, even if it gave exam- such text corresponded to our finding in Study 1 ples of experimenter bias, biased observation, that only about 4% of physics departments in and so on, but did not state why it did so, we our sample offered the kind of general introduc- still categorized the text as one that used scien- tory methods course for which such a text tific literature to motivate the methods it de- would be relevant. scribed based on the assumption that the im- Procedure. According to our analysis de- plicit message was that humans are fallible scribed above, some psychology methods texts without the use of scientific methodology. could be expected to be written from the per- spective of logical positivism and therefore Results should use and emphasize its terminology in the form of operationalism, the logic of hypothesis For ease of discussion, we describe the re- testing, and how the scientific method identifies sults using the numbers for the textbooks as- causal relations. These terms should be suffi- signed in Table 2 instead of referencing text- ciently emphasized that they should appear in book titles or author names. The introductory the textbooks’ Indexes. Therefore, we searched psychology methods texts that support a need the Indexes or Subject Indexes of texts in our for scientific methods by citing empirical data to sample for the terms “operational definition” or demonstrate human fallibility in cognitive pro- “operationalism,” for “hypothesis” or “hypoth- cessing were: 1, 3, 5, 7, 8, 9, 11, 12, 13, 15, 16, esis testing,” and for “causal relation” or “cau- 17, 18, and 19. Only the texts numbered 2, 4, 6, sation versus correlation.” 10, and 14 did not. Thus, 14 textbooks sup- According to the textbook science and tourist ported a need for scientific methodology by brochure views of texts, it could be hypothe- citing empirical data on human fallibility and 5 sized that some texts might not acknowledge the did not. In short, it was common for introduc- philosophical origins of these terms. That is, the tory psychology methods texts to cite empirical Indexes of some texts should show usage of literature to show human fallibility in process- logical positivism’s terminology and concepts ing a wide variety of information. Recall that but not identify logical positivism as the origin. these are the results to be expected if introduc- Furthermore, some texts might be expected to tory psychology methods texts appeal to empir- adopt, either implicitly or explicitly, logical ical evidence to show that basic human cogni- positivism’s position that its use would over- tive activities such as seeing, deciding, and come human fallibilities. That is, some texts remembering are fallible, and that human falli- might propose operational definitions, the logic bility therefore must be overcome by objective of hypothesis testing, and ways to identify scientific methodology if psychology is to be causal relations as ways to overcome limitations considered a legitimate science. in human perception such as illusory correla- We next searched the Index or a separate tions (e.g., Chapman, 1967, 1969; Redelmeier Subject Index if available of each of the 19 texts & Tversky, 1996), or biases in human judgment for positivistic terminology defined above; that (including use of statistical information and is, we searched for “operational definition” or judgment in the face of social pressure or au- “operationalism,” “hypothesis,” or “hypothesis thority), human memory (including eyewitness testing,” and “causal relation” or “causation memory; e.g., Loftus & Hoffman, 1989), or versus correlation.” The results were striking; top-down or knowledge-driven processing such 18 of the 19 texts referenced all three terms, and as confirmation bias (e.g., Wason, 1960, 1968) the remaining text referenced two of the three. and of cultural bias. Therefore, we defined a Finally, we asked whether these commit- category of texts that appealed to scientific lit- ments to positivism were acknowledged in the erature on these limitations. These texts encour- texts that displayed them. We examined the 19 aged students to see why scientific methods are texts that used positivistic terminology, to de- needed because of the mistakes that can be termine whether positivism was explicitly cited made if scientific methods are not used. We and specifically the 14 texts (1, 3, 5, 7, 8, 9, categorized a text as motivating scientific meth- 11, 12, 13, 15, 16, 17, 18, and 19) that cited ods by demonstrating human fallibility even if it empirical literature on human fallibility to mo- INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 35 tivate the need for scientific methodology. Of viewing grant proposals, and advising students the 19 texts’ Indexes, seven (1, 2, 3, 4, 7, 13, what to read and what not to read. An implicit and 15) contained the word or words “positiv- commitment to logical positivism specifically ism,” “post positivism,” “logical empiricism,” encourages a belief that empirical methodology and/or ”logical positivism“ in their Indexes. is more fundamental to establishing a discipline Thus, overall, 7 of 19 referenced positivism in as a science than is speculative theory, integra- one way or another. Of the 14 texts that used tive theory, or conceptual analyses of standard human fallibility to justify scientific methods to terms, and encourages a belief in empirical facts overcome them, 1, 3, 7, 13, and 15 referenced that are independent of historical, societal, positivism. Thus, only 5 of 14 that appealed to or theoretical context and a belief in the possi- empirical data on human fallibility to justify a bility of objective, theory-free observation (see need for scientific methodology explicitly ac- Hanson, 1958; Kuhn, 1996). Such a belief can knowledged positivism in their Indexes. In sum- produce optimistic statements about the efficacy mary, virtually all texts depended on logical of scientific methodology to overcome ordinary positivism by using its terminology and con- human cognitive and social fallibilities, as in the cepts, yet fewer than half made this dependency following: “The good news is that despite the explicit. subtlety of many of the problems we have iden- tified. . . all can be addressed (in many cases Discussion quite easily) by appropriate experimental de- sign” (Haslam & McGarty, 1998, p. 9). Most introductory psychological methods However, if empirical data were used as the texts in our sample used language and concepts standard by which knowledge is judged to be compatible with logical positivism. By itself, reliable and objective, then one might come to this result would not necessarily imply a com- the opposite conclusion about the scientific mitment to that philosophy because some texts method as a means to overcome human fallibil- also used these same concepts and terminology ity, such as in the following: “Like any product and explicitly described its shortcomings and manufactured by the social labor of imperfect described alternatives to it. Agnew and Pyke organisms, science has a checkered track re- (2007), Breakwell, Hammond, Fife-Schaw, and Smith (2006), and Mertens (2005), for example, cord. . . The same checkered track records can described historical and philosophical contexts be observed in the histories of any of our other in some detail such that students could see how hallowed social institutions, from religion to the development of empirical methods de- athletics to art” (Klee, 1997, p. 182). That is, the pended on the historical and philosophical con- actual empirical record several of the surveyed texts in which they were derived, as well as on methods texts describe is that humans are falli- specific experimental requirements. ble in their cognitive and social processes. But We suggest that it constitutes an implicit the steps of the scientific method involve these commitment to logical positivism when a text very same processes. Study 2 therefore demon- used both concepts and terminology of positiv- strates that contemporary psychology methods ism and did not explicitly acknowledge doing textbooks tend to maintain Peirce’s (1877/1934) so. Why does it matter if a psychology methods contradiction in their claims that scientific text is implicitly committed to logical positiv- methodology can overcome human fallibility in ism? It matters because a commitment to a cognitive and social processes even though the philosophy of science generally colors what one scientific method itself involves and often even believes science is, and that in turn impacts relies upon those very same cognitive and social what one believes is good science, what is ap- processes. We recommend Winston (2004) for propriate scientific methodology, and what is an especially telling individual case history in the proper relation between science, truth, and how one prestigious, authoritative faculty im- society (Feist, 2006; Fuller, 2003; Koch, 1992a, plicitly advocated on behalf of logical positiv- 1992b; Koch & Leary, 1992; Lacey, 2005; ism in the 1930s. Our broader results show how Longino, 1990; Ravetz, 1971). Therefore, it af- this kind of advocacy still rests today, as it has fects the process of doing science, including for over a hundred years, on logic, philosophy, peer review of journal articles, writing and re- and authority. 36 COSTA AND SHIMP

Our personal experiences agree with these effectiveness, experimenter bias, the perception historical and philosophical analyses of Klee of color, or to the analysis of data from exper- (1997), Koch (1992a, 1992b), Toulmin and iments in human categorization. This instruc- Leary (1992), and others: behavioral scientists tion that presents methodology as though it can frequently be seen to display positivistic were independent of theory or content is com- optimism about the power of scientific method mon in introductory level psychology. A prin- without showing a great deal of knowledge cipal goal of logical positivism was to advocate about the role philosophy of science plays in for precisely this kind of methodology that it their daily practice of psychological science. In claimed was objective because it was indepen- our experience, such scientists ironically resist dent of specific content or theoretical assump- learning about philosophy of science in the be- tions (Carnap, 1936, 1937; Hardcastle & Rich- lief that it demands that they conform to a vision ardson, 2003; Stevens, 1935). Our results of science developed by philosophers who they showed very little evidence that instruction in believe do not understand how science actually physics advocates for this same sharp contrast works. This belief leads these scientists un- between general methodology and theory. In knowingly and ironically to accept and to put this way, current instruction in psychological into practice a philosophy of science that is one methodology tends, more than instruction in which many philosophers now see primarily as physics methodology, to display a commitment outdated. We speculate that these scientists to logical positivism’s advocacy of a “theory have succumbed to the views of Kuhn (1996) free” methodology as the key to turning a dis- and Fleck (1935/1979) who described a text- cipline into an objective science (Feyerabend, book vision of science as one having the allure 1975; Koch, 1992a; Kuhn, 1996; Lacey, 2005; of being more coherent, logically convincing, Laudan, 1990; Longino, 1990; Slife, Reber, & and generally appealing, than the highly com- Richardson, 2004; Toulmin & Leary, 1992). plex and sometimes contradictory literatures upon which the textbooks are based. Textbook General Discussion science and texts viewed as tourist brochures have been used in explanations of the self- We empirically examined whether the philos- propagating nature of “normal science,” the ophy of logical positivism impacts instruction mode of science that, according to Kuhn (1996), in psychological research methods, and if so, most scientists practice most of the time; he how. Before summarizing our results and de- speculated that normal science tended to protect scribing what they imply for how psychology itself from potentially threatening contradic- sees itself as a science, we would like to make tions, unexplained complications, and revolu- explicit some qualifications imposed by our tionary ideas simply by largely ignoring them, specific empirical method. First, we used archi- thereby preserving the current paradigm. val research methods by examining published Study 2 suggests that textbook science may literature, in our case, textbooks and Web-based explain, at least in part, why logical positivism course catalogs. Such descriptive methods can serves as the philosophical foundation for em- give important insights (Merton, 1938/1970; pirical methodology in psychology more than it Price, 1961), but they lack logical positivism’s does in physics. While we found that the total standards for determining causal relations numbers of methods texts in psychology and through the use of experimental control. We are physics were very similar, we found no intro- sufficiently persuaded by empirical, philosoph- ductory physics general methods texts corre- ical, and historical scholarship that identify sponding to the introductory psychology gen- problems with these standards (Guba & Lin- eral methods texts in our survey. In our sample, coln, 2005; Hanson, 1958; Koch, 1964, 1992a; recommended psychological methods were not Livingston, 2003; Rozin, 2009; Slife, Reber, & qualified in terms of the empirical content or Richardson, 2004; Winston, 2004) to be untrou- psychological theories to which they were said bled by this lack, but we wished to make this to apply. For example, one does not read that a feature explicit. double-blind method, random sampling, a t test, Second, our samples of academic institutions or the analysis of variance, is applicable only to and of texts are not random so again, inferential specific content such as peer pressure, teaching statistics are not applicable here. We hope our INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 37 results are sufficiently clear and interesting that American South for cultural reasons. We think they speak for themselves, especially in the these varying opinions on the role of culture in sense of inducing in a reader’s mind the ques- science suggest that it would be important for tion of whether our results in fact do apply at his future empirical research on how philosophy or her institution and in the textbooks he or she impacts instruction in psychological methodol- uses. If the institution is American, the present ogy to examine how the methods they describe data suggest that they likely do. and recommend depend on cultural traditions Third, whether our results apply to curricula and religious beliefs (Merton, 1938/1970). The or texts designed for students in other cultures assumption that science and methodology are remains an open question for future research to culture- and gender-free is similar to logical determine. We think this qualification points to positivism’s notion that methodology is “the- important future extensions of our archival ory-free.” In contrast, the emerging science of work because the positivistic methodology texts science redefines this assumption as an empiri- in our sample sometimes recommended the uni- cal question. We suggest empirical methods be versal use of their methods; some texts see their used to understand and address such questions methods as not only theory free, but culture free about science, instead of merely presenting ar- as well. Consider the following view: guments derived from tenacity, logic, or author- ity, to justify such claims about science to stu- Scientific statements are rational in the sense that they are the result of an explicit process of reasoning that dents. follows the rules of logic and can thus, in principle, be A critical issue facing not just science but followed by anyone. This means that scientific knowl- society as a whole is how to coordinate what is edge is trans-cultural—a scientific statement carries the seen as objective science with human values. same information in any part of the world (Dyer, 2006, Science’s customary discourse about one phys- p. 3). ical reality and hence one scientific truth, and Note no data are provided to support this about objectivity and rigorous methodology that claim, only authority, logic, and philosophy. is impervious to human fallibilities, sometimes The philosophy is the same one which we have seems at odds with society’s pressing need to shown persists in psychology methods texts and adopt a more tolerant attitude about diverse curricula, logical positivism, in its guise as a views that may seem fundamentally incompat- rational, logical, universally objective authority. ible (Shimp, 1999). We hope our present results From this perspective, cultural and gender dif- suggest how archival and other empirical re- ferences are biases that can be removed from search methods might facilitate future research objective scientific knowledge the way other to make cultural and scientific beliefs and atti- biases and confounds can be removed, by care- tudes that are now only implicit more nearly ful logical, sophisticated statistical, or other explicit, so that their social roles in science and methodological means. Being biases, they society may be better understood. should be removed. In contrast to this position, Fourth, we do not provide an historical anal- others see cultural differences as critical to un- ysis of the impact of philosophy on the devel- derstanding what science is; cultures constrain opment of instruction in psychological methods. how scientists perceive methods and concepts, FitzGerald (1979) developed an illuminating how they frame them, and therefore can rede- analysis of the impact of the history of Ameri- fine them. We recommend Fuller (2003, pp. can culture on the development of American 106–136) on the view from Islam and Japan, history texts, and Fuller (1997) discussed, with- Agnew and Pike (2007) who include a chapter out supporting data, however, the historical de- entitled “Sex and Science” (pp. 406–431) in velopment of science texts in general. A corre- their methods textbook, and Livingstone (2003, sponding analysis of the specific impact of the p. 1) who wrote that “Scientific knowledge is history of philosophy on the history of instruc- made in a lot of different places. Does it matter tion in psychological methods would be, we where?” In our opinion, the methods texts we think, of corresponding importance. The present examined too often imply that the answer is no. work gives only a contemporary snap shot of Livingstone wrote that the answer is yes, and the impact of the history of philosophy on in- described how Darwinism was interpreted and struction in psychological methods. We hope received differently in New Zealand versus the that our methods may be expanded to facilitate 38 COSTA AND SHIMP an analysis of the historical development of by discovering empirical methods that are the- texts in psychological methodology, and of the ory free. However, despite conforming in this impact of philosophy, religion, gender, and functional way to logical positivism, most did other cultural influences on them. For the mo- not explicitly acknowledge doing so. ment, however, our more limited goal was sim- Why might psychology persist in presenting ply to show that an outdated philosophy of itself in curricula and texts in a manner consis- science does influence contemporary instruction tent with a philosophy of science seen by many in psychological research methods and that it is as problematic, often without acknowledging common for these texts not to acknowledge any doing so? We suggest that an answer lies in such influence at all. Fleck’s (1935/1979) proposal that science texts With these qualifications made explicit, let us present science from what he called a “textbook now summarize the overall results. Study 1 sur- science” perspective and in Kuhn’s (1996) sim- veyed course offerings in psychology and phys- ilar proposal that they present science in a man- ics departments in top colleges and universities ner that resembles how tourist brochures present and found that there were more psychology tourist destinations. There are several reasons methods courses than physics methods courses, why psychology might use “textbook science.” relative to the respective total numbers of For example, consider that the APA recently courses. Study 1 showed further that many more developed strategic plan has one of its three psychology than physics departments in our “top priorities” being “increasing recognition of sample offered introductory general methods psychology as a science” (Anderson, 2009, p. courses. Correspondingly, Study 2 showed that 9). Consider also the following from a recent there were more introductory psychology gen- article by the current president of APA (Bray, eral methods texts than introductory physics 2010, p. 365), “STEM (science, technology, general methods texts, and the psychology engineering, and mathematics) is the term used methods texts tended to promote language and to refer to basic science disciplines. Although concepts derived from logical positivism, a phi- psychology is a STEM discipline and contrib- losophy of science that arose in the 1920s that utes to STEM education in other science disci- advocated the use of concepts and terminology plines, it is not always considered a core STEM of operational definitions, hypothesis testing, discipline.” Anderson (2009) and Bray (2010) and a particular view of causal relations. view psychology as a science not adequately Study 2 also showed that this commitment is recognized as such. We are concerned that often presented only implicitly, so that students APAs efforts to promote greater recognition of are unlikely to see it and critique it. We inter- psychology’s scientific legitimacy may resem- preted all these results in terms of there being a ble and contribute to how psychology’s curri- greater tendency in psychology than in physics cula and methods textbooks often promote psy- to emphasize features of logical positivism, es- chology’s legitimacy, that is, by tenaciously pecially its emphasis on methodology as the key emphasizing that psychology is scientific be- to scientific legitimacy. So far as we know, cause its methods are scientific. We are con- these are the first empirical studies to show the cerned that these public relations efforts will not greater impact of logical positivism on instruc- show any greater awareness than do present tion in methodology in psychology than in psychology methods texts that psychology’s physics. mainstream methodologies may conform to Furthermore, Study 2 found that several of those compatible with the outdated and flawed the psychology texts in our sample advised us- philosophy of logical positivism. ing scientific methods in a manner consistent Our results support the suspicions of others with positivism by empirically demonstrating (Bayer, 2004; Fuller, 1997, 2003; Koch, 1992a) human fallibilities that intrude into methods of that psychology as a discipline believes in the investigation other than the scientific method. importance of legitimizing itself in terms of its Virtually all of the texts in our sample used key methods to a greater degree than physics would. terms, essentially defining terms, of logical pos- We are therefore hopeful that when APA or itivism, a philosophy of science that argued that other authoritative organizations in the disci- the scientific method, correctly employed, en- pline advertise that psychology is a science be- abled scientists to overcome these fallibilities cause its methods are scientific, they will make INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 39 explicit the dependence of their claims on the ambiguity in the emerging science of science, fragile contemporary status of logical positiv- but we do not think that ambiguity should stop ism, and that logical positivism itself was de- empirical and theoretical work. Instead, we veloped through the use not of empirical science think it should be acknowledged and empiri- but through philosophy, logic, and authority cally examined, as our two studies do here. The (Bridgman, 1927; Carnap, 1936, 1937; Pratt, emerging science of science is an iterative pro- 1939; Stevens, 1935). A minority of authors of cess according to which research on human psychological methods texts already acknowl- fallibilities, for example, is used to inform how edges these issues, and we hope their approach psychological research, including research on becomes more widely adopted. Otherwise, if fallibilities, is conducted. Irony arises when un- APA advocates for the legitimacy of psycho- necessary logical, philosophical, or authoritar- logical science the way our data suggest psy- ian claims are made, or implied, such as that chology’s curricula and methods textbooks do, science involves methods that overcome human then we predict other more established sciences, fallibilities to achieve impersonal objectivity, and hence ultimately the public too, could re- culture-free truths, observation that is not theory main skeptical. laden, methods that apply across any possible We see an opportunity for APA and other theoretical context, and so on. These are the psychological professional organizations to be- claims that seem to us to be inherent, or even come more impactful in influencing national explicit, in introductory psychological methods science policy by promoting psychology’s role texts. We make no such claims. We do claim, in the development of a science of science, nevertheless, that empirical methods provide rather than by implicitly promoting logical pos- tools by which our understanding of what it itivism. Much psychological work is now being means to develop and use empirical methods directed to psychology’s role in this new disci- can be informed. One need look no farther than pline (e.g., Feist, 2006, 2008; Klahr & Nigam, contemporary research on the science of science 2004; Newcombe, Ambady, Eccles, Gomez, to see evidence in favor of this argument. To Klahr, & Mix, 2009) and we hope that psychol- results like our own archival data on how phi- ogy curricula, methods texts, and professional losophy impacts instruction on method, we organizations will in the future give more atten- would add results of research on peer review, tion to this work in how they advocate for confirmation bias, experimenter bias, theory psychology as a science. laden observation, irrational aspects of decision There are potential contradictions and ironies making, irrational neglect of base rate informa- in phrases such as “the science of science.” This tion, and so on. We anticipate that future work may be especially the case here because the along all these lines will show the utility of psychology of science suggests that the scien- cross cultural psychological science of the sci- tific method may be flawed, yet we are advo- ence of psychology, given the certainty that cating the use of empirical methods to inform non-Western science will play an increasingly our understanding of how science is practiced. important role in determining the nature of sci- This irony clearly warrants some discussion. ence. In short, at the present time there does not Analogous ironies involving reflexive state- seem to be any irony involved in the use of ments have been known since philosophy’s in- science, broadly conceived, to study science ception. For contemporary examples, consider itself, in part because so much remains to be Kuhn’s (1996) argument that all scientific par- known about what science is, and in part be- adigms are context dependent, except for his cause the emerging science of science seems scientific paradigm for the development of sci- more motivated to explicitly describe the qual- entific paradigms. This irony has nevertheless ifications that pertain to its methods, so that they not prevented Kuhn’s work from being useful. may be more easily critically examined. It Also, Shimp (2001) argued for the reflexive would go a long way to remove the irony that application of dynamic, computational theories can be seen in Peirce’s work from some intro- of behavior to the behavior of behaviorists ductory psychological methods texts if they ex- themselves, and yet he argued that application plicitly revealed the historical, philosophical, might even be essential to the understanding of and cultural assumptions behind their methods, behavioral science. So, we concede there is and revealed also how empirical investigations 40 COSTA AND SHIMP of those assumptions might clarify the nature of did not require a student to suppress interest in the scientific method and thereby reveal ways to theorists such as Freud, Luria, Piaget, and Ge- improve the methods of psychological science stalt psychologists. Several of the methods texts (see also Feist, 2006, 2008; Feyerabend, 1975; in our sample referred to work by highly influ- Fleck, 1935/1979; Hanson, 1958; Klahr & ential psychologists such as Freud only as Nigam, 2004; Proctor & Capaldi, 2006; pseudo science relying on improper methodol- Mertens, 2005; and Slife, Reber, & Richardson, ogy. We think it is pedagogically problematic to 2004). describe the work of some of the most influen- In our sample of texts, less than half of those tial psychological theorists only as examples of that presented scientific methods in terms of a how not to do science. This conveys the posi- positivistic perspective actually acknowledged tivistic message that method is vastly more im- doing so. Therefore, many students will not portant than speculative theory, but leaves un- learn that their texts present science only from explained or even unacknowledged why these the perspective of logical positivism and most psychologists have been so influential. That is, will not learn about the philosophical origins of it leaves the role of speculative theory in science the material in their methodology texts or the unmentioned or dismissed. We feel that ac- subsequent philosophical debates that have knowledging the impact of these and other like made positivism seem problematic. Students scholars might accomplish three things. First, therefore are at risk for assimilating a philo- because they have influenced how professional sophical perspective without realizing they do scholars view psychology, they presumably so. Instruction in methods in this way again would be of interest to students as well. Second, corresponds to Fleck’s (1935/1979) and Kuhn’s acknowledging how influence depends on the- (1996) descriptions of science texts as market- ory as well as on rigorous method would let ing tools. We see this as problematic because students more clearly understand how logical methods texts claim objectivity to be their goal, positivism’s perspective on the primacy of yet many of them fail to describe how this claim method does not explain how science actually depends on unstated and unevaluated assump- works. According to Kuhn (1996) and Feyera- tions, or how there might be a difference be- bend (1975), for example, as theory changes so tween science as they describe it and science as must method. In that sense, theory has primacy it is actually practiced. The lack of awareness over method, as opposed to method having pri- with which this intellectual attitude appears to macy or method being theory-free. Finally, we be acquired suggests to us it is likely to be feel that speculative theory might drive future passed on from academic generation to aca- empirical work, as is often the case in the field demic generation without participants under- of theoretical physics and, according to Lakatos standing their own assumptions and without (1978), as it does in “progressive” research pro- understanding the possibility of empirically ex- grams in general. amining them. We see little or no evidence that promoting We also speculate that introductory psycho- the scientific legitimacy of psychology by logical methods courses might be more appeal- means of “textbook science” and “tourist bro- ing to students than they sometimes now are if chures” encourages students to form a healthy, their implicit philosophical content were made sophisticated, realistic and critical attitude to- more explicit. We suggest that one reason why ward the role of methodology in psychological instructors often face a difficult challenge in science. At the very least, we think methods courses on methodology is that the implicit pre- texts that present their content from the perspec- sentation of psychology as a science from the tive of a philosophy of science should explicitly perspective of logical positivism requires a acknowledge that they do so. We see such a sharp line between rigorous method and specu- “truth in advertising” statement as serving mul- lative theory to be drawn, and that in turn, tiple purposes, including reminding instructors requires them to omit the work of many of the as well as students that philosophy still plays a most influential psychologists in history, about crucial role in psychology and encouraging stu- whose work many students are often eager to dents to critically examine the philosophical learn. That is, student interest in rigorous meth- and methodological assumptions of a discipline odology might be improved if such an interest to better understand the nature of science. We INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 41 understand the complex costs and benefits of Chapman, L. J. (1967). Illusory correlation in obser- introducing students in introductory general vational report. Journal of Verbal Learning and methods courses to complex, abstract philo- Verbal Behavior, 5, 151–155. sophical issues, and we sympathize with the Chapman, L. J. (1969). Illusory correlation as an pedagogical goals of authors who have wished obstacle to the use of valid psychodiagnostic signs. to bypass these issues on behalf of efficiency Journal of Abnormal Psychology, 4, 271–280. Cohen, R. S. (1981). Herbert Feigl: Inquiries and and other practical constraints. We think, how- provocations, selected writings 1929–1974. Dor- ever, that the literatures on the teaching of psy- drecht, Netherlands: Reidel. chology, on the metascience of the psychology Davis, S. F., & Buskist, W. (2002). The teaching of of science, and on how to present psychology to psychology: Essays in honor of W. J. McKeachie the public (Davis & Buskist, 2002; Koch, and Charles L. Brewer. Mahwah, NJ: Erlbaum. 1992a; Proctor & Capaldi, 2006; Saville, 2008; Diener, E. (2009). Introduction to the special issue: Smith, Harre, & Langenhove, 1995; Toulmin & Improving psychological science. Perspectives on Leary, 1992) suggest numerous constructive al- Psychological Science, 4, 1. ternatives to “textbook science” or “tourist bro- Dyer, C. (2006). Research in psychology: A practical chures,” that would better serve both students guide to methods and statistics. Oxford, England: and scientists. Blackwell. Estes, W. K. (1959). The statistical approach to learn- ing theory. In S. Koch (Ed.), Psychology: A study of a science (Vol. 2, pp. 380–491). New York: References McGraw-Hill. Faust, D. (1984). The limits of scientific reasoning. Agnew, N. McK., & Pyke, S. (2007). The science Minneapolis, MN: University of Minnesota Press. game: An introduction to research in the behav- Fechner, G. (1860/1966). Elements of psychophysics. ioral and social sciences (7th ed.). Oxford, Eng- New York: Holt, Rinehart and Winston. land: Oxford University Press. Feigl, F. (1970). The “orthodox” view of theories: Anderson, N. B. (2009). Our first strategic plan. Remarks in defense as well as critique. In H. Feigl Monitor on Psychology, 40, 9. & G. Maxwell (General Eds.) & M. Radner & S. Bayer, B. M. (2004). On cultural history as transfor- Winokur (Vol. Eds.), MN Studies in the Philoso- mation – or, what’s the matter with psychology, phy of Science: Vol. IV. Analyses of theories and anyway? In A. C. Brock, J. Louw, & W. van methods of physics and psychology (pp. 3–16). Hoorn (Eds.), Rediscovering the history of psy- Minneapolis, MN: University of Minnesota Press. chology: Essays inspired by the work of Kurt Dan- The psychology of science and the ziger (pp. 119–140). New York: Kluwer Academ- Feist, G. J. (2006). ic/ Plenum Press. origins of the scientific mind. New Haven, CT: Benham, B., & Shimp, C. P. (2004). Falsification in Yale University Press. social science method and theory. In K. Kempf- Feist, G. J. (2008). Psychologists of science have Leonard (Ed.), Encyclopedia of social measure- finally arrived (in Berlin). Journal of Psychology ment (Vol. 2, pp. 9–14). San Diego, CA: Aca- of Science and Technology, 1, 34–35. demic Press. Feyerabend, P. (1975). Against method (3rd ed.). Bray, J. H. (2010). APA Presidential Address: The London: Verso 1993. future of psychology practice and science. Ameri- Feyerabend, P. K., & Maxwell, G. (Eds.) (1966). can Psychologist, 65, 355–369. Mind, matter, and method: Essays in philosophy Breakwell, G. M., Hammond, S., Fife-Schaw, C., & and science in honor of Herbert Feigl. Minneap- Smith, J. A. (Eds.). (2006). Research methods in olis, MN: University of Minnesota Press. psychology (3rd ed.). London: Sage. FitzGerald, F. (1979). America revised: History Bridgman, P. W. (1927). The logic of modern phys- schoolbooks in the twentieth century. Boston: At- ics. New York: Macmillan. lantic-Little, Brown. Bridgman, R. W. (1959). The way things are. Cam- Fleck, L. (1935/1979). Genesis and development of a bridge, MA: Harvard University Press. scientific fact. Chicago: University of Chicago Capaldi, E. J., & Proctor, R. W. (1999). Contextual- Press. ism in psychological research? A critical review. Fuller, S. (1997). Science: Concepts in social Thousand Oaks, CA: Sage. thought. Minneapolis, MN: University of Minne- Carnap, R. (1936). Testability and meaning. Philos- sota Press. ophy of Science, 3, 419–471. Fuller, S. (2003). Kuhn vs. Popper: The struggle for Carnap, R. (1937). Testability and meaning. Philos- the soul of science. New York: Columbia Univer- ophy of Science, 4, 1–40. sity Press. 42 COSTA AND SHIMP

Gholson, B., Shadish, Jr., W. R., Neimeyer, R. A., & Kuhn, D., Amsel, E., & O’Loughlin, M. (1988). The Houts A. C. (Eds.). (1989). Psychology of science: development of scientific thinking skills. San Di- Contributions to metascience. Cambridge, Eng- ego, CA: Academic Press. land: Cambridge University Press. Kuhn, T. S. (1993). Logic of discovery or psychology Gopnik, A., & Schulz, L. (2007). Causal learning: of research? In J. H. Fetzer (Ed.), Foundations of Psychology, philosophy, and computation. Oxford, philosophy of science: Recent developments (pp. England: Oxford University Press. 364–380). New York: Paragon House. Gorman, M. E. (1992). Simulating science: Heuris- Kuhn, T. S. (1996). The structure of scientific revo- tics, mental models, and technoscientific thinking. lutions (3rd ed.). Chicago: University of Chicago Bloomington, IN: Indiana University Press. Press. Gorman, M. E., Tweney, R. D., Gooding, D. C., & Lacey, H. (2005). Values and objectivity in science: Kincannon, A. P. (2005). Scientific and technolog- The current controversy about transgenic crops. ical thinking. Mahwah, NJ: Erlbaum. Oxford, England: Lexington Books. Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic Lakatos, I. (1978). The methodology of scientific re- controversies, contradictions, and emerging con- search programmes: Philosophical papers (Vol. fluences. In N. K. Denzin & Y. S. Lincoln (Eds.), 1). Cambridge, England: Cambridge University The Sage handbook of qualitative research (3rd Press. ed., pp. 191–215). Thousand Oaks, CA: Sage. Laudan, L. (1990). Science and relativism: Some key Hanson, N. R. (1958). Patterns of discovery: An controversies in the philosophy of science. Chi- inquiry into the conceptual foundations of science. cago: University of Chicago Press. Cambridge, England: Cambridge University Press. Livingstone, D. N. (2003). Putting science in its Hardcastle, G. L., & Richardson, A. W. (2003). Log- place: Geographies of scientific knowledge. Chi- ical empiricism in North America. Minneapolis, cago: University of Chicago Press. MN: University of Minnesota Press. Loftus, E. F., & Hoffman, H. G. (1989). Misinfor- Hartshorne, C. & Weiss, P. (Eds.). (1877/1934). Col- mation and memory: The creation of new memo- lected papers of Charles Sanders Peirce. Cam- ries. Journal of Experimental Psychology: Gen- bridge, MA: Harvard University Press. eral, 118(1), 100–104. Haslam, S. A., & McGarty, C. (1998). Doing psy- Longino, H. (1990). Science as social knowledge; chology: An introduction to research methodology Values and objectivity in scientific inquiry. Prince- and statistics. London: Sage. ton, NJ: Princeton University Press. Hempel, C. G. (1966). Foundations of natural sci- Mahoney, M. J. (1976). Scientist as subject: The ence. Englewood Cliffs, NJ: Prentice Hall. psychological imperative. Cambridge, MA: Ball- Hull, C. L. (1943). Principles of behavior. New inger. York: Appleton-Century-Crofts. Meehl, P. E. (1970). Some methodological reflec- Klahr, D., & Nigam, M. (2004). The equivalence of tions on the difficulties of psychoanalytic research. learning paths in early science instruction: Effects In H. Feigl & G. Maxwell (General Eds.) & M. of direct instruction and discovery learning. Psy- Radner & S. Winokur (Vol. Eds.), MN Studies in chological Science, 15, 661–667. the Philosophy of Science: Vol. IV. Analyses of Klee, R. (1997). Introduction to the philosophy of theories and methods of physics and psychology science: Cutting nature at its seams. Oxford, Eng- (pp. 403–416). Minneapolis, MN: University of land: Oxford University Press. Minnesota Press. Koch, S. (1992). Afterword. In S. Koch & D. E. Mendelsohn, E., Weingart, P., & Whitley, R. (1977). Leary (Eds.), A century of psychology as science The social production of scientific knowledge. (pp. 928–950). Washington, DC: American Psy- Dordrecht-Holland: D. Reidel. chological Association. Mertens, D. M. (2005). Research and evaluation in Koch, S. (1992a). Psychology and emerging concep- education and psychology: Integrating diversity tions of knowledge. D. Finkleman & F. Kessel with quantitative, qualitative, and mixed methods (Eds.), Psychology in human context: Essays in (2nd ed.). Thousand Oaks, CA: Sage. dissidence and reconstruction (pp. 51–90). Chi- Merton, R. K. (1938/1970). Science, technology and cago: University of Chicago Press. society in seventeenth-century England. New Koch, S. (1992b). Psychology’s Bridgman versus York: Harper & Row. Bridgman’s Bridgman. D. Finkleman & F. Kessel Newcombe, N. S., Ambady, N., Eccles, J., Gomez, (Eds.), Psychology in human context: Essays in L., Klahr, D., Linn, M., Miller, K., & Mix, K. dissidence and reconstruction (pp. 366–392). Chi- (2009). Psychology’s role in mathematics and sci- cago: University of Chicago Press. ence education. American Psychologist, 64, 538– Koch, S., & Leary, D. E. (Eds.). (1992). A century of 550. psychology as science. Washington, DC: Ameri- Peirce, C. S. (1877/1934). The fixation of belief. In can Psychological Association. C. Hartshorne & P. Weiss (Eds.), Collected papers INTRODUCTORY GENERAL METHODS IN PSYCHOLOGY 43

of Charles Sanders Peirce (pp. 223–247). Cam- Slife, B. D., Reber, J. S., & Richardson, F. C. (Eds.). bridge, MA: Harvard University Press. (2004). Critical thinking about psychology: Hid- Pratt, C. C. (1939). The logic of modern psychology. den assumptions and plausible alternatives. Wash- New York: Macmillan. ington, DC: American Psychological Association. Price, D. J. (1961). Science since Babylon. New Smith, J. A., Harre, R., & Langenhove, L. V. (Eds.). Haven, CT: Yale University Press. (1995). Rethinking methods in psychology. Lon- Proctor, R. W., & Capaldi, E. J. (2006). Why science don: Sage. matters: Understanding the methods of psycholog- Smith, L. D. (1986). Behaviorism and logical posi- ical research. Malden, MA: Blackwell. tivism. Stanford, CA: Stanford University Press. Quine, W. V. (1951). Two dogmas of empiricism. Stevens, S. S. (1935). The operational definition of Philosophical Review, 60, 20–43. psychological concepts. Psychological Review, 42, Ravetz, J. R. (1971). Scientific knowledge and its 517–527. social problems. Oxford, England: Oxford Univer- Toulmin, S., & Leary, D. E. (1992). The cult of sity Press. empiricism in psychology, and beyond. In S. Koch Redelmeier, D. A., & Tversky, A. (1996). On the & D. E. Leary (Eds.), A century of psychology as belief that arthritis pain is related to the weather. science (2nd ed., pp. 594–617). Hyattsville, MD: Proceedings of the National Academy of Science, American Psychological Association. 93(7), 2895–2896. U.S. News & World Report. (2009, November 6). America’s best colleges. Rozin, P. (2009). What kind of empirical research Retrieved from http:// www.usnews.com/rankings should we publish, fund, and reward? A different Wason, P. C. (1960). On the failure to eliminate perspective. Perspectives on Psychological Re- hypotheses in a conceptual task. Quarterly Journal search, 4, 435–439. of Experimental Psychology, 12, 129–140. A guide to teaching research Saville, B. K. I. (2008). Wason, P. C. (1968). Reasoning about a rule. Quar- methods in psychology. Malden, MA: Blackwell. terly Journal of Experimental Psychology, 20, Shimp, C. P. (1999). Tolerance in a rigorous science. 273–281. Journal of the Experimental Analysis of Behav- Watson, J. B. (1913). Psychology as the behaviorist ior, 71, 284–288. views it. Psychological Review, 20, 158–177. Shimp, C. P. (2001). Behavior as a social construc- Whitehead, A. N., & Russell, B. (1910). Principia tion [Special Issue on The Longer View: 20th mathematica (Vol. I). Cambridge, England: Cam- Century Quantitative Analyses of Behavior]. Be- bridge University Press. havioural Processes, 54, 11–32. Winston, A. S. (2004). Controlling the metalanguage: Simon, H. A. (1986). Understanding the processes of Authority and acquiescence in the history of science: The psychology of scientific discovery. In method. In A. C. Brock, J. Louw, & W. van Hoorn T. Ganelius (Ed.), Progress in science and its (Eds.), Rediscovering the history of psychology: social conditions (pp. 159–170). Oxford: Perga- Essays inspired by the work of Kurt Danziger (pp. mon Press. 53–73). New York: Kluwer Academic/Plenum Skinner, B. F. (1938). The behavior of organisms. Press. New York: Appleton-Century-Crofts. Skinner, B. F. (1945). The operational analysis of Received March 26, 2010 psychological terms. Psychological Review, 52, Revision received August 26, 2010 270–277. Accepted September 1, 2010 Ⅲ