Working Paper Version 256 Guidelines for Science: Evidence and Checklists J. Scott Armstrong The Wharton School, University of Pennsylvania, Philadelphia, PA and Ehrenberg-Bass Institute, University of South Australia, Adelaide, SA, Australia [email protected] Kesten C. Green University of South Australia Business School and Ehrenberg-Bass Institute University of South Australia, Adelaide, SA, Australia [email protected] July 29, 2016 Draft: Guidelines for Science-256.docx Acknowledgements: We thank Hal Arkes, Jeff Cai, Rui Du, Lew Goldberg, Ray Hubbard, Gary Lilien, Scheherbano Rafay, and Arch Woodside for their reviews. This is not to imply that they agree the guidelines presented in this paper. In addition, Mustafa Akben, Len Braitman, Heiner Evanschitsky, Andreas Graefe, Don Peters, and Paul Sherman provided useful suggestions. Esther Park helped with copy-editing. Authors’ notes: (1) Each paper that we cite has been has been read by at least one of us. (2) To ensure that we describe the findings accurately, we are attempting to contact all authors whose research was cited as evidence. (3) We provide our oath that we did our best to provide objective findings and full disclosure. (4) Voluntary disclosure: We received no funding for this paper. (5) Estimated reading time for a typical reader is about an hour. 1 Abstract Problem: Evidence shows that most papers in scientific journals violate scientific principles. As a consequence, most published papers are useless. The problem addressed by this paper is, how can scientific practice in the management sciences and applied economics be improved? Approach: Operational guidelines to help researchers follow scientific principles were hypothesized from logic, established scientific best-practice, and from the authors’ own experiences. Experimental evidence about the scientific method was reviewed, and the guidelines were modified and extended to be consistent with the evidence. Findings: Key causes of the problem of unscientific research are government funding and regulation of research, and the use of invalid criteria for evaluating research by university managers and journal reviewers. We developed solutions in the form of checklists of (1) twenty- six evidence-based guidelines for scientists who wish to undertake useful scientific research, and (2) seven guidelines for science to help journal editors, funders, users, courts, and other stakeholders to evaluate whether a research paper is scientific. Limitations: Overlooked evidence or future research may suggest the need to modify or to extend the guidelines presented in this paper. No formal research has been done on the relative costs or effectiveness of the two checklists in improving the practice of science. Originality: This paper provides the first comprehensive evidence-based checklists of guidelines for conducting and evaluating the scientific quality of research efforts. Usefulness: Research stakeholders have the opportunity to increase the publication of useful papers and to discourage the publication of useless papers by replacing current approaches for evaluating research papers and researchers with the evidence-based guidelines and associated checklists in this paper. Keywords: advocacy; econometrics; evidence-based checklists; experiments; multiple hypotheses; objectivity; regression analysis; replication; statistical significance 2 Introduction Benjamin Franklin, the founder of the University of Pennsylvania, called for universities to be involved in the discovery and dissemination of useful knowledge (Franklin, 1743). Does this happen today in the management and social sciences? If not, why not-- And what can be done to fix the situation? Defining Science In his 1620 Novum Organum, Sir Francis Bacon suggested that the scientific method involves induction from systematic observation and experimentation. In the third edition of his Philosophiae Naturalis Principia Mathematica, first published in 1726, Newton described four “Rules of Reasoning in Philosophy”. The fourth rule reads “In experimental philosophy we are to look upon propositions collected by general induction from phænomena as accurately or very nearly true, notwithstanding any contrary hypotheses that may be imagined, till such time as other phænomena occur, by which they may either be made more accurate, or liable to exceptions” Bererlson and Steiner’s (1964, p.16-17) research on the scientific method provided six guidelines that are consistent with the above definitions, and identified prediction as one of the primary purposes of science. Friedman (1953) also asserted the central role of predictive validity in science. The Oxford English Dictionary (2015) offers the following in their definition of the scientific method: “It is now commonly represented as ideally comprising some or all of (a) systematic observation, measurement, and experimentation, (b) induction and the formulation of hypotheses, (c) the making of deductions from the hypotheses, (d) the experimental testing of the deductions…” Starting with the above definitions, we use the following definition of useful science: A process that studies important problems by using experimental evidence to compare multiple hypotheses. To do so, it uses cumulative scientific knowledge, systematic measurement using valid and reliable data, valid and simple methods for analysis, and logical deduction that does not go beyond the evidence in the paper. Finally, it fully discloses all information that would be needed to replicate the process. Advocacy and Other Problems with the Practice of Science "The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present – and is gravely to be regarded. Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the… danger that public policy could itself become the captive of a scientific-technological elite." Dwight D. Eisenhower, Farewell Address, 1961 The trend that concerned Eisenhower had begun in the early 1900s, yet it did not seem to have affected research at universities in a major way by 1960. Was Eisenhower right to be concerned that politics would eventually corrupt scientific research? In this section, we summarize current problems in the practice of science. We then describe the effects of advocacy and incentives on research. 3 Advocacy The human understanding when it has once adopted an opinion draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusion may remain inviolate. Bacon, Francis (1620), The New Organon and Related Writings. A survey of scientists with 77 useable responses by Mahoney and Kimper (1976) found that a great majority strongly preferred experiments that confirm a favored hypothesis. More than half of the respondents did not even recognize disconfirmation as a valid reasoning form. We refer to the single-minded focus on confirmation in research as “advocacy.” In a study using a task to determine what rule was used to construct a series of numbers— Wason’s (1960) “2-4-6 problem”—Mahoney and DeMonbreun (1977) tested the logical reasoning of 30 scientists against that of 15 Protestant ministers. Confirmatory bias was common in both groups, but the ministers used disconfirming tests almost twice as often as the scientists. In practice, researchers often adopt a preferred hypothesis, and design studies in order to support for their hypothesis. In other words, they act as advocates for their favored hypothesis. Mitroff (1969a, 1972a, 1972b) interviewed 40 eminent space scientists. He found that the scientists with the highest prestige did not live up to the scientific standard of objectivity. Instead, they were advocates for their hypotheses and resisted disconfirming evidence. Rather than viewing advocacy as harmful to the pursuit of useful knowledge, Mitroff considered it to be a legitimate and effective way to do science. Armstrong (1982a) used the advocacy method to prove that Mitroff was a fictitious author, while ignoring the disconfirming evidence that he knew Mitroff. Advocacy is common in the management sciences. An audit of 120 empirical papers published in Management Science from 1955 to 1976 found that that 64 percent used advocacy (Armstrong, 1979). An audit of 1,700 empirical papers in six leading marketing journals from 1984 to 1999 found that 74 percent used advocacy (Armstrong, Brodie and Parsons 2001). The primary strategies used by advocates are: 1. Analyses of non-experimental data. Support for a preferred hypothesis is easily obtained by applying multiple regression analyses and similar methods to non-experimental data. For example, advocates’ analyses of non-experimental data have found that competitor- oriented objectives—such as market share—lead to higher profits. In contrast, analyses of experimental studies showed that market share objectives are detrimental to the profitability and survival of firms (Armstrong and Green 2007). 2. Statistical significance testing. Armstrong (1970) showed how easy it is to get statistically significant findings from random numbers when using standard search rules for stepwise regression. Hubbard and Armstrong (1992) analyzed 32 randomly selected issues of each of the Journal