Critical Thinking Understanding and Evaluating Dental Research Third Edition

Donald Maxwell Brunette, PhD Professor Emeritus Department of Oral Biological and Medical Sciences Faculty of Dentistry University of British Columbia Vancouver, Canada

With contributions by

Ben Balevi, DDS, MSc Helen L. Brown, MLIS, MAS, MA Adjunct Professor Reference Librarian Faculty of Medicine University of British Columbia Library University of British Columbia Vancouver, Canada Private Practice Vancouver, Canada

Berlin, Barcelona, Chicago, Istanbul, London, Mexico City, Milan, Moscow, Paris, Prague, São Paulo, Seoul, Tokyo, Warsaw Library of Congress Cataloging-in-Publication Data

Names: Brunette, Donald Maxwell, author. Title: Critical thinking : understanding and evaluating dental research / Donald Maxwell Brunette. Description: Third edition. | Batavia, IL : Quintessence Publishing Co, Inc, [2019] | Includes bibliographical references and index. Identifiers: LCCN 2019017538 (print) | LCCN 2019018451 (ebook) | ISBN 9780867158014 (ebook) | ISBN 9780867158007 (softcover) Subjects: | MESH: Dental Research Classification: LCC RK80 (ebook) | LCC RK80 (print) | NLM WU 20.5 | DDC 617.6/027--dc23 LC record available at https://lccn.loc.gov/2019017538

5 4 3 2 1

All rights reserved. This book or any part thereof may not be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Editor: Zachary Kocanda Design: Sue Zubek Production: Sarah Minor

Printed in Korea TTTTTableableableableable ofofofofof contentscontentscontentscontentscontents

Dedication vii Preface viii

1 Reasons for Studying Critical Thinking 1

2 Scientific Method and the Behavior of Scientists 13

3 The Components of a Scientific Paper 32

4 Rhetoric 40

5 Searching the Dental Literature 55 with Helen L. Brown

6 Logic: The Basics 74

7 Introduction to Abductive and Inductive Logic: Analogy, Models, and Authority 84

8 Causation 97

9 Quacks, Cranks, and Abuses of Logic 110

10 Elements of Probability and Statistics, Part 1: Discrete Variables 128

11 Elements of Probability and Statistics, Part 2: Continuous Variables 145

12 The Importance of Measurement in Dentistry 152 13 Errors of Measurement 163

14 Presentation of Results 173

15 Diagnostic Tools and Testing 197 with Ben Balevi

16 Research Strategies and Threats to Validity 232

17 Observation 245

18 Correlation and Association 257

19 Experimentation 274

20 Experiment Design 288

21 Statistics As an Inductive Argument and Other Statistical Concepts 301

22 Judgment 315

23 Introduction to Clinical Decision Making 333 with Ben Balevi

24 Exercises in Critical Thinking 356

Appendices 375 Index 397 DDDDDedicationedicationedicationedication

To the late Doug Waterfield, with whom I collaborated extensively once it became clear through the in vivo studies of Babak Chehroudi that macrophages played an important role in tissue response to the topography of implants. Doug was a tireless advo- cate for students in the Faculty of Dentistry and spared no effort to help them succeed.

vii PPPPrefacerefacerefacerefacereface

The need for a third edition of Critical Thinking was by researchers to demonstrate the impact of their driven by developments in dental and biomedi- work, is also now considered, and there is updated cal research conditions and practice, as well as information on impact factors of dental journals reader comments. The third edition emphasizes and other bibliographic tools. the practical application of critical thinking in A second new contributor is Ben Balevi, who the production and evaluation of scientific takes over the chapter on diagnostic tests and publications. A number of issues are now more measures from my former student Carol Oakley, prominent. whose contributions continue to influence the Thanks to the Internet, patients of today have current chapter. Ben has a master’s degree in much more information (both accurate and evidence-based dentistry and has been recognized inaccurate) than the patients of yesteryear. The for his contributions in this area by the American consequence to dentists is that they must explain Dental Association. and be prepared to justify their clinical decisions There have been considerable changes in other with evidence. This requires that dentists be aspects of the book as well. A new chapter discusses capable of unearthing relevant information and the function of each section of the scientific paper evaluating alternatives by accepted criteria so so that researchers can check that their papers are that patients can be confident in their dentist’s fulfilling the expectations of readers and referees. recommendations. In line with the virtual revolu- The coverage on logic has expanded to include tion entailed in dealing with the ever-expanding more material on informal logic, for example, the scientific literature, the chapter on searching use of Walton’s critical questions for evaluating the dental literature now emphasizes scholarly causation. The informal logic used in the refer- publishing issues at all stages of research in addi- eeing process in which the burden of proof shifts tion to covering more traditional topics such as from authors to reviewers and back has been types of resources and searching techniques. The added to complement the argumentation maps preparation of systematic reviews is discussed in found in the chapter on judgment that features greater detail, as are the strengths and weaknesses tactical details such as rebuttals, caveats, and of various services such as Web of Science and supporting evidence. The intent is to give inves- Google Scholar as well as approaches to identi- tigators a practical approach to anticipating and fying relevant evidence in grey literature and its dealing with criticisms. The chapter on judgment use in reviews. The author of the revised chapter now includes materials of both historical (Darwin’s is Helen Brown, who has taken over from Kathy decision to marry and Benjamin Franklin’s moral Hornby, for whose contribution I continue to be algebra) and novel decision-making heuristics, grateful. Helen has master’s degrees in English such as fast frugal heuristics. literature and library science, along with frontline Clinical decision making has been given its own experience resolving student problems in litera- chapter and expanded to include detailed, worked- ture searches. She has made significant changes out calculations and clinical scenarios. In addition in the emphasis of this chapter. For example, the to decision tree analysis, two elements of central increasing trend toward open access publications importance to patients are presented with worked- and open science initiatives is addressed, provid- out calculations: cost-effectiveness analysis and ing important information for both prospective willingness to pay analysis. authors and readers with limited access to expen- The chapter on diagnostic tools and testing has sive subscription-based resources. The strengths been expanded to include new worked-out calcu- and weaknesses of the h-index, now widely used lations and examples, as well as illustrations

viii to clarify this topic, which many find difficult and inform numerous sections. Regular University of Brit- confusing. ish Columbia (UBC) contributors in this way include The chapter on experimentation now covers Babak Chehroudi, Tim Gould, Hugh Kim, Chris Over- randomized field trials, a method employed in caus- all, Mandana Nematollahi, Eli Konorti, and Nick Jaeger ally dense environments such as business problems, (Electrical and Computer Engineering). Colleagues often combined with trial-and-error approaches. The outside UBC, Christopher McCulloch (University of technique is evolving but has been found useful by Toronto), Ken Yaegaki (Nippon Dental University), organizations such as Google that run thousands of and Doug Hamilton (Western University), have also randomized field trials. provided advice and encouragement. I also thank my The statistics chapter has expanded coverage of collaborators at the Swiss Institute of Technology Bayesian statistics, as this approach is expected to (Zurich), Professors Marcus Textor and Nicolas Spencer, become more widely used. The chapter on presenting and my former post-doc Dr Marco Wieland, for continu- results contains new illustrations and more extensive ing my education in surface science. Of course, I would discussion of the integrity-based principles for display- be an ingrate if I failed to acknowledge the continuing ing information emphasized by Tufte. The section on support of my wife Liz, sons Max and Regan (and their the behavior of scientists has been expanded with partners Heather and Malizza), who provided various discussion on the reproducibility crisis now shaking examples used in the book including the exploits of some scientists’ faith in the reliability of reported my problem-solving grandson Calixte and developing results, even including those published in high-impact gymnast granddaughter Genèvieve. journals. All chapters have also been improved by the I also thank Quintessence for their ongoing support addition of updates, revisions, or examples. through three editions of this work. This edition has I acknowledge the support of my colleagues who been facilitated through the efforts of Bill Hartman and have never hesitated to provide advice in their respec- Bryn Grisham. Zach Kocanda has been exemplary in his tive domains of expertise. I also thank my former attention to detail, often under trying circumstances, students, friends, and collaborators, whose insights in editing the manuscript.

1 1

Reasons for Studying

1 1 1 Critical Thinking 1 1 1

Critical Thinking

Critical thinking has been defined many ways, from the simple—“Critical thinking is deciding rationally what to or what not to believe”2—to the more detailed “Critical thinking is concerned with reason, intellectual honesty, and open-mindedness, as opposed to emotionalism, intellectual laziness, and closed-mindedness”3—to the nearly comprehensive: It has happened more than once that I found Critical thinking involves following evidence where it leads; considering all possibilities; relying on reason rather than emotion; being it necessary to say of precise; considering a variety of possible viewpoints and explana- one or another eminent tions; weighing the effects of motives and biases; being concerned colleague, ‘He is a very more with finding the truth than with being right; not rejecting “ unpopular views out of hand; being aware of one’s own prejudices busy man and half of 3 and biases; and not allowing them to sway one’s judgment. what he publishes is Self-described practitioners of critical thinking range from doctrinaire true but I don’t know postmodernists who view the logic of science with its “grand narratives” which half.’” as inherently subordinating4 to market-driven dentists contemplat- ing the purchase of a digital impression scanner. In this book, critical ERWIN CHARGAFF1 thinking, and in particular the evaluation of scientific information, is conceived as “organized common sense” following Bronowski’s view of science in general.5 Of course, common sense can be quite uncom- mon. A secondary use of the term critical thinking implies that common sense involves a set of unexamined and erroneous assumptions. For example, prior to Galileo, everyone “knew” that heavy objects fell faster than lighter ones. Critical thinking as organized common sense takes the systematic approach of examining assumptions. The professional use of critical thinking is particularly complex for dental professionals because they live in two different worlds. On the one hand, they are health professionals treating patients who suffer from oral diseases.

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 On the other hand, dentists typically also inhabit the misinformation. Dentists thus are more likely to be business world, where decisions may be based on the questioned by patients on proposed treatment plans principle of maximizing income from their investment. and options. In responding to such questions, it is Dental practice is based only very loosely on responding clearly advantageous for dentists to be able to present to disease6; less than one-third of patient visits result a rational basis for their choices. Chapter 23 covers an in identifying a need for restorative care.7 Twenty evidence-based approach to clinical decision making percent of work is elective, such as most of orthodon- and appendix 9 provides a template for dental offices tics, tooth whitening, and veneers, and typically that to use in documenting their decisions based on recent work comprises the most lucrative aspects of prac- evidence. tice. Thus, the information that must be evaluated in A systematic approach to analyzing scientific papers performing these disparate roles covers the spectrum has to be studied, because this activity requires more from advertisements to financial reports to systematic rigor than the reasoning used in everyday life. Faced meta-analysis of health research. with an overabundance of information and limited Dentists are health professionals, people with special- time, most of us adopt what is called a make-sense epis- ized training in the delivery of scientifically sound temology. The truth test of this epistemology or theory health services. The undergraduate dental curriculum of knowledge is whether propositions make superficial is designed to give dental students the basic knowl- sense.9 This approach minimizes the cognitive load and edge to practice dentistry scientifically, at least to the often works well for day-to-day short-term decision extent allowed by the current state of knowledge. But making. In 1949, Zipf of Harvard University published if any guarantee can be made to dental students, it Human Behaviour and the Principle of Least Effort, in which is that dentistry will change, because the knowledge he stated: base of biomedical and biomaterial sciences grows continually. Most dentists today have had to learn tech- The Principle of Least Effort means, for example, niques and principles that were not yet known when that in solving his immediate problems he will they were in dental school. In the future, as the pace view these against a background of his probable of technologic innovation continues to increase and future problems, as estimated by himself. Moreover, the pattern of dental diseases shifts, the need to keep he will strive to solve his problems in such a way up-to-date will be even more pressing. Means of staying as to minimize the total work that he must expend current include interacting with colleagues, reading the in solving both his immediate problems and his dental literature, and attending continuing education probable future problems.10 courses—activities that require dentists to evaluate information. Yet, there is abundant historical evidence Zipf used data from diverse sources ranging from that dentists have not properly evaluated information. word frequencies to sensory sampling to support his Perhaps the best documented example in dentistry thesis. Although the methods and style of psychologic of a widely accepted yet erroneous hypothesis is the research have changed, some more recent discover- focal infection theory. Proposed in 1904 and accepted ies, such as the concept of cognitive miser in studies of by some clinicians until the Second World War, this persuasion,11 coincide with Zipf’s principle. Kahneman untested theory resulted in the extraction of millions of in Thinking, Fast and Slow has elevated the principle to sound teeth.8 But errors are not restricted to the past; a law noting that we “conduct our mental lives by the controversial topics exist in dentistry today because law of least effort.”12 new products or techniques are continually introduced In science, the objective is not to make easy short- and their usefulness debated. Ideally, dentists should term decisions but rather to explain the phenomena become sophisticated consumers of research who can of the physical world. The goal is accuracy, not neces- distinguish between good and bad research and know sarily speed, and different, more sophisticated, more when to suspend judgment. This goal is different from rigorous approaches are required. Perkins et al9 have proposing that dentists become research workers. One characterized the ideal skilled reasoner as a critical objective of this book is to provide the skills enabling epistemologist who can challenge and elaborate hypo- a systematic method for the evaluation of scientific thetical models. Where the makes-sense epistemologist papers and presentations. or naive reasoner asks only that a given explanation or A marked addition to the challenges of dental prac- model makes intuitive sense, the critical epistemologist tice in recent years is that patients have increased moves beyond that stage and asks why a model may be access through the Internet to information as well as inadequate. That is, when evaluating and explaining,

2 Critical Thinking

Table 1-1 | Level of evidence guideline recommendations of the United States Agency for Healthcare Research and Quality

Grade of Level Type of study recommendation

1 Supportive evidence from well-conducted RCTs that include 100 patients or more A

2 Supportive evidence from well-conducted RCTs that include fewer than 100 patients A

3 Supportive evidence from well-conducted cohort studies A

4 Supportive evidence from well-conducted case-control studies B

5 Supportive evidence from poorly controlled or uncontrolled studies B

6 Conflicting evidence with the weight of evidence supporting the recommendation B

7 Expert opinion C RCTs, randomized controlled trials. the critical epistemologist asks both why and why not a of influencing future research (less than 25% of all postulated model may work. The critical epistemologist papers will be cited 10 times in all eternity,14 and a arrives at models of reality, using practical tactics and large number are never cited at all), and (4) a good deal skills and drawing upon a large repertoire of logical and of the research on a clinical problem may be irrelevant heuristic methods.9 to a particular patient’s complaint Psychologic studies have indicated that everyday The actual rate of growth of the scientific literature cognition comprises two sets of mental processes, has been estimated to be 7% per year of the extant System 1 and System 2, which work in concert, but literature, which in 1976 comprised close to 7.5 million there is some debate whether they operate in a paral- items.11 This rate of growth means that the biomedical lel or sequential manner. System 1 operates quickly literature doubles every 10 years. In dentistry, there and effortlessly, whereas System 2 is deliberate and are about 500 journals available today.15 Many dental requires attention and effort.12 System 2 is a rule-based articles are found in low-impact journals, but, ignoring system, and engaging System 2 is the surest route to these, there were still 2,401 articles published in 1980 fallacy-free reasoning.13 System 2 becomes engaged in the 30 core journals.16 More recently, it has been when it catches an error made by the intuitive System estimated that about 43,000 dental-related articles are 1. The good news is that extensive work by Nisbett and published per year. colleagues (briefly reviewed by Risen and Gilovich13) However, the problem is not intractable. Relman,17 showed that people can be trained to be better reason- a former editor of the New England Journal of Medicine, ers and that people with statistical backgrounds were believes that most of the important business of scien- less likely to commit logical fallacies. Nisbitt and tific communication in medicine is conducted in a colleagues further demonstrated that even very brief very small sector of top-quality journals. The average training was effective in substantially reducing logical practitioner needs to read only a few well-chosen peri- and statistical errors. Thus this book has chapters on odicals.17 The key to dealing with the problem of the logic and statistics. information explosion is in choosing what to read and A second objective of the book is to inculcate the learning to evaluate the information. habits of thought of the critical epistemologist in read- Dentists are exposed to diverse information sources, ers concerned with dental science and clinical dentistry. and the important issues vary depending on the source. For example, a dentist may wish to determine whether potassium nitrate toothpastes reduce dentin hypersen- The scope of the problem sitivity. One approach would be to look up a systematic review on this topic in the Cochrane Library,18 which In brief, the problems facing anyone wishing to keep up many regard as the highest level in the hierarchy of with developments in dentistry or other health profes- evidence (Table 1-1). The skills required to understand sions are that (1) there is a huge amount of literature, the review would include a basic knowledge of statis- (2) it is growing fast, (3) much of it is useless in terms tics and research design. The same dentist, facing

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 the competitive pressures of his or her local market, A good deal of the literature circulated to dentists does might also want to determine whether a particular not meet these requirements. But even excluding the laser-bleaching process should be adopted for the throwaway or controlled-circulation magazines that are practice. In that instance, there might not be a rele- little more than vehicles for advertisements, the amount vant Cochrane review, and there may not even be a of information published annually appears formidable. relevant paper in a refereed journal to support a deci- One approach to dealing with a large number of sion. Available evidence might consist of advertising papers is to disregard original papers and receive brochures and anecdotes of colleagues. The dentist information secondhand. Dental and medical journals may have to employ a different set of skills, ranging present reviews of current research in specific clini- from evaluating the lie factor in graphics (see chapter cal or scientific fields; some journals, such as Dental 14) to disentangling rhetoric from fact. Advertisements Clinics of North America and the Journal of Evidence Based and salesmanship are persuasive exercises; the chapter Dentistry, are exclusively devoted to this approach. on rhetoric (chapter 4) deals with means of persuasion. Although this tactic reduces the volume of literature to Typically, dentists acquire information on innova- be covered, it does not solve the problem of evaluating tive procedures through participation in networks in the information contained in the reviews. To perform which their colleagues supply informal data on the this task effectively, a researcher must be able to assess effectiveness of the innovations. Nevertheless, dentists the soundness of the reviewer’s conclusions. In decid- cite reading peer-reviewed dental literature and exper- ing to accept information secondhand, the researcher imental studies as the gold standard for determining is also deciding whether the author of the review is the quality of innovations.19 New technology is often a reliable, objective authority. Thus, the problem of introduced into their practices through trial and error; evaluation has been changed, but not eliminated. dentists take the pragmatic approach of directly deter- This book focuses on the primary literature, where it mining what works in their hands in their practice.19 is hoped that new, true, important, and comprehensi- Doubtless, some of the personal and financial expenses ble information is published. The systematic review, a typical of the trial-and-error approach could be reduced relatively new review form, attempts to deal with some with more effective evaluation of information prior to of the more glaring problems of traditional reviews and selecting a material or technique for testing. is covered briefly in chapter 5. Although useful for some This book focuses on evaluating refereed scientific purposes, the systematic review has its own shortcom- papers, but many of the issues of informational quality ings, and the researcher must judge how these affect and questions that should be asked apply equally to the conclusions. Journals vary in quality; chapter 5 also other less formal channels of communication. discusses bibliometric approaches of ranking journals. In the following section, I present a brief review of how articles get published that may help explain some of What is a scientific paper? this variation.

The Council of Biology Editors defines a scientific paper as follows: The Road to Publication An acceptable primary scientific publication must be the first disclosure containing sufficient infor- The author mation to enable peers (1) to assess observations; (2) to repeat experiments; and (3) to evaluate The author’s goal is to make a significant contribu- intellectual processes; moreover, it must be sensi- tion to the scientific literature: a published paper. To ble to sensory perception, essentially permanent, accomplish that goal, the author will have to produce available to the scientific community without a submission for publication whose contents are new, restriction, and available for regular screening true, important, and comprehensible. Moreover, the by one or more of the major recognized secondary author wants to publish the paper in a journal whose services.20 readers will likely find the paper of interest and hope- fully be influenced by it. As journals vary in the rigor Similar ideas were stated more succinctly by they demand and the length of papers they accept, DeBakey,21 who noted that the contents of an article the author needs to identify the best journal for his or should be new, true, important, and comprehensible. her purposes.

4 The Road to Publication

Refereed versus nonrefereed journals The literature available to dental health professionals ranges the spectrum of refereed to nonrefereed, from The first hurdle faced by an article submitted for publi- low (or no) rejection rates to high rejection rates. The cation is an editor’s decision on the article’s suitability Journal of Dental Research, for example, used to have a for the journal. Different journals have different audi- 50% rejection rate (Dawes, personal communication, ences, and the editors are the arbiters of topic selection 1990), but that has risen so that 25 years later some for their journal. Editors can reject papers immediately 90% of submissions are rejected.24 Even among high-im- if they think the material is unsuited to their particular pact journals, however, there is no guarantee that the journal. referees did a good job. In fact, these considerations In some journals, acceptance or rejection hinges solely only serve to reinforce the view “caveat lector”—let on the opinion of the editor. However, this method is the reader beware. problematic because informed decisions on some papers can only be made by experts in a particular field. There- fore, as a general rule, the most highly regarded journals Editors and referees ask the opinion of such specialists, called referees or editorial consultants. Referees attempt to ensure that a The editor of the journal and the referees are the “gate- submitted paper does not demonstrably deviate from keepers” who decide whether a manuscript is accepted. scientific method and the standards of the journal. In science the basic rule appears to be something akin Whether a journal is refereed can be determined by to “if it doesn’t get published, it doesn’t exist.” Thus the consulting Ulrich’s Periodicals Directory (ulrichsweb. rewards in science go to those who publish first, not com). Editors usually provide referees with an outline of the first scientist to observe a phenomenon. Obviously, the type of information that they desire from the referee. pleasing these gatekeepers is essential to a scientific The criteria for acceptance will necessarily include both career. objective (eg, obvious errors of fact or logic) and subjec- The editor and the referees are the representatives tive (eg, priority ratings) components. Unfortunately, of the readers of the journal. They protect the read- the task of refereeing is difficult and typically unpaid. ers from wasting their time on obviously erroneous or Refereeing is often squeezed in among other academic uninteresting or unsuitable or unoriginal or opaque or activities, so it should not be surprising that it some- trivial submissions. The papers in the journal must be times is not done well and that referees often disagree. relevant to the readership. Studies of the reliability of peer-review ratings are An important part of the editor’s job is to protect disappointing for readers wanting to keep faith in the authors from unjust criticism that can arise from such peer-review system. Reliability quotients, which can things as personal animosity between an author and a range from 0 (no reliability) to 1 (perfect reliability), for referee or an attempt by a referee to block publication various attributes of papers submitted to a psychology of a competitor’s work. Unfortunately, the scientists journal22 follow: who are best able to evaluate a submission may be individuals who can suffer most from its publication, • Probable interest in the problem: 0.07 as for example occurs when the referee’s own work is • Importance of present contribution: 0.28 “scooped” (ie, published earlier by a competitor). • Attention to relevant literature: 0.37 To justify readers’ expenditure of time the paper • Design and analysis: 0.19 should address a significant problem or concern and • Style and organization: 0.25 provide a significant amount of information. The • Succinctness: 0.31 length of journal articles varies; some journals publish • Recommendation to accept or reject: 0.26 “letters” rather than full-length papers for interesting but only briefly developed findings. Editors are inter- Despite such issues, there is evidence that the review ested in publishing papers that are likely to be cited process frequently raises important issues that, when in the future or, expressed another way, are build- resolved, improve the manuscript substantially.23 ing blocks for future research or clinical application. After consulting with referees, the editor decides Tacker25 notes that journals differ in the sophistica- whether the paper should be (1) published as is—a tion of their readership. A general medical journal (eg, comparatively rare event; (2) published after suit- JAMA) is written at the comprehension level of a third- able revision; or (3) rejected. Journals reject papers year medical student, whereas a specialty journal is in proportions varying from 0% to greater than 90%. written for a first- or second-year resident. A scientific

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 journal should be understandable to third- or fourth- An indicator of what editors want is provided by the year PhD candidates or above in the general field. instructions given to referees of journals, often in the form of a checklist or a score sheet that incorporates The editor specific questions for reviews completed online. As an example, I compiled an indicator of some of editors’ The editor decides ultimately whether to accept or concerns by simply looking at the instructions sent to reject a submission. As a general rule the editor is me by ten journals. The following characteristics were an unpaid (or lowly paid) volunteer of distinguished emphasized: standing in the field covered by the journal. The editor defines the scope of the journal (ie, what subjects • 90% (ie, 9/10) concise are published in it), and if a manuscript falls outside • 70% clear the journal’s mandate, it will probably be returned • 70% evaluate by section (eg, introduction, methods) promptly to the author. Similarly, an editor may reject • 70% adequacy of references a paper on the grounds that a submission does not • 60% originality advance the field sufficiently or has a low potential • 60% adequacy of illustrations for future impact. Such judgments are subjective but • 50% relationship of conclusions to results nevertheless may need to be made. I call this the “de gustibus” standard after the Latin adage, De gustibus Overall the instructions emphasize economy of non est disputandum: “In matters of taste, there can be expression, ignoring the folk wisdom that “Sermons no disputes.” As the adage indicates, if a decision is on brevity and chastity are about equally effective.”26 made on this basis it will be difficult to persuade the Nevertheless it is useful for prospective authors to editor to reverse it. obtain a specific checklist for the journal to which they Editors are often responsible for diverse other tasks are submitting so that they can attempt to meet the such as recruiting referees and persuading them to journal’s expectations. submit their reviews in a timely manner. Some journals have associate editors who oversee submissions in The referees their area of expertise, and the editor must coordinate their activities as well as consult with editorial boards The referees are unpaid volunteers; nevertheless they and deal with the various business matters. Despite do receive some nonmonetary rewards. They get first the importance of their job, editors are not always access to new information in a field that interests them, appreciated by their colleagues, who may resent some and their decisions can influence the direction of that decisions. Chernin playfully suggests, “Editors are also field. On occasion that information may be useful—for the people who separate the wheat from the chaff and example, a submission could contain a reference of frequently publish the chaff.”26 which the referee was unaware or a new technique After the manuscript is accepted by the editor, it that might be beneficial to the referee’s own research, may be passed on to a managing editor to take the or reading the article might prompt an idea for the manuscript though the production and publication referee’s future research. Finally, in doing a favor to process. Day27 states that editors and managing editors the editor in refereeing a manuscript, the referee might have jobs that are made impossible by the attitudes acquire a store of goodwill that might help when his of authors who submit to their journals. For example, or her own manuscript is submitted to the journal. authors might ignore the rules and conventions spec- (Another well-accepted persuasive factor—reciproca- ified by the journal (eg, the format for the references). tion).28 Nevertheless, refereeing papers is a low-yield Or authors and referees may have irreconcilable views, task—the referees’ efforts help the editor and those and the editor may be caught in the middle. Given that whose papers are published, but the referee typically the editor’s decision could affect the author’s career, gets no tangible benefit save the good feeling that it is clearly wise not to irritate editors or referees, but comes from doing the right thing. Spending their own rather to make their job in dealing with the submission time on work for which others will benefit is bound to as easy as possible. That is, authors want the editor to lead to resentment if those potentially benefitted make like them, and as has been extensively studied in the the task more difficult than it need be. The applica- psychology literature,28 liking can be a key factor in ble golden rule then is to do unto the referees as you persuasion, in this case persuading the editor that the would have them do unto you. Make it easy for the submission should be published. referees in the hope they will make it easy for you. In

6 The Road to Publication

this spirit then authors should attempt to meet the or reject, priority, overall length of the paper. More expectations of referees, in particular not wasting their detailed points are given in the comments to authors time. In general, referees expect a scientific writing or the editor or editor’s assistant. The confidential style characterized by the following qualities: comments to the editor give the referee the possibil- ity to offer frank criticism that might be construed by • Objectivity. Data obtained from scientific observation some authors as being insulting. For example, a referee should be capable of being repeated by any compe- might comment that the paper is poorly written and tent observer, and the interpretations should be needs revision by a native English speaker, and such a similarly identical among investigators. Expressed comment might be insulting to an author who was in another way, in the “Storybook” version of scientific fact a native English speaker. method, there is no room in science for subjective personal data collection and interpretation. Some- times writers attempt to emphasize their objectivity, Referees versus authors and this desire to appear objective can lead to over- use of the passive voice. Of course investigators do Typically referees make critical comments on the have an axe to grind, as they want to be published so papers they are reviewing ranging from the easily that they can reap the rewards of publication—recog- correctable, such as typographic errors or formatting, nition and employment being the chief among these. through more difficult problems to correct, such as lack So a tradition has arisen whereby authors attempt of clarity in organization, deficiencies such as inappro- to appear to be objective while being strong advo- priate methodology, or erroneous logic, that lead to cates for their position. Thus, authors make “Verbal unsupported conclusions. Typically the referees will choices . . . that capitalize on a convenient myth . . . number their comments, and the editor will require reason has subjugated the passions.”29 In any case, the author to address each of them. So in effect the readers have come to expect that scientific writers authors and each of the referees enter into a debate will present at least a veneer of objectivity (practi- presided over by the editor, who might also provide tioners of qualitative methods might disagree), but some comments, that might be classed according to readers have other expectations of authors as well. the conventions of informal logic or pragmatics as a • Logic. Logic not only in organization but in sentence “persuasion dialogue.”33 The participants are obligated structure and meeting reader expectations.30 to give helpful and honest replies to their opponents’ • Modesty. Related to the scientific norm of humility questions. In theory each participant in the dialogue (extravagant claims will attract close and probably is supposed to use arguments exclusively composed of critical attention). premises that are commitments of the other partici- • Clarity. Scientific writers should follow the common pant. But in argumentation, as in life, commitments advice to all writers such as avoiding misplaced are notoriously difficult to extract from an opponent, modifiers, dangling participles, nonparallel construc- and pretty much the best one can hope for is plau- tions, stacked modifiers, etc. (There are numerous sible commitment to an opinion based on reasoned books on writing style, such as Strunk and White’s evidence. In conducting the argument the participants The Elements of Style31 or Zinnser’s On Writing Well.32) are also obligated with a burden of proof, which shifts • Precision. Use of precise terminology to avoid confu- from one to the other during the dialog. For exam- sion and the fallacy of equivocation. ple, in submitting the paper the author, as proponent, • Brevity. To conserve readers’ time. assumes the burden of proof for the conclusions of • Justified reasoning. Making the reason for statements the paper, and the components of the paper (ie, meth- clear by referring to data in the paper (eg, “see Figure ods, data, figures, tables, and logic), constitute the 1”) or references to the literature. means of bearing that burden. Similarly the referee, • Use of signposts (eg, “our first objective…,” “our in making a criticism, assumes the burden of proof of second objective…”), linkage, etc. justifying the criticism. This may be done by various means such as citing deficiencies in the evidence in the Referees typically submit their reports by filling paper, external scientific evidence (such as previously out forms online often accompanied by explanatory published papers), or expected standards in the field remarks in an uploaded text file. of study. The editor forwards the referees’ criticisms Typically the form starts with what might be along with a preliminary decision to the authors who, if called high-level assessments—questions like accept they want the submission to proceed to publication, are

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 expected to bear the burden of proof in responding to probable, or beyond any reasonable doubt) and this the criticisms. This dialogue can be carried over several feature can hold the key to resolution of conflicts. cycles. Often in my experience, it seems that referees Authors can back off or limit their claims to account seldom accept or commit to the author’s arguments; for the views of the referees, and the editor can in good rather they merely concede by terminating discussion. conscience publish the article. In science, as in life, it is difficult to say “Sorry, I was wrong.” In some instances agreement between the referees and the authors is never achieved, but the The readers issues are clarified to an extent that the editor can make a decision. The end users of the published paper, the readers, have The question arises of the logic used by editors in been defined as anyone who reads the text “with an making their decisions. First it should be noted that intentional search for meaning.”35 Editors and refer- different types of reasoning employ different stan- ees are knowledgeable about their fields and like the dards of proof, and this is not unusual in human authors suffer from the problem of familiarity with the affairs. In law for example the standard of proof in assumptions, conventions, and expectations of inves- criminal cases is “beyond reasonable doubt” whereas tigators in their field so that they tend to “fill in” what civil cases are decided on the “balance of probabilities.” an author might leave out. General readers, however, Scientific arguments can be complex and may entail differ from the editors and referees in that on average various forms of logic ranging from the certainty of they are less familiar with the research field and may deductive logic employed in mathematics to inductive lack information required to understand the submis- logic, which can deal with calculated probabilities, to sion. Expressed another way, they can’t fill in what the informal logic that balances many factors but does not author leaves out. As the readers vary widely in their necessarily proceed by strict numeric calculation so expertise, it falls to the author to determine what they that the conclusions are classed qualitatively in terms are likely to know (ie, what is common knowledge to of their relative plausibility. everyone in the field), and conversely what the author Perhaps the reasoning process most employed by needs to point out to them. Anything that is novel or editors, who have to make a practical decision, would unusual needs to be described in detail; for example, be the pragmatic model devised by the philosopher investigators may vary from the standard methods in Stephen Toulmin34 (see also chapter 22 for more on their measurements or calculations, and such changes argumentation maps), which specifies a system for need to be highlighted and explained. scientific explanation that includes Claims (such as conclusions in the paper) justified by Evidence and Warrants. A Warrant is the means that connects the Editorial independence Claim to the Evidence; it may be, for example, a scientific principle or a connection established by previous Ideally, the contents of the journal should be indepen- work. An important aspect of Toulmin’s approach is dent of economic issues, but this is not necessarily the that it is field dependent so that appropriate standards case. Publication of color illustrations can be prohib- are employed for differing types of scientific endeavor. itively expensive, and many respected journals are One can see this aspect in action in the scientific litera- publications of learned societies that operate on lean ture by observing the content of papers where the rigor budgets. The Journal of Dental Research, for example, is of the methods, the quantity of data, or the articulation published and subsidized by the International Asso- of the findings differ among different fields of science ciation for Dental Research. Such a journal would be or among the journals within one field of science. It expected not to be subject to advertisers’ influence. is the editor who determines the standards of his/ Other journals have a need to generate income, and, in her journal, and differences between editors in what some instances, entire issues appear to be sponsored they consider important findings or flaws can result by a commercial interest. It is not unreasonable to in a paper rejected by an editor of one journal being wonder whether the advertiser influenced the editorial accepted by another one. There are other elements in content, for “he who pays the piper calls the tune.” In the Toulmin model, including the Rebuttal arguments recent years, Internet-based journals have arisen that that restrict or counter the claim and the Qualifier, are financed by authors through charges per page. As which indicates the degree of certainty that the propo- hard copy of the articles are not produced or distrib- nent assigns to the Claim (eg unlikely, possibly, highly uted, costs are minimal, and the potential for profit is

8 The Road to Publication

great. There is thus an incentive for such journals to deficient. Returning to the example, because scientists have a very low (or no) rejection rate, and questionable have been studying tooth development for decades quality may result. using standard histologic techniques, there is not much hope that reworking the same approach would provide anything exciting; new methods would be required to Three general questions bring new insights. As a consequence of scientific progress, methods A scientific paper is not necessarily an unbiased become outdated and standards change. Changing account of observations; it is more likely an attempt to standards can be seen in biochemistry by examining the convince the reader of the truth of a position. As noted standards for publication of data using polyacrylamide by Ziman,36 it is important to realize that much of the gels. Early publications using the technique showed research literature of science is intended, rhetorically, photographs of gels that did not have good resolution or to persuade other scientists of the validity of received uniformity and showed poor staining. The photographs opinions. Thus, a reader can expect an author to of gels were often so uninformative that Archives of present his or her data in the most favorable light. Oral Biology instructed authors to submit densitomet- Tables, figures, and even calculations may be done ric tracings of the gels. Currently, gel separations are so that differences between groups are accentuated done in two dimensions with superb resolution, and the and the appearance of error is minimized. A reader’s proteins are stained with much greater sensitivity. A defense as a consumer of this information is an atti- photograph of a gel that would have been acceptable tude of healthy skepticism. Three general questions a 30 years ago would not be acceptable for publication skeptical reader should ask are: Is it new? Is it true? today. In judging papers, therefore, a key question is Is it important?37 whether the techniques and approach are up-to-date as well as whether the question is original. Is it new? This principle is so well accepted that investigators sometimes rush to apply new techniques to appear A minimum requirement for publication in most up-to-date. Fisher,39 the pioneer statistician and author instances is that the information is new. However, of a classic work on experimental design, warned, “any new can be defined in various ways. If a paper using brilliant achievement . . . may give prestige to the standard histologic techniques reporting the develop- method employed, or to some part of it, even in appli- ment of teeth in lynx were to be published tomorrow, cations to which it has no special appropriateness.” it might well be new, because, as far as I am aware, An exception to the requirement of “newness” for a the development of lynx teeth has not been described publication is the need to report confirmations of previ- previously. However, it probably would not be new in ous work. One journal for which I refereed placed the adding anything to our knowledge of tooth develop- category “valuable confirmation of previous work” in ment in general. Such a paper would merely fill in the third place in their ranking system below exciting orig- gaps, however small, in our knowledge. I think that inal research and interesting new findings, but above journal editors are fairly lenient in their judgments on categories related to rejection. This type of research what constitutes new information. Kuhn38 states that is taking on increasing importance in the light of the one of the reasons why normal puzzle-solving science “reproducibility crisis” to be discussed later. seems to progress so rapidly is that its practitioners concentrate on problems that do not tax their own lack Is it true? of ingenuity. The quality that often distinguishes good scientific Sound conclusions are the result of reliable observations papers from the mediocre is originality. Funding agen- combined with valid logic. Knowledge of measurement, cies are probably better gatekeepers of science in this types of observational errors, experimental design, regard, because an essential criterion for funding is and controls give some basis to assessments of the originality. Originality can appear in any component reliability of observations. Thus, sections of this book of the research process, including the questions being deal with these topics and the logic used to interpret asked, the methods employed, the research design, or the observations. But the ultimate test of any scien- even the interpretation. Because science is a progres- tific observation is reproducibility; indeed, a practical sive business, approaches that were once original definition of truth for the purposes of pragmatic work- and sufficient can with time become derivative and ing scientists is that a scientific statement is true if

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 it allows us to make useful, reliable predictions that and medical literature that there is no shortage of are confirmed when tested by a competent scientist well-documented threats to truth. under the specified conditions. There are theoretical or practical limitations to any approach. Newton’s laws Is it important? of motion would be perfectly valid when applied to billiard balls colliding on a pool table but not useful at The importance of a paper cannot be tested in a very small scales of the world of subatomic particles completely objective manner. Maxwell41 has argued— where quantum physics would be preferred. Note that in my opinion, persuasively—that real progress in confirmation does not imply the exact same numeric science is assessed in terms of the amount of valu- result, but rather one that is within the specified inter- able factual truth that is being discovered and that val of reported uncertainty. the accumulation of vast amounts of trivia (even if A clue to the reproducibility of an observation is factually correct) does not amount to progress. The the consistency of the results within the body of the problem is that value judgments are highly subjective. paper. Another means for evaluating the reliability of One approach to measuring impact of a paper is the observations in a paper is to read what other scientists number of citations to the paper, an aspect that will be have written about the work, and citation analysis is discussed in chapters 5 and 22. Many scientists have an efficient means of uncovering that information. For accepted this criterion and include the citation record various reasons, to be discussed later, there is a current of their papers in their curriculum vitae or include “reproducibility crisis” perceived in which confidence indices derived from their citation record such as the in the reproducibility of findings, even those published h-index (discussed in chapter 5). in high-impact journals, is waning. One can speculate about what qualities an ideal A student might wonder whether it is necessary to evaluator should have. Beveridge42 has suggested learn such diverse concepts and examine the litera- the concept of scientific taste, which he described ture to such a detailed extent, particularly when it as a sense of beauty or esthetic sensibility. Beveridge seems likely that the vast majority of publications are explained scientific taste by stating that: produced in good faith and come from institutions of higher learning. Ioannidis,40 however, has argued The person who possesses the flair for choos- that most published research findings are false. Ioan- ing profitable lines of investigation is able to see nidis’ estimate is sensitive to the pretest probability of further where the work is leading than are other the hypothesis being true, and a low estimate of this people because he has the habit of using his imag- value will lead to a higher proportion of papers’ conclu- ination to look far ahead, instead of restricting sions being classed as false. Nevertheless, as will be his thinking to established knowledge and the discussed later, current research into reproducibility of immediate problem. findings has provided more direct evidence indicating a significant proportion of findings are false in that they A person with scientific taste would be a good judge cannot be reproduced. In Ioannidis’ common sense of the importance of a scientific paper. Traditionally, view, a research finding is less likely to be true when the skill of judgment is learned in the apprentice- effect sizes are small, when there are a large number of master relationship formed between graduate student tested hypotheses that have not been preselected, and and supervisor. Techniques may come and go, but when there are great flexibilities in designs, definitions, judging what is important and how it can be innova- outcomes, and data analyses. Other problems impact- tively studied are the core business of scientists, and ing the truth of the conclusion are financial issues and these skills are learned similarly to a child learning other interests and prejudices as well as the number his prayers at his mother’s knee: Graduate students of teams in a field chasing statistical significance. I hone their critical skills in the supervisor’s office or believe it is unlikely that most research findings are lab meetings. Thus, much importance is attached to false, because if they were there would be more papers the pedigree of a scientist, and some scientists take reporting failure to confirm results (though admittedly pride in tracing their scientific pedigrees to leading publishing such negative results can be difficult) and figures in a field of study. many fewer papers confirming—albeit often indi- Given the large variation in laboratory and supervi- rectly—replication. Nevertheless, the considerations sor quality, there will always be significant differences listed by Ioannidis serve to warn readers of the dental in judgment. This diversity is evident in an extensive

10 References

study of proposals submitted to the National Science 15. Glenny A, Hooper L. Why are systematic reviews useful? In: Clark- Foundation. The study found that getting a research son J, Harrison JE, Ismail A (eds). Evidence Based Dentistry for Effective Practice. London: Martin Dunitz, 2003:59. grant significantly depends on chance, because there 16. Garfield E. The literature of dental science vs the literature used is substantial disagreement among eligible reviewers, by dental researchers. In: Garfield E (eds). Essays of an Informa- and the success of a proposal rests on which reviewers tion Scientist. Philadelphia: ISI, 1982:373. happen to be accepted.43 Moreover, there is evidence 17. Relman AS. Journals. In: Warren KS (ed). Coping with the Bio- that complete disagreement between pairs of referees medical Literature. New York: Praeger, 1981:67. assessing the same paper is common.43 In biomedical 18. Worthington H, Clarkson J. Systematic reviews in dentistry: The role of the Cochrane oral health group. In: Clarkson J, Harrison science, the frequency of agreement between refer- JE, Ismail A (eds). Evidence Based Dentistry for Effective Practice. ees was not much better than that which would be London: Martin Dunitz, 2003:97. expected by chance.44 Hence, it appears that objective 19. Chambers DW. Habits of the reflective practitioner. Contact Point and absolute criteria for the evaluation of a paper prior 1999;79:8–10. to publication are not available. Chapter 22 attempts 20. Day RA. How to Write and Publish a Scientific Paper. Philadelphia: ISI, 1979:2. to cultivate the skill of judgment by providing infor- 21. DeBakey L. The Scientific Journal. Editorial Policies and Practic- mation on recognized sources of errors in judgments es: Guidelines for Editors, Reviewers, and Authors. St Louis: Mos- as well as citation analysis, a technique that can be by, 1976:1–3. used to access broadly based scientific assessments of 22. Simonton DK. Creativity in Science: Chance, Logic, Genius, and published works. Zeitgeist. Cambridge: Cambridge University, 2004:85–86. 23. Goodman SN, Berlin J, Fetcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of In- ternal Medicine. Ann Intern Med 1994;121(1):11. 24. Giannobile WV. Editor’s Report for the Journal of Dental Re- References search–2015. http://dentalresearchblog.org/jdr/?p=338. Ac- cessed 18 July 2018. 1. Chargaff E. Triviality in science: A brief meditation on fashions. 25. Tacker MM. Parts of the research report: The title. Int J Prostho- Perspect Biol Med 1976;19:324. dont 1990;3:396–397. 2. Norris SP. Synthesis of research on critical thinking. Educ Lead- 26. Chernin E. Producing biomedical information: First, do no harm. ersh 1985;42:40–45. In: Warren KS (ed). Coping with the Biomedical Literature. New 3. Kurland DJ. I Know What It Says—What Does It Mean? Critical York: Praeger, 1981:40–65. Skills for Critical Reading. Belmont, CA: Wadsworth, 1995:164. 27. Day RA. How to Write and Publish a Scientifc Paper. Philadelphia: 4. Butler CB. New ways of seeing the world. In: Butler CB (ed). Post- ISI, 1979:72. modernism: A Very Short Introduction. New York: Oxford Univer- 28. Cialdini RB. The science of persuasion. Sci Am 2001;284:76 –81. sity, 2002:37–42. 29. Gross AG. The Rhetoric of Science. Cambridge: Harvard Univer- 5. Bronowski J. The common sense of science. In: Bronowski J (ed). sity, 1990. The Common Sense of Science. New York: Vintage, 1967:97–118. 30. Gopen G, Swan J. The science of scientific writing. Am Sci 6. Chambers DW. Lessons from badly behaved technology transfer 1990;78:550–558. to a professional context. Int J Technol Transfer Commercialisa- 31. Strunk W, White EB The Elements of Style, ed 4. Boston: Allyn tion 2005;4:63. and Bacon, 1999. 7. Chambers DW. Changing dental disease patterns. Contact Point 32. Zinnser W. On Writing Well, ed 2. New York: Harper and Row, 1980. 1985;63:1–17. 33. Walton D. Argument as reasoned dialogue. In: Walton D (ed). 8. Fish W. Framing and testing hypotheses. In: Cohen B, Kramer IRH Informal Logic: A Pragmatic Approach, ed 2. New York: Cambridge (eds). Scientific Foundations of Dentistry. London: William Heine- University, 2008:1–34. mann, 1976:669. 34. Toulmin SE,The Uses of Argument, updated ed. Cambridge: Cam- 9. Perkins DN, Allen R, Hafner J. Difficulties in everyday reasoning. bridge University, 2003. In: Maxwell W (ed). Thinking: The Expanding Frontier. Philadel- 35. Lang TA. How to Write Publish and Present in the Health Scienc- phia: Franklin Institute, 1983:177. es: A Guide for Clinicians and Laboratory Researchers. Philadel- 10. Zipf GK. Human Behaviour and the Principle of Least Effort. Cam- phia: American College of Physicians, 2010:41. bridge: Addison-Wesley, 1949. 36. Ziman J. Reliable Knowledge: An Exploration of the Grounds for 11. Pratkaris AR, Aronson E. Age of Propaganda: The Everyday Use Belief in Science. Cambridge: Cambridge University, 1978:7. and Abuse of Persuasion. New York: WH Freeman, 2001:38. 37. DeBakey L. The Scientific Journal: Editorial Policies and Practic- 12. Kahneman D. Attention and effort. In: Kahneman D (ed). Thinking, es: Guidelines for Editors, Reviewers, and Authors. St Louis: Mos- Fast and Slow. Toronto: Anchor Canada, 2013:31–38. by, 1976:1. 13. Risen J, Gilovich T. Informal logical fallacies. In: Sternberg RJ, 38. Kuhn TS. The Structure of Scientific Revolutions, ed 2. Chicago: Roediger HL, Halpern DF (eds). Critical Thinking in Psychology. Chicago University, 1970:184. Cambridge: Cambridge University, 2007:110–130. 39. Fisher RA. The Design of Experiments, ed 8. Edinburgh: Oliver & 14. Garfield E. Current comments: Is the ratio between number of Boyd, 1953:184. citations and publications cited a true constant? Curr Contents 40. Ioannidis JP. Why most published research findings are false. 1976;6:5–7. PLoS Med 2005;2:e124.

REASONS FOR STUDYING CRITICAL THINKING

1 1

1 1 1 1 1 1 41. Maxwell N. Articulating the aims of science. Nature 1977;265:2. 43. Cole S, Cole JR, Simon GA. Chance and consensus in peer review. 42. Beveridge WI. The Art of Scientific Investigation. New York: Vin- Science 1981;214:881. tage, 1950:106. 44. Gordon M. Evaluating the evaluators. New Sci 1977;73:342.

12 IIIIIndexndexndexndexndex

Page numbers followed by “f” denote figures; those followed by “t” tables; and those followed by “b” denote boxes

by analogy, 76 Bias A from authority, 89–91 assimilation, 320 Abduction, 76, 84–85 chains, 82 availability, 338 Aberrant behavior, 17–18 deductive, 77 commission, 320 Aberration of logic, 85 definition of, 74 confirmation, 246–248, 320 Absorbance errors, 289 formal analysis of, 83 definition of, 265 Abstract inductive. See Inductive arguments. detection, 270, 306 function of, 34–35 standards for assessing, 74 information, 267 structured, 35–36 tu quoque, 122 observational error, 307 Academic science, 23, 315, 328 Argumentation maps, 316–318, 317f reporting in, 118 Acceptance date, for scientific Aristotle, 40–42 response, 38, 307 papers, 34 Arrows, in illustrations, 187 results affected by, 166 Accountability, 28 Articulation, 46 sampling, 307 Accuracy Artifact, 165, 181 selection, 269, 306 of measurements, 155, 201b Assertability question, 35, 76 treatment allocation, 307 of ratios, 161 Assertions, 75 unintentional, 18–19 of scientific writing, 316 Assessment, 198 volunteer, 306 precision versus, 164, 164f Assimilation bias, 320 withdrawal, 306 Active control, 278–279 Association, 257 Bimodal distribution, 161 Active reading, 32–33 Assumptions, 271–272 Binomial coefficient, 134 Ad hominem, 122 Audience, 45–46, 50–51 Binomial distribution, 133, 150 Ad populum, 123 Auriculotherapy, 121 Binomial effect size display, 181 Adaptive sampling, 291 Author(s) Biologic variation, 154, 208 Advanced Google search, 72, 72f goal for, 4 BioMed Central, 62 Affective error, 320 institutional affiliations of, 34 Bivalent experiments, 292 AGREE II appraisal tool, 59 of scientific papers, 33–34 Bivariate frequency distribution Agreement indices, 210 ordering of, 34 tables, 142. See also Alternative health industry, 119 referees versus, 7–8 Contingency tables. Alternative hypothesis, 132 significance of, 34 Black-box approach, 89 Altmetrics, 65 Authority, 53, 89–91 Blinding, 284–285 AMSTAR, 58 Author-level citation-based metrics, Blocking, 293 Analogy, 87–88 65 Blunders, 167 Analysis of variance. See ANOVA. Auxiliary hypotheses, 92 Bohr model, 87–88 Analytic variation, 154 Availability bias, 338 Books, 60 AND operator, 68 Availability heuristic, 321 Boolean operators, 67–68, 69f Anecdotal evidence, 110 Axelrod, Robert, 16 Burt, Cyril, 16–17 ANOVA analysis of errors by, 171–172 factorial design analysis, 296 B C Freidman two-way, 158t Babbage, Charles, 17 CADTH. See Canadian Agency for one-way, 159t Bandwagons, 52 Drugs and Technologies in reliability estimations, 212 Bar graphs, 191–192 Health. two-factor, 159t Bayesian approach, 312–313 Calibration errors, 170–171 units of measurement, 160 Bean machine, 168 Canadian Agency for Drugs and Antecedent, 91 Begging the question, 124 Technologies in Health, 62, 62b Applied kinesiology, 121 Benevolent behaviors, 25–26 Canadian Institutes of Health Applied research, 280 Benford’s law, 310 Research, 63 Argument(s) Bernard, Claude, 111 Canons of induction, 99

397 INDEX

Carry-across effect, 298 Clinical studies, 22 Correlational cross-sectional study, Cartesian skepticism, 76 Cochrane Collaboration, 58 258, 258t Case study, 250, 254–255 Cochrane Database of Systematic Cost-effectiveness analysis, 345–347, Case-control designs, 266–268, 267f Reviews, 59 348f Case-control study, 237 Cochrane Library, 3 Counterbalanced designs, 297–298 Case-series analysis, 255 Coefficient of determination, 262 Covariates, 263 Categorical statements, 79–80 Coefficient of variation, 208–209 CPGs. See Clinical practice guidelines. Categorical syllogisms, 80–82 Cognitive dissonance, 48–49 Cranial osteopathy, 121 Causal chain, 102f, 103 Cognitive miser, 2 Cranks, 114–115 Causal conclusions, 244 Cognitive response, 49–50 Credibility, 25, 43, 47, 90, 250 Causal factors, 100 Cohen kappa, 259–260 Critical theory, 251 Causal reasoning, 321 Cohort, 268 Critical thinking Causality, 98 Cohort studies, 237, 268–269 decision making and, 331–332 Causation Cold fusion, 19 description of, 1, 32, 315–316 assessment of, 100–101, 106–107 Collaboration, 16 Crossover design, 297t, 297–298 confounding variables and, 101– Collective judgment, 328 Crossoveritis, 298 106 Commission bias, 320 Cross-sectional designs, 238 correlation and, 270 Common sense, 1, 13, 98 Cross-sectional survey, 258t, 258–265 criteria for, 105–106 Common theory-evidence Cross-validation, 88–89 direct, 102 relationships, 112–114 Cut-off lines, 181 indirect, 102 Communalism, 14 patterns of, 102–105 Comparisons, 176 pleiotropy, 102–103 Complementary and alternative D problems in determining, 270–272 medicine, 115–116 Darwin, Charles, 332 Sackett’s diagnostic tests for, Complex etiology, 104–105 Data. See also Results. 105–106 Compliance of subjects, 285 analysis of, 182–183, 310 Cause-and-effect hypotheses, 97–100 Composition fallacy, 122–123 derived measures for, 180–181 Cause-and-effect relationships, 104f Concept mapping, 316 from experiments, 285–286 convincing evidence for Conceptual model, 88 hiding of, 193f, 194 establishing, 106 Concurrent validity, 156 magnifying a small amount of, ends and means, 107–108 Conditional probability, 130, 205–206 194, 195f interpretation of, 261 Conditional statements, 91–94 manipulation of, 180 investigation of, 105–106 Confidence intervals, 147–150, 169 normalization of, 180 Cause-effect correlations, 125, 261 Confidence limits, 148, 168 presentation of, 178–180 Central limit theorem, 168 Confirmability, 251 qualitative, 251 Central persuasion, 47 Confirmation bias, 246–248, 320 quality of, 175–176 Central tendency measures, 184–185 Confounding, 270–271 unidimensional, 195–196 Chains, 82, 103 Confounding variable, 101, 260, 263 Data density index, 190, 194 Chartjunk, 188 Conjunctions, 102f, 103 Data reporting Cheating, 18 Consequences-based validity, 156 conventions in, 186 Chiropractic treatment, 113 Consequent, 91 minimum requirements for, 183 Chi-squared test, 158t Consistency, 51–52 significant digits, 186 Chronologic approach, 44 CONSORT guidelines, 36 Databases, 66 Cialdini principles of influence, Construct-based validity, 156, 244 Data-ink ratio, 190 51–53 Content-based validity, 156 Decision analysis, 201b, 202f Cicero, 43 Contiguity, 99 Decision balance sheet, 336 Circular reasoning, 124 Contingency tables, 142–143, 156f Decision making Citation analysis, 10, 24, 34, 63–65, Continuous response variables, 284 critical thinking and, 331–332 326–329 Continuous variables, 145 decision tree analysis for, 347, Citation frequency, 329 Contrast, 51 350–351 Citation tracking, 71 Contributory causes, 100–101 description of, 86 Clarity, of scientific writing, 316 Controls, 277–279, 293 evidence-based, 335, 350–351 Classical probability, 129 Controversial care, 121 medical, 335 Cleveland’s hierarchy of graphical Convenience sample, 242 one-reason, 332, 351 perception, 190–192 Conventionalism, 29, 95 in patient care, 333–350 Clinical decision making, 331–333. Convincing evidence, 106 Decision tree, 336–337 See also Decision making. Cooperation, 16 Decision tree analysis Clinical dentistry fallacies, 124–125 Correlation decision-making uses of, 350–351 Clinical diagnosis–decision causation and, 270 definition of, 335 algorithm, 203f description of, 257, 260–261 economic analysis and, 350 Clinical judgment, 315, 322–323 regression versus, 261–264 example of, 343 Clinical outcomes, 338 Correlation coefficient, 155, 172, limitations of, 347–350 Clinical practice guidelines, 57 209–210, 261–262 steps involved in, 335–345 Clinical problem, 336 Deduction, 42

398 Deductive argument, 77 Ecologic study, 265–266, 266f Evidence-based decision making, Deductive logic Economic analysis, 345–347, 350 335, 350–351 description of, 76–78 Editorial consultants, 4 Evidence-based dentistry, 56, 56f, 335 errors in, 82–83 Editors, 5–6 Evidence-based medicine, 110–111 in scientific articles, 77–78 Effect-size approach, 304–305 Examiner variability, 209–214 syllogisms. See Syllogisms. Einstein, Albert, 17, 93 Existential assumption, 81 Degrees of freedom, 149, 185, 303– Elementary outcomes, 129 Expected utility value, 341–344, 349 304 Emergent design, 250 Experiment(s). See also Research. Dentistry, 1–4 Emotional appeal, 42 bivalent, 292 Dependability, 251 Endogenous research, 251 controls, 277–279 Dependent variables, 274–275 Ends and means, 107–108 data quality, 285–286 Descriptive statistics, 128, 305 Enthymeme, 42 dependent variables, 274–275 Descriptive study, 237 Epidemiologic measures, 245. See also functional, 292 Detection bias, 270, 306 Prevalence. independent variables, 274–275 Determinate error, 164, 165–167 Epistemology, 2 measurement, 279–280 Developmental research, 280 Equivocation, fallacy of, 45, 316 random, 129 Diagnosis Error(s) requirements for, 276–280 algorithm for, 203f analytic variation, 154 research, 280–281 decision analysis for, 201b ANOVA used for analysis of, tactics, 281–286 principle of, 200 171–172 validity of, 280 process of, 201–204 assignable causes of, 167 variables, 286 Diagnostic rates, 207t biologic variation, 154 Experiment design Diagnostic tests and measurements blunders, 167 counterbalanced, 297–298 decision analysis, 201b calibration, 170–171 crossover, 297t, 297–298 description of, 200 causes of, 165 description of, 288 reliability of, 206–214 data analysis, 310 error management, 288–291 uses of, 204 in deductive logic, 82–83 factorial, 295–296 Diagnostic tool, 204–206 determinate, 164, 165–167 matched-group, 293 Diagrams, 187 distributing of, 289 pre-experimental, 298–299 Dichotomous response variable, 284 indeterminate, 164, 167–169 quasi-experimental, 298–299 Digital photography, 174 law of propagation of, 169–170 randomized controlled trials, 235– Digression, 50 management of, 288–291 236, 291–292, 299–300 Direct causes, 102 measurement of, 289–290 split-mouth, 298 Discrete variables, 130, 145 method of least squares, 168f, 169 true, 298–299 Discussion/conclusions section, of normal law of, 167–168 Experts, 89–90 scientific papers, 38 observation-related, 165 Explanation, 108 Disease of combined measurements, Explanatory data analysis, 182 description of, 200 169–170 Explanatory trials, 238 pretest probability of, 202 personal, 166 Exploratory data analysis, 182–183 Disinterestedness, 14–15 random, 164, 167, 279 Exploratory research, 22, 280 Disjunctive syllogism, 82, 124, 135 reducing the size of, 290–291 Exposure measurement error, Dispersion measures, 185–186 risk of, 133 264–265 Dispositio, 44 standard. See Standard error. External validity, 242–244, 298 Distribution systematic, 164, 165–166, 279 Extrapolation, 292 a priori, 131 type I, 138, 138t bimodal, 161 type II, 137, 138t binomial, 133, 150 units as cause of, 310 F frequency, 131, 134 Error sum of squares, 169 Face validity, 156 Gaussian, 141, 146, 157, 167 Estimation, 149–150 Facet approach, 44 normal. See Normal distribution. Ethical appeal, 42–44 Factorial designs, 295–296 Poisson, 140–142 Ethnography, 251 Failure fallacy, 125 probability, 133–134 Ethos, 42–44 Fallacies reference, 133 Event, 129 genetic, 123 Student, 149 Evidence of affirming the consequent, 91 t, 149 anecdotal, 110 of apples and oranges, 123–124 uniform, 145 common theory and, relationships of biased statistics, 305–307 Doctrinaire rules-based approach, between, 112–114 of clinical dentistry, 124–125 111 comprehensiveness of, 116 of composition, 122–123 Double-blind treatment/trial, 284, convincing, 106 of concomitant variation, 260 291 fallacies of, 75–76 of equivocation, 45, 316 DTA. See Decision tree analysis. incomplete, 86t of evidence search and insufficient or inappropriate, 75–76 interpretation, 320 level of, 3t of existential assumption, 81 E line of, 23 of failure, 125 EBD. See Evidence-based dentistry. Evidence pyramid, 56, 57f of informal logic, 121–125

399 INDEX

of insufficient or inappropriate Heterogeneity of treatment effects, principles of, 85–91 evidence, 75–76 237, 300 Inductive arguments of insufficient statistics, 302–303 Heuristics, 50–51, 251, 321–322, 325, analogy, 87–88 of no evidence, 75 331–332 argument from authority, 89–91 of origin, 122 High-impact research, 15–16 comprehensiveness of evidence of special pleading, 122 H-index, 64–65 regarding, 116 of success, 124–125 Histogram, 192. See also Relative criterion for acceptability of, 116 UFO, 124, 307 frequency histogram. logic of criticism of, 95–96 Fast and frugal heuristics, 331–332 Historical-controlled trials, 237 models, 88–89 FFH. See Fast and frugal heuristics. Homeopathy, 113 statistical inference as, 302–305 Figures. See Illustrations. Hume, David, 98–99, 261 Inductive logic File drawer problem, 309 Humility, 15 additional evidence in, 85–86 Fisher exact test, 158t Hypercompetition, 27–28 arguments. See Inductive Focal infection theory/hypothesis, Hypothesis arguments. 2, 321 acceptance of, 24 description of, 76, 85 Follow-up (cohort) design, 268–269 alternative, 132, 307 Inferential statistics, 128, 305 Formal deductive arguments, 77 auxiliary, 92 Infinite dilution, 19 Franklin, Benjamin, 333, 350 cause and effect, 97–100 Informal logic, 77 Fraud, 17–18, 37 comprehensiveness of, 93 Information bias, 267 Freidman two-way ANOVA, 158t criticizing of, 94 Information resources Frequency distributions, 131, 134. See definition of, 91 citation analysis, 10, 24, 34, 63–65, also Contingency tables. description of, 23 326–329 Frequentist probability, 129 disproving of, 91–92 description of, 3–4 Frequentists, 338 evaluation of, 93 Information searches Functional experiments, 292 internal consistency of, 93 Advanced Google, 72, 72f null, 132, 135, 308–309 Boolean operators, 67–68, 69f objections to, 94b conducting of, 65–72 of no effect, 312 databases, 66 G simplicity of, 93 defining of question for, 65–66 Galton board, 168, 168f successful predictions, 93 Google Scholar, 70–72, 71f Gaussian distribution, 141, 146, 157, testability of, 93, 116 help in, 55–56 167 Hypothesis testing keyword, 66–67, 69f Generality, 46–47 confidence intervals used to parentheses in, 69 Generalizability, 252 predict results of, 149 phrases in, 69 Genetic fallacy, 123 description of, 88–89, 91 PubMed, 69–70, 70f Goodness of fit, 138–142 steps involved in, 131–137 resources used in, 66 Google Scholar, 70–72, 71f Hypothetical inference, 91–94 subject headings, 66–67, 69 Graphs techniques for, 66–69 bar, 191–192 tips for, 72–73 Cleveland’s hierarchy of graphical I truncation, 68–69 perception, 190–192 ICER. See Incremental cost- Information sources deception in, 192–196 effectiveness ratio. books, 60 description of, 187 Idealism, 21 filtered, 56 exploratory data analysis use of, Illustrations grey literature, 60, 62, 62b, 66, 86 183 arrows in, 187 guidelines, 59, 59b layering of, 189 deception in, 192–196 journals, 60 parallelism of, 189 description of, 186–187 knowledge synthesis reviews, presentation techniques for, graphs. See Graphs. 57–59, 59b 187–188 presentation techniques for, open access literature, 62–63 separation of, 189 187–188 overview of, 56–57 strategies for using, 188–189 Impact factor, for journals, 64 peer review, 60 Tufte’s evaluative ratios for, Incidence, 245 pre-appraised clinical tools, 59–60 189–190 Incomplete evidence, 86t selection of, 66 zero value of the ordinate, 194, Incremental cost-effectiveness ratio, systematic reviews, 57–59 195f 346, 346t Inhibitors, 283 Green open access, 63 Independent events, 130 Insufficient but redundant part of Grey literature, 60, 62, 62b, 66, 86 Independent variables, 274–275 an unnecessary but sufficient Grounded theory, 250–251 Indeterminacy principle, 165 condition, 100 Groupthink, 52 Indeterminate errors, 164, 167–169 Insufficient or inappropriate Guidelines, 59, 59b Indirect causes, 104 evidence, 75–76 Induction Integrity, 37–38, 173–175 Bayesian approach to, 312–313 Intention-to-treat analysis, 285 H canons of, 99 Interaction effect, 296 Health-state utility, 340–341 description of, 42 Interactions, 102f, 103 by enumeration, 302 Interestingness, 47

400 Inter-examiner comparison, 209 deductive. See Deductive logic. MIBBI. See Minimum information Internal validity inductive, 76, 85–91 for biological and biomedical description of, 239, 244, 298 informal, 77 investigations. threats to, 239–241 reasons for studying, 84 Micrograph, 187 International Committee of Medical in scientific writing, 316 Mill, John Stuart, 99 Journal Editors, 33 semi-formal, 77 Minimum information for biological Interpolation, 292 Logistic regression, 264 and biomedical investigations, Interquartile range, 185 Logos, 42 37 Interval scale, 159, 159t Longitudinal study, 240 Misclassification, 264–265 Intra-examiner comparison, 209 Mixed hypothetical syllogism, 81 Introduction section, of scientific Mode, 184, 184f papers, 36 M Models, 89, 320–321 Inventio, 41–44 Magnitude, 46 Modus tollens, 92 Irreproducibility epidemic, 26 Make-sense epistemology, 2 Moral algebra, 333, 350 Malevolent behaviors, 25–26 Multidisciplinary collaborative Managing editors, 6 research, 42 J Mann-Whitney U test, 158t Multiple linear regression, 263 Joint stretching, 88 Mapping, 176–178 Mutually exclusive events, 129 Journals Markov chain Monte Carlo, 313 as information sources, 60 Matched-group experiment designs, citation analysis of, 64 293 N description of, 2, 4 Matching, 293 Narrative reviews, 58, 58b impact factor for, 64 Materials and methods section, Naturalistic inquiry, 250 open access publishing models 36–37 Necessary condition, 99–100 for, 62 McLuhan, Marshall, 188 Negative control, 278 refereed, 60, 61b MCMC. See Markov chain Monte Negative predictive value, 201b, 207t reporting practices of, 27 Carlo. Negative results papers, 95 Judgments McNemar chi-squared test, 158t Neyman-Pearson approach, 311 balanced, 318–319 Mead, Margaret, 19 Nominal scale, 157 clinical, 315, 322–323 Mean, 184, 184f Nondisjoint outcomes, 129 scientific, 315 Means and ends, 107–108 Nonparametric statistics, 140 under uncertainty, 319–323 Measurements Nonparametric tests, 157, 158t Judicial precedent, 87 accuracy of, 155 Normal distribution Jujitsu method, 282 aggregated, 181–183 advantages of, 150 choice of, 175 definition of, 146 combined, 169–170 sampling from, 148–149 K definition of, 152 shape of, 184 Kappa, 210–211 diagnostic. See Diagnostic tests standard, 147 Keyword searches, 66–67, 69f and measurements. Normal law of error, 167–169 Knowledge synthesis reviews, 57–59, errors of. See Error(s). Normalization, 180 59b operational definitions, 152–153 Nosographic rates, 207t Koch’s postulates, 105 precision of, 154, 164, 164f, 209 NOT operator, 68 Kruskall-Willis test, 158t reliability of, 172, 206–214 Novel indices, 181 resolution, 153–154 Null hypothesis, 132, 135, 308–309 results of. See Results. Numerical method, 110 L scales for comparisons among. See Numerical quality, 176 Lack of objectivity, 116–117 Scales. Nutrition quackery, 121 Law, analogy and, 87 true score theory of, 154 Law of cognitive response, 49–50 units of, 160–161 Law of propagation of errors, 169– validity of, 155–157 O 170 Measures of central tendency, Objective information, 198 Law of small numbers. See Poisson 184–185 Objectivity, 7, 16–19, 116–117, 247 distribution. Measures of dispersion, 185–186 Obsequience bias, 307 Letter to the editor, 24 Mechanism, 88 Observation(s) Level of evidence, 3t Median, 184, 184f authority versus, 89 Level of significance, 133 Medical decision making, 335 case study design affected by, Librarians, 55 MEDLINE, 67 254–255 Lie factor, 46, 190, 195 Mendel, Gregor, 138–140 classification of, 22, 248 Life history, 251 Mental processes, 3 discoveries through, 245 Liking, 52–53 Mere, Chevalier de, 129 error caused by, 165 Logarithmic scales, 193 MeSH, 35, 67 limitations of, 21 Logic Meta-analysis, 57, 149, 323, 325–326 selection of, 21–22, 249 abductive, 84–85 Method of difference, 277 unreliability of, 307 aberration of, 85 Method of least squares, 168f, 169 Observational error bias, 307 abuses of, 121–125 Observational reactivity, 249

401 INDEX

Observational studies, 237, 271 Percent reproducibility, 212 Psychologic studies, 3 Observation-description strategy Peripheral persuasion, 47 Ptolemy, Claudius, 17 advantages of, 248–249 Personal accountability, 28 Public Library of Science, 62 case study, 254–255 Personal errors, 166 Publication case-series analysis, 255 Personal experience, 117 authors, 4, 7–8 in clinical interventions, 253–255 Perspective mania, 196 description of, 23 epidemiologic measures, 245 “Persuading with pap,” 187–188 editors, 5–6 operational considerations for, 248 Persuasion, 46–49, 173 nonrefereed journals, 5 technologic influences, 249 Phenomenology, 251 readers of, 8 weaknesses associated with, 249 Physical model, 88 refereed journals, 5 Occam’s razor, 93, 135 PICO, 56, 65–67, 336, 339 referees, 5–8 Odds ratio, 259 Placebo effect, 117–118, 124 Publishing pressure, 17, 27 One-reason decision making, 332, Plato, 41 PubMed, 69–70, 70f 351 Pleiotropy, 102f, 102–103 PubMed Central, 62 One-tailed tests, 137, 311 Point-of-care tools, 59–60 Pure hypothetical syllogism, 81 One-way ANOVA, 159t Poisson distribution, 140–142 Pure research, 280 One-way sensitivity analysis, 344 Population Purposive sampling, 250 Online profiles, 65 definition of, 184, 305 Open access, 62–63 of known dispersion, 147–148 Open data, 62–63 target, 305, 307 Q Open science, 62–63 variability in, 185 QALY. See Quality adjusted life years. Open-blind study, 284 whose dispersion is not known “a QAPY. See Quality adjusted Open-mindedness, 74 priori,” 149 prosthetic years. Openness, 25–29 Population overlap, 308 Quackery, 115, 120–121 Operational definitions, 152–153 Positive control, 278–279 Quacks OR operator, 67–68 Positive predictive value, 207t complementary and alternative ORCID, 65 Positivism, 29, 95 medicine, 115–116 Ordinal scale, 157–159, 158t Post hoc, 123 definition of, 114–115 Organizational schemes, 44 Posttest likelihoods, 201b lack of objectivity, 116–117 Organized skepticism, 14, 24 Posttest probability, 207t personal experience used by, 117 Origin fallacy, 122 Pragmatic model, of reasoning, 8 strategies and tactics, 118–119 Originality, 9, 15, 280 Pragmatic trials, 238 Qualitative data, 250 Outcome(s) Pre-appraised clinical tools, 59–60 Qualitative methods, 252–253 in dentistry, 339t Precision, 154, 164, 164f, 209, 279 Qualitative observations, 22–23 description of, 129–130 Predictive validity, 155 Qualitative research, 251–252, 252t patient-centered, 339–340 Pre-experimental designs, 298–299 Qualitative studies, 29, 95 probability of, 338 Premises, 74, 82 Quality adjusted life years, 341 Outcome measures, 284 Pretest probability, 201–202, 207t Quality adjusted prosthetic years, Outcome of interest, 105 Prevalence, 201b, 205, 245 341–342, 345 Outcome variables, 22, 263 Price index, 326 Quality of results, 175–176 Overtreatment, 125 Principal principle, 48 Quantitative observations, 22–23 Principle of Least Effort, 2 Quantitative research, 250, 252t Principle of social proof, 52 Quasi-experimental designs, 236– P PRISMA, 58, 59b 237, 298–299 P value, 135 Probabilistic cause, 100 Quincunx, 168 Paired t test, 159t Probability Pairing, 293 calculations, 129–130 Papers. See Scientific papers. classical, 129 R Parallelism, 189 conditional, 130, 205–206 Random error, 164, 167, 279 Parameter, 140, 274 conditions associated with, 205 Random experiment, 129 Parametric tests, 157 definition of, 129 Random sampling, 250 Parent population, 305 frequentist, 129 Random variable, 130 Parentheses, 69 history of, 129 Randomization, 275–276, 291–292 Pareto’s law, 167 measuring, 205 Randomized block test, 159t Participatory action research, 251 subjective, 129 Randomized controlled trials, 235– Pascal, Blaise, 129 Probability density, 145 236, 291–292, 299–300 Passive voice, 16, 43 Probability distribution, 133–134 Range, 185 Pathologic science, 19 Problem solving, 88 Ratio, 161, 170, 181 Pathophysiologic approach, 110–111 Profiles, online, 65 Ratio scale, 159t, 159–160 Pathos, 42 Proof by exclusion, 124 Rational appeal, 42 Patient-centered outcomes, 339–340 Proof of elimination, 124 Rationality, 74 Pearson correlation coefficient, 209 Propagation of errors theorem, Rationalization trap, 48 Peer review, 60 169–170 Readers, 8 Peer-review ratings, 5 Proportion agreement, 210 Real world research, 250–251 Percent agreement, 210 Prospective cohort study, 238 Realism, 21

402 Reasoning study designs, 238–239 Science, 2, 10 by analogy, 87–88 subjects for, 238–239, 241, 285 Scientific arguments, 8 types of, 8 variables, 286 Scientific discovery, 87–88 Reciprocation, 51 Research proposal, 233, 235 Scientific judgments, 315 Recognition, 15 Research question, 132 Scientific literature Reconstruction experiment, 290 Residual sum of squares, 169 citation analysis used to evaluate. Red herring, 50 Residuals, 262–263 See Citation analysis. Reductionism, 122 Resolution, 153–154 growth rate of, 3 Referee(s) Response bias, 38, 307 meta-analysis of, 149 authors versus, 7–8 Response variables, 263 review of, 4 description of, 5–6 Results. See also Data. Scientific method Refereed journals, 4, 60, 61b bias effects on, 166 common sense and, 13 Reference distribution, 133 comparisons, 176 hypothesis, 23 Reflexology, 121 confidence intervals used to objectivity. See Objectivity. Regression predict, 149 observations, 21–23 correlation versus, 261–264 frequency of, 136 statistics and, 301 description of, 118, 241 goodness of fit, 138–142 storybook version of, 20–25 Regression models, 272 in illustrations. See Illustrations. Scientific misconduct, 28 Relative frequency histogram, 131, integrity, 173–175 Scientific papers 131f, 133, 134f, 136f, 141f location of, 136 abstract, 34–36 Relevance, 176–178 presentation of, 173–196 acceptance date for, 34 Reliability quality of, 175–176 active reading and, 32–33 definition of, 206 relevance of, 176–178 authors of, 33–34 of measurements, 172, 206–214 in tables, 186 collaboration on, 16 validity versus, 207, 208f, 213 Results section, of scientific papers, comparisons in, 176 Reliability coefficient, 212 37–38 components of, 33–38 Repeatability index, 210 Retaliation, 16 date of acceptance for, 34 Repeated conjunction, 99 Retrospective cohort study, 238 date of submission for, 34 Repetition, 25, 28 Rhetoric definition of, 3 Replication, 213 canons of, 41–46 discussion/conclusions section Reporting bias, 118 decision making and, 40 of, 38 Reporting practices, 27 dispositio, 44 evaluator of, 10 Representativeness heuristic, 322 elocutio, 44–45 importance of, 10 Reproducibility inventio, 41–44 introduction section of, 36 description of, 10, 18 medieval versus contemporary, materials and methods section of, lack of, 26 43b 36–37 lack of incentives for, 26–27 overview of, 40–41 multi-authored, 16 openness and, 25–29 in scientific papers, 23 objectivity in, 7 personal accountability in, 28 Risk factors, 270 originality of, 9 plans to enhance, 28 Risk of error, 133 publication of, 23 Reproducibility crisis purposes of, 44, 173, 235 comments regarding, 28–29 questions to ask about, 9–11 defective reporting of S reproducibility of, 10 methodology, 27 Sackett’s diagnostic tests for results section of, 37–38 hypercompetition, 27–28 causation, 105–106 rhetoric in, 23 lack of incentives, 26–27 Sample style of, 23 multifactorial reasons for, 26–28 definition of, 133 submission date for, 34 seriousness of, 26 from normal distribution, 148–149 suitability of, 4 Reproducibility ratio, 212 from population of known systematic approach to analyzing, Research. See also Experiment(s). dispersion, 147–148 2 discovery-based strategies, 235 from population whose dispersion title of, 33 experimental, 280–281 is not known “a priori,” 149 truthfulness of, 9–10 external validity of, 242–243 Sample size, 148f, 265, 303–304 writing style for, 7 funding of, 18, 232–234 Sampling, 243, 250 Scientific taste, 10 internal validity, 239–241 Sampling bias, 307 Scientific theories, 87 niche for, 233–235 Sampling space, 129 Scientific writing, 45, 316 non-experimental design of, 235 Scales Scientist behavior open access, 63 deceptive uses of, 192–194 aberrant, 17–18 pre-experimental design of, 236 interval, 159, 159t collaboration, 16 problems associated with, 242–243 logarithmic, 193 everyday life, 14 qualitative, 251–252, 252t nominal, 157 motivators of, 15 quantitative, 250, 252t ordinal, 157–159, 158t objectivity. See Objectivity. quasi-experimental design of, purpose of, 157 principles of, 14–15 236–237 ratio, 159t, 159–160 traditional portrayal of, 15 real world, 250–251 Scarcity, 53 Segmented regression analysis, 299

403 INDEX

Selection bias, 269, 306 Subjective information, 198 True-negative rate. See Specificity. Self-censorship, 52 Subjective probability, 129 True-positive rate. See Sensitivity. Semi-formal logic, 77 Subjects, 238–239, 241, 285 Truncation, 68–69 Semi-log plot, 193–194 Submission date, for scientific Trust, 16, 91 Sensitivity, 201b, 207t papers, 34 Tu quoque argument, 122 Sensitivity analysis, 343–345 Success fallacy, 124–125 Tufte’s evaluative ratios for graphs, Sign test, 158t Sufficient condition, 99–100 189–190 Significance level, 133 Sum of squares, 169 Two-factor ANOVA, 159t Significant digits, 186 Suppressed premises, 82 Two-tailed tests, 137, 311 Silver amalgam replacement, 121 Surrogate variables, 22 Two-way ANOVA, 158t Simpson paradox, 181 Surveys, 38, 238, 258–265 Type I error, 138, 138t Snow job, 51 Syllogisms Type II error, 137, 138t SOAP format, 198 categorical, 80–82 Soapbox effect, 50 categorical statements, 79–80 Social validation, 52 definition of, 78–79 U Special pleading fallacy, 122 disjunctive, 82, 135 UFO fallacy, 124, 307 Specificity, 201b, 207b mixed hypothetical, 81 Ullrich’s Index of Periodicals, 4 Split-mouth design, 298 pure hypothetical, 81 Uncertainty Square of opposition, 79, 79f Synthetic approach, 282 description of, 149, 279–280, Standard(s) Systematic errors, 164, 165–166, 279 337–339 changing of, 9 Systematic reviews, 3, 57–59, 58b judgments under, 319–323 correlational experiments Unidimensional data, 195–196 involving humans, 269–270 Uniform biologic response, 99 of the field, 318 T Uniform distribution, 145 Standard deviation, 154, 170, 185 t distribution, 149 Unintentional bias, 18–19 Standard error, 148, 185–186 t test, 159t Units, 160–161, 309–310 Standard error of measurement, 212 Tables, 186 Universalism, 14 Standard normal distribution, 147 Target population, 305, 307 Utility, health-state, 340–341 Statistical adjustment, 264 Technical quality, 175–176 Utility value, expected, 341–344, 349 Statistical data. See Data. Temporal proximity, 98 Statistical generalization, 302 Temporality, 98–99 Statistical inference Temporary abandonment principle, V as inductive argument, 302–305 281 Valid, 85 description of, 131 Testability of hypotheses, 93, 116 Valid categorical syllogism, 80 error, 137–138 Tests. See Diagnostic tests and Validity features of, 138 measurements; Statistical external, 242–244, 298 hypothesis testing, 131–137 tests. internal. See Internal validity. nonparametric, 140 Textbook errors, 25 of experiments, 280 one-tailed versus two-tailed tests, Theory, 87, 113 of statistical tests, 155–157 137, 311 Threshold approach, to decision reliability and, 207, 208f, 213 Statistical modeling, 270 analysis, 202, 202f Value judgments, 339 Statistical power, 125 Title, of scientific papers, 33 Variability Statistical regression, 241 Transfer method of research, 280 biologic, 154, 208 Statistical tests Transferability, 250–251 from examination equipment and differences in conditions, 323–324 Treatment(s) environment, 208 nonparametric, 158t anecdotal evidence, 110 examiner, 209–214 types of, 158t approaches to, 110–111 reporting, 208–209 validity of, 155–157 cause-effect reasoning and, 105 Variables Statistics complex, 118 confounding, 101, 260, 263 definition of, 128 definition of, 200 continuous, 145 descriptive, 128, 305 ineffective treatments appearing dependent, 274–275 inferential, 128, 305 successful, 117–118 discrete, 130, 145 nonparametric, 140 numerical method, 110 independent, 274–275 ongoing development of, 150 pathophysiologic approach to, outcome, 263 reporting of results, 135 110–111 response, 263 resources for, 143 personal experience about, 117 Variances, 154, 171 rules for, 311 questions about, 111–112 Venn diagram, 129, 130f scientific method and, 301 variability of conditions, 118 Visual analog scale, 340f, 340–341 Stereotype threat, 157 Treatment allocation bias, 307 Volunteer bias, 306 Stratified randomization, 291 Triangulation, 251 Structured abstract, 35–36 Trimming, 17 “Structured discussion,” 38 True biologic variation, 171 W Student distribution, 149 True control, 278 Walton, Douglas, 77, 87 Subgroup analysis, 181 True experimental design, 237, Washout period, 298 Subject heading searches, 66–67, 69 298–299

404 Wilcoxon rank sum test. See Mann- WTP. See Willingness to pay analysis. Whitney U test. Z Willingness to pay analysis, 345–347 z score, 146, 159t, 180 Wisdom of crowds, 24 Y Withdrawal bias, 306 Yoked control, 293 Writing style, 7

405