FOLLOW-UP

How Not To Review Research

GARY E. SCHWARTZ

ost rational scientists agree that maintain his belief drat die phenomenon accurate and informed decision. What we the credibility and integrity of in question is impossible? As I document strive for is seeking the truth as reflected a review of a body of research below, Hyman resorts to (consciously in Harvard's motto "Veritas." M and / or unconsciously) selectively ignor- is that it includes all die important infor- I appreciate Hyman's effort to outline mation, not just the reviewer's favored ing important information dial is incon- some of the possible errors and limitations information. 's review "How sistent with his personal beliefs. in the mediumship experiments discussed Not To Test Mediums" (January/February Selective ignoring of facts is not in The Afterlife Experiments. However, as 2002) is a textbook example of the selec- acceptable in science. It reflects a bias that Hyman emphasizes in his review, I do tive ignoring or dismissing of historical, obviates the purpose of research and disal- "strongly disagree" with him about his procedural, and empirical facts to fits lows new discoveries. I have made die interpretations. The two fundamental dis- statement that the survival consciousness one's preferred interpretation. The result agreements I have with Hyman's argu- hypothesis does account for the totality of is an inaccurate, mistaken, and biased set ments are: of conclusions of the current data. the research data to date. Of course, this does not make the survival hypodiesis the 1. Hyman has chosen to ignore numer- Hyman is a distinguished professor ous historical, procedural, and emeritus from the Department of only or correct hypothesis—my state- empirical (acts that are inconsistent Psychology at the University of Oregon, ment reflects die status of die evidence to with his interpretive descriptions of who has had a longstanding career as a date, not necessarily the truth about the our experiments; and skeptic focused on uncovering potential underlying process. This is why more 2. Hyman has chosen not to acknowl- research is needed. edge the totality of the findings fol- flaws in research. Hyman lowing Occam's heuristic principle as is well skilled in carefully going through Note diat I do not use die word a means of integrating the total set of die conventional checklist of potential "believe" in relationship to the statement. findings collected to daic. sources of experimental errors and limita- This is not a belief. It is an empirical Space precludes my providing a tions in research designs. observation derived from experiments. detailed and diorough commentary here Hyman's overall appraisal of the It is correct that some of die single- illustrating how pervasively Hyman research conducted to date is implied by blind and double-blind studies have ignores and omits important information. his conclusion: "Probably no other weaknesses—we discuss the experimental (An extensive commentary has been pub- extended program in psychical research limitations at some length in our pub- lished on various Web sites, including deviates so much from accepted norms of lished papers as well as in The Afterlife www.openmindsciences.com.) Four sam- scientific methodology as does this one." Experiments. However, these weaknesses ples of important ignored facts are pro- Is Hyman's summary conclusion based do not justify dismissing die totality of vided below. upon a thorough review of the total body the data as mistaken or meaningless. of research? Or does it reflect the system- Quite the contrary, an honest and accu- Selective Ignoring of Historical, atic ignoring of important historical, pro- rate analysis reveals that the data, in total, Procedural, and Empirical Facts cedural, and empirical facts—a cognitive deserve serious consideration. Veritas 1: In his review, Hyman failed to bias used by the reviewer in order to Our research presents all the find- mention the important historical fact that ings—the hits and the misses, the creative our mediumship research actually began Gary E. Schwartz is with the Human aspects of the designs and dieir limita- with double-blind experimental designs. Energy Systems Laboratory, University of tions^—so that the reader can make an For example, die published experiment Arizona, Tucson.

5 8 May/June 2003 FOLLOW-UP

referred to in The Afterlife Experiments as Hyman fails to mention mat NIH, for mation, by chance, could fit multiple sit- "From Here To There and Back Again" example, requires that investigators who ters—an erroneous conclusion that can be with Susy Smith and Laurie Campbell apply for research grants calculate statisti- reached only if we do what Hyman did was completed almost a year before cal power and sample size to determine and accept the information selectively. we conducted die more naturalistic multi- what n is required to obtain a statistically medium/multi-sitter experiments in- significant result. This is accepted scien- Veritas 4: Hyman's conclusion that expe- volving John Edward, Suzanne tific practice and is required for obtaining rienced cold readers can readily replicate Northrop, George Anderson, Anne NIH funding. the kinds of specific information obtained Gehman, and Laurie Campbell. The early Smith-Campbell double-blind studies did not suffer from possible sub- tle visual or auditory sensory leakage or Just as I don't take the claims of the rater bias—and strong positive findings mediums on faith, I don't take the claims were obtained. of the magicians on faith either. Our decision to subsequently conduct more naturalistic designs (which are inherendy less controlled), was made par- tially for practical reasons (e.g., develop- Conclusion: Hyman would rather dis- under the conditions of our experiments ing professional trust with highly visible miss the fact that the highly accurate rat- is mistaken at best and deceptive at worst. mediums) and pardy for scientific ones ings obtained in the single-blind pub- Under experimental conditions where (e.g., we wished to examine under labora- lished study for GD were indeed repli- (a) professional cold readers do not know tory conditions how mediumship is often cated in the double-blind published die identity of die sitters (i.e., cheating is conducted in die field). study, than to admit the possibility that ruled out), and (b) cold readers are not Conclusion: Hyman makes a factually individual differences in sitter characteris- allowed to see or speak with the sitters erroneous criticism when he reports diat tics are an important and genuine factor (i.e., cueing and feedback is ruled out), it double-blind experiments were initiated in mediumship research. is (c) impossible for cold readers to use only late in our research program, and whatever pre-obtained sitter specific therefore makes a serious interpretative Veritas 3: It is curious that among the information they might have obtained, mistake when he decides tliat all the early many examples of readings provided in and (d) impossible for cold readers to use data can be dismissed because they were The Afterlife Experiments, one early subset their feedback tricks to help them get not conducted double-blind. (cluster/pattern) of facts happened to fit information from the sitters. Hyman nicely. It is true that mention of At the two-day meeting I convened in Veritas 2: In an exploratory double-blind the "Big H," a "father-like figure," an Los Angeles of seven highly experienced long distance mediumship experiment "HN sound" would fit Hyman's father professional mentalist magicians and cold where George Dalzell (GD) was one of six like it did the sitter's husband mentioned readers, they all agreed that they could not sitters and Laurie Campbell (LC) was the in the book. apply their conventional mentalist tricks medium, Hyman states "because nothing Hyman chose not to report the fact under these stria experimental condi- significant was found, die results do not that many other pieces of specific infor- tions. However, a vocal subset (Hyman warrant claiming a successful replication mation also reported for the "Big H" did was one of rhe three), made the unsub- of previous findings." not fit Hyman but did fit the sitter pre- stantiated claim that if dicy had a year or However, Hyman minimizes the fact cisely. Moreover, Hyman consistently two to practice, they might be able to fig- that the number of subjects in this failed to report scores of examples from ure out a way how to fake what the medi- exploratory experiment was small («=6). readings reported verbatim in the book ums were doing. More importandy, Hyman fails to cite a that were highly unusual and unique to My response to this vocal subset was important conclusion mat we reached in individual sitters (e.g., John Edward see- simple. It was "show me." Just as I don't me discussion: "If die binary 66 percent ing a deceased grandmother having two take the claims of me mediums on faith, I figure approximates (1) LC's actual ability large poodles, a black one and a white don't take die claims of die magicians on to conduct double-blind readings, cou- one, and die white one "tore up the faith either. I am a researcher. Mentalist pled widi (2) the six sitter's ability, on die house"). magicians who make these claims will average, to score transcripts double-blind, Conclusion: The reason Hyman railed have to "sit in the research chair" and die 66 percent figure would require only to mention these numerous examples is show us that diey can do what diey claim an n of 25 sitters to reach statistical signif- because they contradict the conclusion they can do. icance (e.g., p< .01)." Hyman chose to accept—that the infor- Thus far, the few cold readers who

SKEPTICAL INQUIRER May/June 2003 59 FOLLOW-UP

have made these claims have refused to his preference is to propose mat the set of So what is the truth at the present be experimentally tested. They have findings collected to date must involve a time, based upon the available data? been unwilling to demonstrate in the complex set of subtle cues providing When the totality of die history, proce- laboratory that they can't do what the information in some studies, dures, and findings to date are examined mediums do under these experimental techniques being used in some studies, honestly and comprehensively—not selectively sampled to fit one's particular theoretical bias—somediing anomalous appears to be occurring in die medi- To me the question of whether or not mediums umship research, at least with a select are obtaining anomalous information is a group of evidence-based mediums. Over and over, from experiment to purely scientific one, to be revealed through experiment, findings have been observed a program of systematic research. diat deserve the term extraordinary. In our latest double-blind, multi-center experi- ments, stable individual differences in sit- ters have been observed that replicate conditions; and they have been unwill- rater bias providing inflated scores in across laboratories and experiments. The ing to demonstrate at a later date that some studies, and chance findings in observations are not going away—even with multi-center, double-blind testing. their performance can improve substan- some studies. The idea that mediums tially with practice. might be obtaining anomalous informa- Hyman once told me, "I have no Conclusion: The claim that cold tion that can most simply and parsimo- control over my beliefs." When I asked reading can account for the research niously be explained in terms of die con- him what he would conclude if a perfect findings is not supported when the tinuance of consciousness is presumed large sample multi-center double-blind experimental procedures are honestly categorically to be false by Hyman until experiment was conducted, his response taken into account. proven odierwisc. was, "I would want to see your major I make no such categorical assump- multi-center, double-blind experiment Failure to Integrate Information tions, one way or die other. To me the replicated a few times by odier centers and Appreciate the Process of question of whedicr or not mediums are before drawing any conclusions." Discovery obtaining anomalous information is a This conversation is revealing In most areas of science, no single experi- purely scientific one, to be revealed psychologically. Until multiple perfect ment is perfect or complete. Different through a program of systematic research. experiments are performed and pub- experiments address different conditions Such research must be conducted by mul- lished, Hyman would rather believe that and different alternative explanations to tiple laboratories. The reason for publish- die totality of the findings must be due different degrees. The challenge is to con- ing findings, as diey emerge, is to encour- to some combination of fraud, cold read- nect the dots of the available data and age odier investigators to conduct their ing, rater bias, experimenter error, or integrate the complex set of findings own experiments, and then integrate the chance—even if this requires diat he selectively ignores important aspects of using the fewest number of explanations totality of the findings. the history, designs, and findings in (i.e., Occam's razor). However, the truth is, it is impossible order to hold on to his belief that he Hyman reveals in his review that he to integrate the totality of die findings in (or we) are being "fooled." learned as a teenager that it was easy for any area of science if one selectively (con- Why spend the time and money con- him to fool many people widi palm read- sciously or unconsciously) ignores those ducting multiple multi-center, double- specific findings diat do not fit one's pref- ing. It is also quite easy to fool many peo- blind experiments unless mere are suffi- ple widi fake mediumship, as anyone erences or biases. cient dieoretical, experimental, and social trained in cold reading will tell you. I have reasons for doing so? Scientific Integrity and Changing studied a number of books on cold read- The critical question is, "Is it possible One's Beliefs ing and have taken some classes on cold diat consistent widi die actual totality of reading myself. However, just because it is I admit, quite adamantly, that I do have die data collected to date—viewed histor- possible sometimes to be fooled (espe- one fundamental bias—my bias is ically (e.g., the observations of William cially by die masters of ) doesn't to use the to discover James) as well as across disciplines (e.g., mean diat everyone is fooling you. the truth, whatever it is. Discovering from andiropology to astrophysics)—diat Hyman reluctandy agrees mat it is the truth cannot be achieved through future research may lead us to come to die improbable diat the totality of our find- selective reporting of history, proce- conclusion that consciousness is inti- ings can be explained by fraud. As a result, dures, and data. mately related to energy and information.

6 0 May/lune 2003 SKEPTICAL INQUIRER FOLLOW-UP and diat consciousness, as an expression For example, Hyman's review begins regardless of die specific research area diat of dynamically patterned energy and by stating that I was a professor at Yale is being reviewed or the specific person information, persists in space like die light University for twenty-eight years—die doing die reviewing. from distant stars?" fact is, I was at Yale for twelve years. If the Note diat my argument is not widi This is ultimately an empirical ques- SKEPTICAL INQUIRER had not chosen to Hyman as a person, nor with the tion; it will be answered by data, one way keep Hyman's review secret, and had SKEPTICAL INQUIRER as a publication. or die other. If positive data are asked me to fact check Hyman's review, I My concern is about die process by which obtained—and I emphasize if—accepting would have gladly done so, and therefore Hyman has written his review, and die the data will require diat we be able to enabled both the magazine and the responsibility of SKEPTICAL INQUIRER to change our beliefs as a function of what the data reveal. The Afterlife Experiments reviewer to correct at least the obvious decrease the likelihood that this kind of was written to encourage people to keep errors of fact. Qeariy, iittie mistakes, com- mistaken review will be published in die an open mind about what the future pounded by big mistakes, do not make future. There is a bigger lesson here. It is research may reveal. for a credible publication or review. worth considering, and correcting. I am taking a strong position about Epilogue: What is a Magazine's Acknowledgments accuracy of reporting here not because of Responsibility? the ultimate validity of the survival I diank a number of my colleagues who have If the SKEPTICAL INQUIRER wishes to be hypothesis (i.e., whether it is true or not, graciously taken die time to provide me with viewed as being a credible publication, since that is an experimental question) useful feedback about this commentary. They more like the Philadelphia Inquirer than but because of the nature of scientific include Peter Hayes, Ph.D., Katherine Creath, the National Enquirer, it should take reviewing process itself. Ph.D., Stephen Grenard, Ph.D.. Donald responsibility for fact checking its articles The selective ignoring and omission of Watson, M.D., Emily Kelly, Ph.D., Lonnie and correcting mistakes caused by simple important information cannot be con- Nelson, M A , and Montague Keen. The com- errors and/or the selective ignoring of doned in cither reviewing or publishing. ments provided here are diose of the author, important information. It must be exposed and understood. not necessarily diose of my colleagues.

Hyman's Reply to Schwartz's 'How Not To Review Mediumship Research'

RAY HYMAN

cannot, of course, respond in detail my assessment of the research. I chose to iment with the mediums Susy Smith within the allotted space to each of focus my discussions on those items that and Laurie Campbell that "was com- Schwartz's arguments. Instead, I will Schwartz and his colleagues had empha- pleted almost a year before we con- I sized as the strongest outcomes amongst ducted the more naturalistic multi- comment on his major points and con- clude with a general reaction to his their findings. 1 have refereed and medium/multi-sitter experiments in- rebuttal. reviewed research reports for more dian volving John Edward, Suzanne 1. "Hyman resorts to . . . selectively fifty years for many of the major scien- Northrop, George Anderson, Anne ignoring important information that is tific publications and for major granting Gehman, and Laurie Campbell. The inconsistent widi his personal beliefs." agencies. I applied die same standards to early Smith-Campbell double-blind my evaluation of the afterlife experi- studies did not suffer from possible sub- In preparing my critique of his ments that I have used in my other tle visual or auditory sensory leakage or research program, I not only read The assessments. rater bias—and strong positive findings Afterlife Experiments carefully, I also were obtained." scrutinized in detail every report of his 2. ". . . Hyman failed to mention the research that was available. It was not important historical fact that our medi- possible to discuss each separate piece of umship research actually began with Ray Hyman is professor emeritus of psy- information in my critique. I took each double-blind experimental designs." chology. University of Oregon. E-mail: item into account, however, in making As his example he refers to his exper- rayhym @msn. com.

SKEPTICAL INQUIRER May/June 2003 61