News and Comment Ganzfeld Studies: First Detailed Appraisal Finds Serious Flaws, No Evidence of Psi N THE 11 YEARS since publication have been presented as evidence for psi. Iof the first "ganzfeld" ESP experi­ The first detailed scholarly evalua­ ment in 1974, reporting pro-psi results, tion of the ganzfeld studies has now a series of similar experiments have been been published. The critique, prepared published in the parapsychological liter­ over a period of several years by psy­ ature. Parapsychologists and others chologist (and CSICOP Executive have considered these to be among the Council member) Ray Hyman, of the strongest scientific evidence for the University of Oregon, gives little com­ existence of extrasensory perception. fort to proponents of the ganzfeld Ganzfeld experiments are based on experiments as the best hope for proving the idea that sensory deprivation is con­ psi abilities exist. ducive to the manifestation of psi abili­ "... 1 believe that the ganzfeld psi ties. The research subject is generally data base, despite initial impressions, is isolated from visual and other sensory inadequate either to support the conten­ contact. Then various experiments are tion of a repeatable study or to demon­ carried out to test the subject's ability strate the reality of psi," Hyman con­ to perceive outside information. cludes in his 47-page critical appraisal, For instance, a person undergoing published in the March 1985 Journal perceptual deprivation might be asked of Parapsychology. "Whatever other to "receive" an image from a photo­ value these studies may have for the graph randomly selected from four parapsychological community, they photos and being concentrated on by a have too many weaknesses to serve as "sender" in another location. While the basis for confronting the rest of the doing so he might be requested to ver­ scientific community. Indeed, parapsy­ balize his thoughts, feelings, and images. chologists and others may be doing Afterward, the subject and, in some themselves and their cause a disservice cases, independent judges would be by attempting to use these studies as asked to assess the degree of cor­ examples of the current state of their respondence between the picture and field." the subject's imagery. Positive results The journal follows Hyman's from a number of such experiments analysis with an equally detailed 2 THE SKEPTICAL INQUIRER, Vol. 10 response by parapsychologist Charles ganzfeld study he knew of. Honorton disputing many of his con­ As a result, Hyman evaluated all clusions. Honorton carried out the first 42 studies reported from 1974 through reported ganzfeld ESP experiment in 1981. Honorton classified 23 of them 1974, and his subsequent ganzfeld as having achieved significance as studies include some of the most posi­ evidence of psi. This amounts to a tive (pro-ESP) results. claimed replication rate of 55 percent. Hyman had been asked to prepare Hyman prepared a preliminary a critical appraisal of parapsychology. critique, which he presented at the com­ Rather than attempting to take on the bined meetings of the Society for Psy­ whole field, he looked for a systematic chical Research and the Parapsycho- research program that parapsychologists logical Association in Cambridge, considered especially promising. It England, in August 1982 (SI, Winter needed to consist of a series of studies 1982-83). As a result of comments on carried out by a variety of researchers. that paper by Honorton and others, he He chose the ganzfeld psi studies. prepared a new and more systematic Respected investigators had conducted analysis of the data. them. Hyman was intrigued by their Hyman focused on two questions: claims that significant psi scores had (1) Does the data base, taken as a been achieved in more than half of the whole, supply evidence for psi? (2) Does experiments and that the studies had the ganzfeld psi study yield evidence been conducted with a high level of for psi that is replicable? sophistication and rigor. The basic index for these questions Hyman sought Honorton's cooper­ is some measure of hitting or target- ation. Honorton felt it important to matching compared with a chance base­ have an outside critic like Hyman assess line. This, Hyman noted, creates special the ganzfeld literature. He supplied problems; assumptions about chance Hyman with a copy of every reported levels and probability distributions take Fall 1985 3 on a great burden. gests a tendency to report studies with He divided his critique into four a small sample only if they have sig­ phases: nificant results." —Rechecking the "vote count." Another, related bias Hyman calls —Assessing the actual opposed to a "retrospective bias." "This is the the assumed level of significance. tendency to decide to treat a pilot or —Assigning procedural flaws to the exploratory series of trials as a study if studies. •it turns out that the outcome happens —Analyzing correlations among to be significant or noteworthy." He flaws, positive effects, and significance. found two studies in the data base that The "vote count" check assessed were clearly retrospective and strong whether the studies claimed to be suc­ circumstantial evidence of four others. cessful really amount to 55 percent of Next Hyman considered whether the total. Hyman found a lot depends the chances of getting successful results on how studies containing multiple in ganzfeld ESP studies without invok­ conditions are divided up. He found, ing psi are really as low as psi pro­ for instance, that a study counted as ponents suggest. The studies varied one "successful" replication could be widely in variables and in the questions viewed "with equal justification" as being asked. Notes Hyman, with some adding one successful and 11 unsuccess­ understatement, "Many confusing ques­ ful replications to the total. tions arise about what probability levels Then he considered the "file- to assign to the various tests of signifi­ drawer" problem. How many ganzfeld cance." studies have been conducted but not Generally, a ganzfeld experiment reported? Surveys have identified other is taken to show evidence of psi if the studies, and their inclusion tends to statistics indicate there is no more than lower the success rate. a .05 probability that the results are The important question here was due to chance. Hyman's various analy­ whether there was evidence for biased ses found that the probability of obtain­ reporting—specifically, is there a possi­ ing at least one significant outcome per bility that only those experiments that experiment was instead .24—"over four begin with a string of successes end up times the assumed level of .05." being reported? "It is [easy] to imagine The discrepancy results from the that a large number of experimenters use of multiple indices—the availability . might have begun conducting some of several different ways of getting "hits" trials and then abandoned the study without their being included in the when the first few trials turned out to probability estimates. Hyman found be unpromising. On the other hand, a that more than half the studies he eval­ few of these exploratory ventures might uated "clearly used multiple indices have started with initially successful without taking this into account in trials, encouraging the experimenter computing their statistical significance." either to continue or to stop and write Multiple indices was but one of six up the result as a successful replication." categories of multiple testing he checked Is there any evidence for such a the studies against. Forty percent of the suggestion? Yes, Hyman says. He found studies, for instance, used multiple a tendency for the studies with the fewer baselines; 64 percent used multiple trials to have a higher proportion of groupings. significant outcomes. "The most obvious Using this kind of analysis, Hyman conclusion is that such a strange rela­ found one study that had increased the tionship is due to selective bias. It sug- probability of getting successful results 4 THE SKEPTICAL INQUIRER, Vol. 10 ''"&. V, "^^7: ..I'1"' J ,.*•> >" ,vl" " — \ "s"»'.,x lie* \ . i":'""""wv. *»••». i . "'•"'l.^,*r""'"'inJL**'«.l.l, / "'"fw *"""»'•. -n !/"•*'••• I 1 *" *nnHn I Ganzfeld debate: Hard-hitting critique, spirited rebuttal. "almost surely beyond .50." In other had inadequate security, and 29 percent words there was a better than 50-50 appeared to use erroneous statistical chance of getting a positive, "pro-psi" procedures. result just by chance. "Indeed, if we Hyman says he was very conserva­ consider the eight intervening practice tive in assigning these flaws. Those that conditions, the chances of coming up were not too common or depended on with a significant outcome are well over suspicions or hard-to-objectify criteria .80! And this is just one of the many he did not count. "In any case the studies in this data base that exhibit existence of so many elementary defects such complex options either explicitly in this data base is both disturbing and or implicitly." surprising. Only two studies were Hyman found that the actual rate entirely free of the six procedural flaws. of successful replication is less than 30 And if we include multiple-testing percent. "And the arguments in this sec­ errors, not a single study in this data tion strongly suggest that this rate of base was flawless. 'successful' replication is probably very "It is important to realize that the close to what should be expected by defects being discussed are not obscure chance given the various options for or subtle. Rather, I suspect that a multiple testing exhibited in this data typical parapsychologist would spon­ base." taneously list them as being unaccepta­ Hyman then turned to procedural ble in a psi experiment." flaws in the studies. He found that 36 Are these defects important? Yes, percent of them used improper random­ Hyman believes, in two ways.