Meta-Analysis in Parapsychology: I
Total Page:16
File Type:pdf, Size:1020Kb
Australian Journal of Parapsychology 2006, Volume 6, Number 1, pp. 35-53 Technical Paper No. 11 Meta-Analysis in Parapsychology: I. The Ganzfeld Domain1 LANCE STORM ABSTRACT: The present article is a review of the ganzfeld meta-analytic literature. It is found that significant results were obtained in all but one ganzfeld meta-analysis¾that of J. Milton and R. Wiseman (1999). However, with combinatorial re-construction of the available databases and the uncovering of 11 studies overlooked by Milton and Wiseman, L. Storm and S. Ertel (2001) reconfirmed that the ganzfeld was still the paradigm that delivered one of the highest effect sizes of all the experimental domains in parapsychology. More recent studies support this finding. Parapsychologist and pioneer of ganzfeld research, Charles Honorton (Honorton, 1985) said that the ganzfeld demonstrates a “significant psi effect” (p. 81), and the evidence in the present article supports that claim. INTRODUCTION TO META-ANALYSIS Meta-analysis, a term coined by G. V. Glass in the 1970s, is “the statistical analysis of the summary findings of many empirical studies” (Glass, McGaw, & Smith, 1981, p. 21). By combining the data from a number of different studies, meta-analysis produces an increased data-set that may be analysed to yield a more powerful result. Glass et al. (1981) describe the characteristics of meta-analysis in the following way: 1 The author thanks the Bial Foundation for support in the preparation of this article. 35 Australian Journal of Parapsychology 1. “Meta-analysis is quantitative”—it organizes and extracts “information from large masses of data that are nearly incomprehensible by other means.” (p. 22). 2. “Meta-analysis does not prejudge research findings in terms of research quality.” This policy runs counter to the actions of critics of meta-analysis who would like to exclude many studies on the methodological grounds that they are poorly designed or treatments are badly implemented, even though “evidence is never given to support the assumption that these deficiencies . influence their findings.” (p. 22) 3. “Meta-analysis seeks general conclusions,” and “aims to derive a useful generalization that does not do violence to a more useful contingent or interactive conclusion.” (pp. 22-23) (Naturally, these assumptions have been criticised, and these assumptions will be addressed in the next section.) The importance and usefulness of the ‘meta-analytic’ approach was demonstrated decades ago by prominent statisticians, psychologists, and medical researchers (Cochran, 1954; Edgington, 1972a, 1972b; Fisher, 1932; Mantel & Haenszel, 1959; Mosteller & Bush, 1954; Tippett, 1931). The flexibility and diversity of applications of meta-analysis have also been well reviewed and described.2 Bullock and Svyantek (1985) point out that meta-analysis is “objectively verifiable, using measured concepts, quantitative data, and statistical analysis” (p. 112). As opposed to the traditional literature review, where the reviewer is prone to his or her own “subjective interpretation of results across studies” (p. 112), the reviewer’s selection bias in a meta- analytic review is more explicit, since the inclusion and exclusion of particular domains (i.e., ‘experimental types’ in the context of the present study) is so readily apparent. The advantage for the reader is that the philosophical persuasion and/or general intention of the author can be determined with ease so that the overall merits of the work can be more easily assessed. 2 For examples, see Glass et al.’s (1981, pp. 25-26) meta-analyses of studies relating to the efficacy of psychotherapy in treatment of asthma and alcoholism. See also the meta- analyses by Cook et al. (1992) of juvenile delinquency studies, and studies on psychoeducational care with adult surgical patients, among others. 36 Australian Journal of Parapsychology As mentioned, meta-analysis produces results that are not attainable by other means (at least, not without difficulty), but even these results may not end the debate over a specific area of controversy. Meta- analysis can never be conclusive. It provides a critical examination of the current status of research in a given area. However, by finding flaws in current meta-analytic techniques, and by identifying the more successful experimental domains through up-to-date meta-analytic procedures, newer, more focussed directions for parapsychology can be established, which, in the case of the parapsychologist, may help “settle the question” of whether or not psi exists (Broughton, 1991, p. 284). PARAPSYCHOLOGY AND META-ANALYSIS As will be seen in the next section and Part II of this series (in the next issue of Australian Journal of Parapsychology), meta-analysis has been a godsend for parapsychologists. Specifically, the direct benefits of meta-analysis for parapsychology become clear once it is understood that meta-analysis is the means by which the “signal” of the psi effect can be ‘distilled’ from the “noise” of chance with greater sensitivity than in any single study (Broughton, 1991, p. 281-282).3 Closely related to meta-analysis is the concept of replication. Rosenthal (1986) lists three main reasons for the so-called failure of many single studies to elicit significant psi effects, after previous successes with the same experimental design: 1. Pseudo-failure to replicate due to a poor consideration of the appropriateness of the statistical test(s) used. (See the illuminating example of a comparison between effect size and significance levels given by Rosenthal, 1986, pp. 317-318.) Essentially, the investigator may be too dependent on the significance (or not) of the p value. 3 In fact, the presence (‘signal’) of a paranormal effect can never be specifically demonstrated as and when it occurs, even if 100% of trials are hits. For example, an experiment by Rhine (1937/1950, pp. 74-76) showed 25 hits in 25 trials, but in such an experiment there is no means of distinguishing the successful calls that suggest psi effects from those calls attributable to chance. The statistical inference of a psi effect is always made in the knowledge that the effect is produced within a stochastic framework. 37 Australian Journal of Parapsychology 2. Pseudo-successful replications, where p values are less than .05, but the effect sizes are significantly different, and therefore have not been replicated. 3. Successful replication of Type II error—not finding an effect when one was present due to small sample size, or very weak effects. Consideration of the replication issue has led to an expansion of the meta- analytic methodology and a refinement of its technique (see Storm, 2005). Replication cannot be guaranteed, but it does come in forms other than rejection of the Null hypothesis (for example, replication of effect sizes, z scores, etc.). Meta-analysis is fast becoming the only acceptable evidence that psi might well be an “anomalous effect in need of an explanation” (Utts, 1991, p. 363). However, there is now a growing suspicion of meta-analysis, not simply because it provides suggestive evidence of psi, but because in controversial non-parapsychological fields, such as psychotherapy, meta- analysis is also providing favourable evidence (Smith & Glass, 1977). Thus have some critics (Bandura, 1978; Eysenck, 1978; Oakes, 1986; Shapiro & Shapiro, 1977) focused on the construct validity of meta-analysis. Arguments are made that data from methodologically flawed experiments are thrown into the ‘soup’ with the data from better-designed experiments, therefore corrupting the result. Different procedures in ostensibly the same kind of experiment may also yield a tainted result. Of course, the general implication derived by critics (that meta- analysis is a flawed procedure) also applies to meta-analysis in parapsychology. But these criticisms have not been ignored, and improvements have been made to control for the confounding factors. Rosenthal (1984) advocates differential weighting as an effective way of dealing with “variation in the quality of research” (p. 127). Hence the ‘blocking’ procedure is used to code experiments according to their ‘quality’ and type (i.e., methodology, hypothesis, etc.). Sample size and the population from which the sample is drawn are also critical considerations. Credit is given to studies if sample size is specified in advance, as well as the nature of the analyses¾pre-planned or post hoc. Acceptable randomisation methods are also credited, and even the date of the experiment and the identity of the investigator are now important criteria in meta-analytic studies (Broughton, 1991, p. 283). Although these procedures may be seen as subjective, some degree of qualitative assessment can be made about studies, and these assessments are converted to numerical values to arrive at a more objective, albeit pseudo-precise, 38 Australian Journal of Parapsychology numerical result that is still seen as a gain over previous methods which did not consider study quality. One important criticism made against meta-analysis in parapsychological research (applicable to meta-analysis in general) is that significant results are inevitable, since the majority of studies used in the analysis have significant results already. On the other hand, studies with nonsignificant results are never or are rarely published and, therefore, cannot be included in the meta-analysis (Hyman, 1985). This problem is often referred to as ‘selective reporting’. There are three factors that belie this criticism. First, parapsychology