Psychological Assessment Copyright 2001 by the American Psychological Association, Inc. 2001, Vol. 13, No. 4, 452-471 1040-3590/01/S5.00 DOI: 10.1037//1040-3590.13.4.452

The Rorschach: Facts, Fictions, and Future

Donald J. Viglione Mark J. Hilsenroth Alliant International University Adelphi University

A large body of empirical evidence supports the reliability, validity, and utility of the Rorschach. This same evidence reveals that the recent criticisms of the Rorschach are largely without merit. This article systematically addresses several significant Rorschach components: interrater and temporal consistency reliability, normative data and diversity, methodological issues, specific applications in the evaluation of thought disorder and suicide, meta-analyses, incremental validity, clinician judgment, patterns of use, and clinical utility. Strengths and weaknesses of the test are addressed, and research recommendations are made. This information should give the reader both an appreciation for the substantial, but often overlooked, research basis for the Rorschach and an appreciation of the challenges that lie ahead.

A large body of empirical evidence supports the reliability, process (Meyer et al., in press) provide conclusive empirical validity, and clinical utility of the Rorschach, suggesting that the evidence of strong interrater reliability for the great majority recent criticisms of the Rorschach are largely without merit. We (95%) of Comprehensive System (CS) variables. An extensive address 10 areas in which the Rorschach has recently been chal- amount of data reveals that well-trained raters can score both high lenged: (a) interrater reliability, (b) temporal consistency reliabil- or low base-rate CS variables with good (K and intraclass corre- ity, (c) normative data and diversity, (d) specific applications in the lation [ICC] > .60) to excellent (>.75-.80) reliability (Garb, 1998; evaluation of thought disorder and suicide, (e) meta-analyses, (f) Shrout & Fliess, 1979).1 incremental validity, (g) clinician judgment, (h) implications of clinical and forensic usage, (i) clinical utility, and (j) methodolog- Conclusion: Empirical Data Support Rorschach Interrater ical issues relevant to validity. In each section, we summarize the Reliability empirical findings and their implications, explore the challenges, and recommend areas for future investigation to enhance the The great majority (95%) of the individual variables are coded clinical utility of the Rorschach in personality assessment, so that with good or excellent interrater reliability. Interrater reliability for we may increase our ability to understand our clients and to form quality, individual cognitive special scores, form versus color intervene effectively. This review will give the reader an appreci- dominance, shading subtypes, DOv and DOv/+, and some content ation for the substantial, but often overlooked and discounted, categories could be enhanced. Only a few very low base-rate research basis for the reliability, validity, and utility of the variables (e.g., MQ none) occasionally produce poor reliability Rorschach. coefficients, but they have not yet been analyzed with sufficiently large samples given their miniscule base rates. As in the past Reliability (Exner, 1974, 1995; Viglione, 1997), Rorschach researchers have recognized the limitations of the test. Improvement in training for Interrater Reliability the coding of these variables is the focus of a new reference book Individual research studies published over the past 20 years in currently being prepared for publication (Viglione, 2001). peer-reviewed journals (Acklin, McDowell, & Verschell, 2000; McDowell & Acklin, 1996; Meyer, 1997a, 1997b; Viglione, Empirical Facts Regarding Temporal Consistency 1999), in Exner's volumes (Exner, 1993), and in the publication Reliability

There is general agreement (Meyer, 1997a; Viglione, 1999) with Donald J. Viglione, California School of Psychology at Alliant Interna- Wood and Lilienfeld's (1999) statement that "Exner and his col- tional University; Mark J. Hilsenroth, Derner Institute of Advanced Psy- leagues have reported test-retest reliability coefficients and that chological Studies, Adelphi University. these coefficients are generally quite good" (p. 344). The great We extend our appreciation to William Perry and Leonard Handler for majority of the CS variables with such strong temporal consistency their thoughtful comments on an earlier version of this manuscript. Thanks reliability are central interpretive variables (Exner, 1991,1993). In to our students for their help in collecting materials for this review: Steven Ackerman, Matthew Baity, Nicole Dizon, Erin Eudell, Alice Laing, Kath- leen Montemagni, Joseph Obegi, Suzanne O'Brien, Maria Ortiz, and Todd 1 Our review of interrater reliability only addresses CS reliability. Many Pizitz. other studies have demonstrated good to excellent reliability data for Correspondence concerning this article should be addressed to Donald J. non-CS variables such as the Mutuality of Autonomy (e.g., Fowler, Hilsen- Viglione, California School of Psychology at Alliant International Univer- roth, & Handler, 1996), Aggressive Content (e.g., Gacono & Meloy, 1994), sity, 6160 Cornerstone Court, San Diego, California 92121. and the Rorschach Oral Dependency scales (e.g., Fowler et al., 1996). 452 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 453 other words, these variables are listed among the 68 nonredundant correlation = .66; Mdn = .71), which were almost identical to variables on the lower portion, "below the line" in the CS struc- those of the control group (mean correlation = .72; Mdn = .74). tural summary blank. In turn, the great majority of variables Nonetheless, a review of the responses revealed that experimental without test-retest coefficients (e.g., those identified by Garb, participants gave many more new responses in the second testing Wood, Nezworski, Grove, & Stejskal, 2001) are not included than did control participants. Thus, even though the content of the among these central, interpretive ratios, percentages, and deriva- responses changed, the formal scores for the central interpretive tions. Thus, the variables supposedly without reliability coeffi- variables largely remained the same. cients are not as central to interpretation. Of the 51 variables The six actuarial indexes in the CS (i.e., Schizophrenia [SCZI], identified by Garb et al. (2001) as having test-retest coeffi- depression [DEPI], suicide constellation [S-CON], coping deficit cients, 46 (90%) are central, interpretive ratios, percentages, and [GDI], hypervigilance [HVI], and obsessive style [OBS]) lack derivations. Consequently, the great majority of the most impor- definitive temporal consistency data, but available findings support tant variables have adequate data for temporal consistency reli- their stability. The actuarial indexes are derived from 58 vari- ability evaluation. ables, 47 of which possess or adequate test-retest data in the Garb et al. (2001) presented two tables addressing the issue of research reports. Six of the 11 scores without test-retest data (FQo, CS test-retest reliability. Their Table 1 presents the central inter- FQ-- FQu, Level 2 special scores, and FABCOM2) are subcom- pretive variables that have test-retest reliability coefficients. ponents of other variables (X—%, X+%, Ego Impairment Index Among the 30 variables that Garb et al. identified (in their Table [EII], WSUM6, SUM6, INC + FAB) with good to excellent test- 2) as not having test-retest coefficients, there are 3 variables retest data. (FC:CF + C, a:p, EB) that actually do have adequate data. Garb et al. (2001) cited several studies that seemed to demon- FC:CF + C, a:p, EB are each ratios of two other variables, and all strate less acceptable temporal consistency reliability for the CS, six of the component scores are listed by Garb et al. in their but they did not report their serious methodological flaws. These Table 1 as variables for which test-retest coefficients have been flawed studies include research with a nonstandard form of ad- reported. The text in a study by Exner, Armbruster, and Viglione ministration (Schwartz, Mebane, & Malony, 1990); a dissertation (1978) clearly reveals that evidence of good reliability for EB, involving only 17 older adults with an unspecified test-retest EA:ep (now called EA:es), and FC:CF + C cannot be attributed to interval (Erstad, 1996); and a sample of schizophrenic patients chance. With respect to other variables that Garb et al. said lacked with various illness courses, medications, hospitalizations, psycho- retest data, one, EBPer, is actually just a cutoff score of EB, two therapies, and test-retest intervals ranging from 1 to 18 years of the listed scores do not exist [F-% and H + (Hd):(A) + (Ad)], (averaging 6.4 years; Adair & Wagner, 1992). and one variable is listed (blends), even though Exner (1999) has reported on its retest stability. In addition, Garb et al. listed the Conclusion and Future Research An + Xy score twice, listed the W variable twice, first as W and then as W:M, and listed the variable M (in the W:M comparison), The empirical data support the conclusion that the great majority even though it is also listed in their Table 1 as one of the scores of CS central interpretive ratios, percentages, and derivations pos- that has test-retest data. The variables CONTAM, PSV, and CP are sess impressive temporal consistency. As demonstrated in Table 2, listed, but they have such incredibly low base rates that correla- the test-retest coefficients for CS Rorschach variables compare tional, parametric statistics are unstable without huge samples. quite favorably with those for other tests. For the limited number Among the 30 variables that Garb et al. (2001) alleged to lack of variables without published retest coefficients, most are of test-detest correlation coefficients, there are only 12 variables that secondary interpretive significance. Typically, they enter into cal- actually do not have test-retest data. culations to produce the central interpretive variables, and the Table 1 is a more complete listing of the available test-retest latter have demonstrated adequate or better temporal consistency. coefficients. These test-retest coefficients compare favorably to The CS has more than adequate temporal consistency reliability in the Wechsler scales, second version of the Minnesota Multiphasic all respects, especially in the context of comparing Rorschach data Personality Inventory (MMPI-2) and other tests. Table 2 presents to other established methods of assessment. data culled from many sources, comparing CS test-retest correla- Nonetheless, more test-detest reliability research should be un- tions with those from these other scales. Table 2 presents the recent dertaken. The Rorschach and the CS, like many other psycholog- meta-analyses that address this issue, an exhaustive selection of ical assessment measures (e.g., MMPI/MMPI-2 and Wechsler recent MMPI and MMPI-2 data, and data on two other represen- Adult Intelligence Scale—Revised [WAIS-R/WAIS-III]), con- tative tests, one a performance test and the second a commonly tinue to evolve; therefore, temporal consistency information used research instrument. should be updated and include variables for which no data are Memory effects cannot account for strong temporal consistency available. There were only 22 ratios, percentages, and derivations reliability findings. With respect to retest intervals of 6 months or in the CS structural summary and approximately a dozen other more, psychometric texts (e.g., Anastasi & Urbina, 1996) mini- variables central to interpretation (Exner, Weiner, & Schuyler, mize the possibility that memory effects act as confounds. It is 1976) at the time that the original temporal consistency data were certainly not an issue in 3-year retest intervals (Exner et al., 1978). processed (Exner, Armbruster, & Viglione, 1978). Future research In addition, Haller and Exner (1985) addressed the memory issue could utilize short test-retest intervals and ensure minimal change empirically. They instructed experimental respondents to give in psychiatric, environmental, and target constructs. The sample different answers on the second testing, but they gave no such should include both nonpatients and a variety of patients to avoid instructions to the control groups. The experimental group pro- restrictions of range on actuarial indices and pathology indicators. duced adequate temporal consistency reliability coefficients (mean To analyze data with moderately low base rates, a large sample Table 1 Test-Retest Correlation Coefficients for Comprehensive System Structural Summary Variables Among Adults

Group8 Variables" 12 3 4

N 50 100 35 25 57 Retest period a 1 year z 3 years 3 weeks 3 or 4 days 2 years Participants Adult nonpatients Adult nonpatients Adult depressed inpatients Adult nonpatients Adolescent nonpatients age 14 to 16 Complexity, style, controls R .86 .79 .84 .77 .80 Lambda .78 .82 .76 .82 .78 Eff* (M:WSumQ M .84 .87 .83 .75 .77 WSumC .82 .86 .83 .68 .69 EAa .83 .85 .84 .71 .71 EM .77 .72 .72 .28 .81 m .26 .39 .34 .84 .06 C .73 .67 .67 .77 .69 V .87 .81 .89 .82 .92 T .91 .87 .96 .86 .91 Y .31 .23 .41 .69 .13 eb (FM + m:Sum Sh) FM + m .69 SumSh .66 .71 .74 .71 es .64 .72 .59 .69 .73 D-Score .91 .83 .88 Adj esd .82 .85 Affect FCe .86 .86 .92 .71 CF + Cf .81 .79 .83 .74 .72 C + Cn .56 .51 .59 Afr. .82 .90 .85 .57 .76 S .18 Blend? .74 .71 Ideation perception 3r + (2)/R .89 .87 .90 .74 .73 Fr + rF .82 .89 FD .88 .90 .76 MOR .71 .83 .87 " Citations for the groups are as follows: Group 1 = Exner, 1993; Exner, 1999. Group 2 = Exner, Armbruster, & Viglione, 1978; Exner, 1993; Exner & Weiner, 1995. Group 3 = Exner, 1993, 1999; Exner & Weiner, 1995. Group 4 = Mailer & Exner, 1985. Group 5 = Exner, Thomas, & Mason, 1985. b For those variables listed as missing by Garb et al. (2001), and for which we could find no published retest coefficients, we asked Exner to provide the coefficients. Garb et al. (2001) did not list M—, DQ+, DQv, and Level 2 cognitive special scores as missing. Accordingly, we did not process coefficients for these variables and do not present mem in the table. CP and Mnone are listed as missing by Garb et al. (2001). They have extremely low base rates. Only 5 of the approximately 13,392 responses in the nonpatient reference group of 600 adults (Exner, 2001) were assigned a CP and only 4 were assigned a Mnone. Accordingly, a huge sample is needed to stabilize the retest coefficients for these two variables and values were not requested for them. Test-retest coefficients are still needed for Ma, Mp, Level 2 cognitive special scores, W. D, Dd, DQ+, DQv, Fd, Pure H, (H) + Hd + (Hd), An + X>, and the new comprehensive system variable (Exner, 2001). c EBPer is merely a cutoff score of EB. It is listed as missing by Garb et al. (2001). d Adj D is merely a transformation of EA—Adj es, both of which have coefficients. c Listed by Garb et al. (2001) as lacking test-retest coefficients. f At the time of these publications (Haller & Exner, 1985; Exner, Thomas, & Mason, 1985), there were only five cognitive special scores. SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 455

Table 2 ified for geographic distribution and partially stratified for socio- A Comparison of Test-Retest Correlations Among Frequently economic level. The first of these nonpatient participants was Used and Highly Regarded Tests collected before 1975, followed by various additions and revisions through 1991 (Exner, 1993). All respondents were volunteers, with Correlation the majority coming from places of employment. The gender Tests variables Retest interval M Mdn distribution is balanced, but the sample is rather young; 73% of the participants were under 36 years of age. Nonetheless, there are Multiple studies some differences between this adult group and the 15- and 16- Meta-analysis—Self-report, observer. a year-old sample (Exner et al., 2001). The adult group is relatively and projective 1 year .55 — 5 years .52 well-educated; the median grade level is over 13 years, and only b — Meta-analysis — 8 self-report tests 1 year .78 — 5% had less than 12 years of schooling. Eighteen percent of the 5 years .58 — respondents were non-Caucasian, which is consistent with other Meta-analysis — MMPf 5 years .57 — normative samples, although this percentage could be increased. MMPI— 7 studies0 1 year .57 .58 Single studies For example, the minority percentage for the WAIS-III is 21% MMPI" 1 day to 2 years .68 .68 (The Psychological Corporation, 1997) and for the MMPI-2 it is MMPI-2 1 week* .80 .83 19% (Hathaway et al., 1989). 4 monthsf .73 .74 8 Rorschach researchers (Brinderson, 1995; Kates, 1994; Shaffer, 5 years .72 .73 Erdberg, & Haroian, 1999; Viglione, Gaudiana, & Gowri, 1997; California Verbal Learning Test 1.3 years" .47 .43 1 year1 .53 .54 Weiner, 200 Ib) have questioned the continued suitability of these Attributional Style Questionnaire1 5 weeks' .63 .64 CS norms. For reasons similar to the emergence of concerns about Rorschach 3 years* .79 .80 1 the aging of MMPI norms (e.g., Colligan, Osborne, & Offord, 3 weeks .80 .85 1984), one might contend that social and cultural shifts, cohort 1 year1 .76 .82 3 weeks' .82 .85 differences, and the gradual evolution of the test justify collection 1 year1 .82 .83 of a new sample. 1 year"1 .75 .82 3 to 4 days .71" .74 3 to 4 days .66° .71 Research Data as Approximations of Nonpatient Sampling

Note. Dashes indicate measure was not applicable. MMPI = Minnesota To approximate a contemporary nonpatient reference sample, Multiphasic Personality Inventory. 1 Roberts & DelVecchio (2000). b Schuerger, Zarrella, & Holtz, we reviewed the literature for available adult control and nonpa- (1989). °Mauger (1972) as cited in Dahlstrom, Welsh, & Dahlstrom tient groups. Shaffer et al. (1999) published the most comprehen- (1975); Stone (1965) as cited in Dahlstrom et al. (1975); Sines, Silver, & sive and notable effort to produce normative approximation data. Lucero (1961) as cited in Dahlstrom et al. (1975); Ryan, Dunn, & Paolo To construct Table 3, we selected central interpretive variables that (1995); Milott, Lira, & Miller (1977). "Hunsley, Hanson, & Parker (1988). e Butcher, Dahlstrom, Graham, et al. (1989). 'Putnam, Kurtz, appeared to deviate from CS normative expectations in the Shaffer & Houts (1996). g Spiro, Butcher, Levenson, Aldwin, & Bosse (1993) as et al. study. We then supplemented the Shaffer et al. findings with cited in Putnam et al. (19%). "Paolo, Troster, & Ryan (1997). 'Delis data from five other published samples known to us and one j k et al. (1996) as cited in Paolo et al. Peterson et al. (1982). Exner, previously presented sample (Viglione et al., 1997). These data Armbruster, & Viglione (1978). 'Exner (1999). m Exner (1993). " Mailer & Exner (1985), depressed inpatients. ° Haller & Exner (1985), allowed us to address some issues regarding deviation from CS depressed inpatients instructed to give different answer to evaluate "mem- reference sample expectations. ory effects." R. The data in Table 3 suggest that R is quite variable. Whether or not such variability presents a problem for the Ror- schach has been debated for many years. The distributional pa- (N = 80 or more) should be collected. Interpretive temporal rameters (mean is greater than median, a large standard deviation, consistency based on established interpretive cutoffs should be and a minimum value of 14 with no predetermined maximum included. Systematically assigning multiple raters to recede value) suggest that a great deal of variability is due to a small records would allow one to discriminate the effects of temporal proportion of records with excessively high R. From a clinical inconsistency from interrater disagreement in accounting for utility perspective, long records consume administration and cod- changes over time. ing time and increase cost without necessarily increasing interpre- tive yield. Lambda. The data for Lambda are incomplete, but suggest that Normative Data and Diversity Issues median values are close to CS reference values. The Shaffer et al. The detailed presentation of large reference samples from non- (1999) study provides the only sample that seems to deviate from patients, various patient groups, and children from ages 5 to 16 has the expected distribution. Nonetheless, on the basis of standard contributed to a resurgence of the Rorschach over the past 20 years deviations, within each sample there are records that are much (American Psychological Association, [APA] 1998; Exner, 1974, more form dominant than would be expected. The occasional high 1993; Exner et al., 2001; Wood & Lilienfeld, 1999). To limit our Lambda record does not pose much of a problem for routine discussion of normative data, we focus here on the adult, nonpa- interpretation. On the other hand, low /f-high Lambda records tient CS reference sample, consisting of 600 adults, which is a may lack negative predictive power and may reduce interpretive relatively large normative sample (Exner et al., 2001). It is strat- yields and cost-benefit ratios. 456 VIGLIONE AND HILSENROTH

Table 3 Nonpatient Adults' Data for Some Rorschach Variables

Lambda" x-w WSVM6

N M Mdn SD M Mdn SD M SD M SD M Mdn SD

20" 245 245 93 075 061 051 055 015 020 0 15 90 38 123 20b 17.6 3.7 0.68 0.58 0.52 0.14 0.17 9.7 11.4 35C 26.0 7.3 0.83 0.57 0.52 0.13 0.21 2.8 3.8 25" 1.0 0.80 0.57 0.18 123e 20.8 18 75 1.2 075 1 72 0.51 0 15 021 0.11 66 40 80 35f 20.6 21 7.7 0.16 0.13 2.1 0.0 2.9 83s 22.8 21 7.3 0.58 0.45 0.46 0.55 0.14 0.17 0.17 7.9 5.0 9.3 Summary means across studies Weighted 21.9 21.5 7.3 0.92 0.63 1.01 0.53 0.15 0.19 0.14 6.4 4.4 7.8 Not weighted 22.1 22.2 7.1 0.84 0.60 0.77 0.54 0.15 0.19 0.14 6.4 4.3 7.9 Exner et al. (2001) reference sample 21.9 22 4.4 0.60 0.53 0.31 0.77 0.09 0.07 0.05 4.5 5.0 1.8

"Netter & Viglione (1994), volunteers—no history of psychological hospitalizations or medication, deny substance abuse. bFrueh & Kinder (1994), White, male college students. cGreenwald (1991), college students. dSloan, Arsenault, Hilsenroth, Handler, & Harvill (1996), White, male, military reservists. "Shaf- fer, Erdberg, & Haroian (1999). f Bums & Viglione (19%), female volunteers with superior interpersonal relationships. g Viglione, Gaudiana, & Gowri (1997), nonpatient volunteers, without history of psychiatric treatment, deny substance abuse. h Because of significant problems with skew for this ratio as the denominator, Pure F, approaches zero, means and standard deviations may be misleading. The large standard deviations with means greater than medians suggest that skew is present and that the means are artificially elevated by outliers or skewed high values. ' Distributions are quite normal for X+% and X—%, so that only means and standard deviations are presented.

Form quality. X+% and X—% results suggest that FQu and WSUM6. For cognitive special scores in the form of WSUM6, FQ— are more numerous than anticipated, but they are consistent the large standard deviations suggest that a small proportion of with some previous observations (Viglione, 1989). Interpretive individuals provide many of the elevated values. Given that these ranges for X+%, as a measure of overall conventionality, and respondents are not patients (though it is possible that some had X— %, as a measure of perceptual distortion, need to be modified. been prior to testing), one cannot be sure whether these results are The data in Table 3 may underestimate X+% and overestimate associated with poor sampling, idiosyncratic responses, idiosyn- X—% because one cannot be sure that pathological respondents cratic coding, or overly long, elaborated, and complex records. On were excluded from the samples. Nevertheless, the results do the other hand, central tendencies are consistent with the CS data. suggest that a more appropriate cutoff for the "unconventional" The standard deviations for WSUM6, as well as R and Lambda, interpretation of X+ % may be approximately X+ % < .50 and that suggest that these samples contain more exceptional or extreme the cutoff for the "distortion" interpretation of X—% may be closer cases than does the CS reference group. to X-% > .25. The data suggest that the cutoffs for children should also be more liberal (Hamel, Shaffer, & Erdberg, 2000; Cultural and International Implications Kates, 1994). With improvements in coding FQ, as suggested in the interrater reliability section in this article, these interpretive These normative approximation data also have implications for bands might tighten. It is also important to recognize that FQ has cultural and international issues. Research data on the Rorschach some of the best validity data of any coding CS variable (Dawes, and diversity issues support the use of this measure with diverse 1999; Viglione, 1999). Cases with elevations in the SCZI produced populations (Viglione, 1999). Large-scale cultural diversity and solely by form quality should be closely examined to rule out international projects have been under way for some time, and possible false-positive findings.2 expertise among the project managers around the world is improv- Exner (2000) has made some changes in the form quality ing (Erdberg & Schaffer, 1999). These data have the potential of percentages to address these issues. However, if it is true that 20% teaching us a great deal about cultural and geographic issues. In of the responses in nonpatient records are distorted (i.e., FQ—), terms of thinking about cultural differences and expectations from then we have a problem with the operational definition for the construct of perceptual distortion. One could justify 5% or 10% of 2 The SCZI is being replaced by a dimensional index of thought disor- the responses being distorted, but it does not make sense for us to der, the Perceptual Thought Index (Exner, 2001). Such an index more call 20% of Rorschach responses among nonpatients distorted. We accurately reflects the available data (Hilsenroth, Fowler, & Padawer, may need to narrow the operational definition of FQ—, so that this 1998; Kleiger, 1999) and a modem understanding of thought disorder as code truly represents distorted perception as outlined by Dawes existing on a continuum from optimal to most disordered within and across (1999). diagnoses (Viglione, 1999). SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 457 a normative perspective, the data emerging from these studies are 1974). The findings with R, and the possibility of related variabil- more consistent with the normative approximation data given in ity among the other indices, suggest that constraining R through Table 3 than they are with the CS reference sample. statistical comparisons or procedurally through minor administra- Moreover, emerging childhood reference group data from tive modifications deserves some consideration. United States samples (Hamel et al., 2000; Kates, 1994) and international samples (Erdberg & Schaffer, 1999) are consistent with a normative approximation sample of African American Meta-Analyses children that was collected about 20 years ago (Krall et al., 1983). Do meta-analytic results support the validity of the Rorschach in Garb et al. (2001) cited the Krall et al. study in their challenge of general terms? There are at least six recent and original meta- the diversity applications of the Rorschach. However, this is quite analyses that address Rorschach validity (Atkinson, Quarrington, puzzling because a close examination of the data indicates that the Alp, & Cyr, 1986; Bornstein, 1996,1999; Hiller, Rosenthal, Born- only problematic difference between Krall et al.'s findings and the stein, & Berry, 1999; Meyer & Handler, 1997; Parker, Hanson, & norms available at that time (Exner & Weiner, 1982) is for a form Hunsley, 1988). All concluded with empirically based statements, quality score (i.e., a lower F+%), which is no longer being used in supporting the validity of the Rorschach. Weiner (200la) summa- the CS (Exner, 2000). Krall et al. reported a range for F+% of .48 rized these empirical data and offered them as convincing support to .69, which is consistent with the normative approximation of the overall validity of the Rorschach. Opposing these five results presented in Table 3 and the emerging international data. meta-analyses is one re-analysis of the Parker et al. data (1988) by Also, differences in popular responses, for example, are as ex- Garb, Florio, and Grove (1998). The research in this meta-analyses pected given cultural variations (Mattlar, Carlsson, & Forsander, was published from 1954 through 1982 and contains only 15 1993). Thus, supposed problematic cultural differences evaporate Rorschach studies (compared with 75 utilizing the MMPI), only 5 when one considers the emerging international data. This conclu- of which utilized the CS. As suggested in the original authors' sion is supported by the very limited differences between nonpa- (Parker, Hunsley, & Hanson, 1999) reply to the re-analysis of their tient African American and Caucasian American adults matched data, the relevance of the Garb et al. (1998) re-analysis is highly on important demographic variables (Presley, Smith, Hilsenroth, & questionable and adds little to our contemporary understanding of Exner, in press). Finally, Meyer (in press) extended these findings these two clinical instruments. The Rorschach research and tech- by conducting a series of analyses to explore potential ethnic bias niques have advanced quite a bit over the last 20 years. Likewise, in Rorschach CS variables with a patient sample. After matching the MMPI-2 was not available to be evaluated in either of these on several salient demographic variables, ethnicity revealed no two meta-analytic studies. significant findings, and principal-components analyses revealed Analysis of moderator variables in the Hiller et al. (1999) no evidence of ethnic bias in the Rorschach's internal structure. meta-analysis in this Special Series revealed that the MMPI had larger validity coefficients than the Rorschach for self-report cri- A New Normative Sample? teria and psychiatric diagnosis "whereas the Rorschach had higher validity coefficients in studies using objective outcomes as crite- The Shaffer et al. (1999) and other normative approximation rion variables" (Hiller et al., 1999, p. 291). This is consistent with data are relatively consistent with CS reference group data (Exner Viglione's (1999) finding that the Rorschach variables were valid et al., 1995). Adjusting one's expectations for form quality, as and useful when one emphasizes ecologically valid behavioral outlined above, should eliminate the possibility of overpathologiz- criteria, rather than self-report criteria. Thus, as the criterion im- ing adults. proves, so do the Rorschach validity coefficients. In contrast, the An alternative point of view about these reference data with MMPI does best with criteria that share self-report method vari- nonpatients is also emerging (Meyer, 2001; Weiner, 2001b). This ance. Hiller et al. speculated that impression management plays point of view suggests that there should be a careful consideration less of a part in the Rorschach than in self-report testing. "In these of the guidelines used for collecting this Rorschach reference data. instances, it may be that the Rorschach is more objective than Figure! in this understanding of differences is the recognition that so-called objective instruments" (p. 291). Although Hiller, Exner's (1993) reference sample consists of socially-vocationally Rosenthal, and their colleagues' methods, data, and conclusions functioning nonpatients. In addition, it is equally important to note have been critiqued, the support for the Rorschach from their that CS data collected from both United States and worldwide work, as well as from all the other original meta-analyses, are quite samples are somewhat more healthy than Exner's reference sam- clear and accurate. As a whole, these meta-analyses reveal that the ples of people who are starting outpatient psychotherapy or dif- Rorschach produces valid data to measure many objective, behav- ferent patient groups. The interested reader is directed to Weiner ioral criteria. (2001b), who proposed guidelines for research when obtaining reference samples from patients and nonpatients. Meyer (2001) also provided an extensive and in depth review of possible factors Incremental Validity affecting normative shifts in Rorschach data. Both of these authors presented rational and empirical data to support their position that Research data from a variety of sources reveals considerable CS norms do not overpathologize potential examinees. emerging empirical support for the incremental validity of the Nonetheless, collecting new reference samples, with increased Rorschach. This research is not exhaustive, but it is clear that attention to cultural and other diversity issues, should improve Rorschach variables do possess incremental validity over other reliability and validity, and enhance utility. Like the MMPI, the CS well-known tests and collateral information. As such, the evidence and Rorschach have evolved since they were introduced (Exner, indicates that the Rorschach adds meaningful data to the assess- 458 VIGLIONE AND HILSENROTH ment process. Accordingly, negative characterizations, such as Table 4). These data were not sufficiently recognized by those who "The frequent claim that the Rorschach adds meaningful assess- adopted a critical view of the Rorschach in this Special Series. On ment information to other data has not been supported in any the other hand, consistent with Weiner, Garb (1999) did confirm published study" (Hunsley & Bailey, 1999, p. 271), are false incremental validity support to the EII, the Rorschach Oral De- because they ignore empirical findings. pendency scale, and the Rorschach Prognostic Rating scale (PRS). Meyer (2000) has addressed incremental validity issues in two Empirical Data That Address Incremental Validity meta-analyses. Within six studies, using both the Rorschach PRS and the MMPI Ego Strength scale (Es), the Rorschach PRS dem- Research presented in Weiner's (2001a) and Viglione's (1999) onstrated considerable incremental validity for predicting treat- reviews of the literature contain empirical data consistent with the ment outcome. The uncorrected and corrected effect sizes were conclusion that Rorschach variables possess incremental validity r = .32 and r = .48 (N = 229), respectively. The uncorrected and over other tests, demographic data, and other information (see corrected effect sizes, after excluding the outlier results from one

Table 4 Studies Included in Viglione (1999) With Findings Consistent With Rorschach Incremental Validity

Author and year Findings

Archer & Gordon (1988) Optimal overall correct classification (OCC hit rate) of individual inpatients with schizophrenia for Rorschach Schizophrenia (SCZI = 5) = .80. Optimal OCC for Minnesota Multiphasic Personality Inventory (MMPI) Sc scale (Sc s 75) = .76. Utilizing traditional cutoff scores, OCC rates were as follows: SCZI > 4, OCC = .69; Sc > 65, OCC = .48; and Sc > 70, OCC = .60. Archer & Krishnamurthy (1997) In classification of depression in adolescents with a stepwise discriminant function analysis, Rorschach variables Vista and Afr added R2 = .05 beyond combined R2 = .14 for MMPI-A scales A-DEP and Ma. These four variables had highest positive predictive power over any single variable or combination of variables. Bornstein et al. (1997) Rorschach Dependency scores significantly correlated with both number of significant interpersonal events (r = .84) and impact ratings of these events (r = —.38, p < .01). Self-report measure of dependency was not significantly related to either (r = —.11 and r = .16, respectively). Cooper et al. (1991) The Rorschach Defense scales provided unique prediction of outcome GAP (Global Assessment of Functioning, Health-Sickness) ratings in regression equations beyond initial GAP and borderline personality self-report scale. Hilsenroth et al. (1995) Rorschach variables were able to significantly differentiate (p = .008) those patients prematurely terminating from psychotherapy vs. those continuing in treatment, whereas the MMPI-2 was unable to do so (p = .56). Specifically Rorschach scores from the interpersonal-relational cluster had a mean effect size (ES) of .57, while the MMPI-2 content scale Negative Treatment Indicators had an ES of -.14. Holzman et al. (1974) The classification of a recent schizophrenia diagnosis (hospitalized less than 6 months) and deviant eye tracking was greater (65% accuracy) utilizing Rorschach data alone than a clinical team diagnosis (58%). O'Connell et al. (1989) Rorschach data (Thought Disorder Index [TDI]) predicted the development of psychotic and psychotic-like symptoms over a 2-3 year period in a sample of Axis II and affective disorder patients over and above information from clinical interview on lifetime prevalence of psychotic and psychotic-like symptoms (combined R2 = .21, TDI-beta = .32, p < .03), schizotypal symptoms (combined R2 = .42, TDI-beta = .32, p < .009), or schizotypal and borderline symptoms (combined R2 = .51, TDI-beta = .31, p < .006). Initial TDI scores also demonstrated clinical utility in prediction of psychotic and psychotic-like symptoms at follow-up. Perry & Braff (1994) Human Experience variable component of the Ego Impairment Index (EH) significantly related to neuropsychological markers of schizophrenia (r = -.42, p < .01; r = -.37, p < .025; r = -.35, p < .025), whereas thought disorder scales based on clinical interview (Schedule for Positive Symptoms and Schedule for Negative Symptoms) were not (p > .05). Perry & Viglione (1991) Rorschach EH predicted outcome Beck Depression Inventory (BDI; p < .0002) and Carroll Rating Scale for Depression (p < .01) scores in depressed patients treated with tricyclic antidepressants beyond variance accounted for by gender and baseline scores on BDI and EH. Other demographic variables were also considered but did not affect outcome. Russ (1980) Rorschach measures of adaptive regression (AR) and defensive effectiveness (DE) were significantly related to academic achievement, independent of IQ (AR: r = .45, p < .01; DE: r = .40, p < .01, respectively). Russ (1981) Rorschach measure of AR was significantly related to reading and overall academic achievement independent of IQ (r = .51, p < .001, and r = .47, p < .001). Index AR scores were significantly predictive of reading achievement 1 year later (r = .29, p < .05). Shapiro et al. (1990) Rorschach Depression Index significantly differentiated sexually abused African American girls from controls (p < .005). Children's Depression Inventory scores for sample were not significantly different from controls (p > .05), consistent with incremental validity of the Rorschach relative to self-reported depression. The groups did differ on the Intemalization scale of the Child Behavior Checklist (p < .0001). Skelton et al. (1995) The dependent variable was a ratio of Rorschach TDI over a TDI derived from the Wechsler. This ratio was 2.46 times higher in a group of 25 identity-disordered adolescents than it was among 35 conduct- disordered and oppositional-defiant adolescents (p < .01). SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 459 study (Fiske, Cartwright, & Kirtner, 1964), were r = .40 and r = in a regression analysis. Also, data from the study do not support .59 (N = 187), respectively. Effect sizes for the Es were essentially including R as a predictor The number of responses is not signif- nil (range, -.03 to 03). Clearly, the Rorschach PRS demonstrates icantly correlated with the criteria (Meyer's DxMax, r = .13, ns incremental validity over the MMPI Es scale. Nine studies (N = and in the wrong direction; and Perry's Social Competence Index, 358) included both the Rorschach PRS and an intelligence test. r = .16, ns). Across these studies, the Rorschach PRS demonstrated consider- Similarly, Dawes's decision to create a new variable, XQUAL able incremental validity over and above intelligence in the pre- and to require the EH to predict beyond XQUAL in a regression diction of future outcome. After correcting for methodological equation cannot be justified. The EH contains form quality (FQ—) artifacts, the unique contribution for the PRS was r = .48; and and R among its subcomponents. XQUAL and R are, thus, struc- without such corrections, r = .36. Blatt's (Blatt et al., 1994) turally redundant with EII and produce multicollinearity problems findings also demonstrate incremental validity of Rorschach scales within the regression, reducing the power of the analyses. Among over IQ in the prediction of treatment outcome, as do a number of the arguments for the unit weighting system is that a clinician's studies summarized in Viglione's (1999) synthesis of the literature intuition does not reliably "capture optimal weighting systems" (p. (see Table 4). 299). Its effect in the analysis is to reduce power once again. Blais, Hilsenroth, Castlebury, Fowler, and Baity (2001) ex- Garb et al. (2001) argued that Dawes's (1999) and Meyer and plored the incremental validity of the Rorschach in discriminating Resnick's (1996) use of the severity of clinical diagnosis as a Diagnostic and Statistical Manual of Mental Disorders (4th ed.; criterion is only a gross approximation of impairment or psycho- DSM-IV; American Psychiatric Association, 1994) personality logical disturbance. This is true, as it is for the Social Competence disorders (PD). In a re-analysis of existing data, select Rorschach Index criterion in the Perry, Moore, and Braff (1995) data set. variables and the MMPI-2, PD scales were used to predict chart These measures do not share a great deal of variance with impair- review ratings of DSM-IV antisocial, borderline, histrionic, and ment, the intended target construct of the EII. This limited overlap narcissistic personality disorders in a sample of treatment-seeking between measure and target construct stacks the cards against the outpatients. Hierarchical regression analyses showed that both Rorschach by limiting maximal effect sizes and the power of the MMPI-2 and Rorschach data added incrementally to the prediction analyses. Small effect sizes are to be expected, even in the best of DSM-IV borderline and narcissistic PD criteria. The findings circumstances, with such a global criterion. Under such circum- were less clear for the incremental value of Rorschach and stances, the maximum multiple Rs of .49 and .35, reported by MMPI-2 data in predicting the total number of DSM-IV histrionic Dawes, are quite impressive. PD and antisocial PD criteria. Rorschach data were the best pre- Dawes also introduced DxSum, the sum of the impairment dictors of DSM-IV histrionic criteria whereas MMPI-2 data were scores for up to three diagnoses, as a criterion measure.4 It was best at predicting the DSM-IV antisocial criteria. used without precedent, supporting data, or a straightforward ra- tionale. The DxSum makes the sum of mild diagnoses much more Dawes's (1999) Results Actually Support the Criterion impaired than the most virulent but pure schizophrenia or bipolar and Incremental Validity of the Ell and Rorschach disorder with psychotic features and adds systematic error to DxMax. We obtained the Meyer and Resnick (1996) data set (N = Earlier in this Rorschach Special Series, Dawes (1999) "pre- 187) and found that 36 individuals in the data set had a diagnosis sent[ed] two methods for assessing the incremental validity of a with moderate severity, namely a DxMax score of 3 or less, but a particular Rorschach variable" (p. 297), namely the EH, with the DxSum of 6 or more. For these 36 patients, even though they had expressed purpose of demonstrating how incremental validity of been assigned several diagnoses of mild to moderate severity, their the Rorschach might be investigated. He investigated two data sets DxSum score made them appear more impaired than individuals from: (a) Meyer and Resnick (1996), which includes the Ror- with a single diagnosis of schizophrenia (N = 16). In addition, the schach, MMPI, and impairment as approximated by severity de- data do not support Dawes's argument: All correlations in Dawes's fined by diagnosis (DxMax); and (b) Perry, Moore, and Braff Table 1 between meaningful predictors (the MMPI mean eleva- (1995), which contains the Rorschach, measures of social compe- tion, the MMPI Goldberg index, XQUAL and EII) are higher for tence and thought disorder, and another psychiatric rating scale. DxMax than for Dawes's DxSum. Clearly, the DxSum variable His conclusions are quite critical of the Rorschach but do not fit adds unacceptable error to the criterion and only functions to bias the data he presents. The analytic strategies he used reduced the the results against the Rorschach. probability of finding support for the Rorschach and EII and biased the analyses against the Rorschach. Therefore, we take issue with his methods and conclusions in this article. 3 One may presume that some researchers and clinicians might disagree Reducing the power of the analyses. To re-evaluate these data with the statement that form quality distortions (FQ-) "involve neither sets, Dawes (a) entered the number of Rorschach responses (R) and empirical research to establish their validity nor training in the Rorschach his new form quality measure, XQUAL, before the EII in the to score" (Dawes, 1999, p. 297). However, it appears that Dawes accepts that certain FQ— responses can be seen as clear, in vivo problematic hierarchical regression; and (b) used a unit weighting scheme in behaviors, consistent with understanding the Rorschach as behavioral addition to multiple regression. The argument for using the number sample of cognitive problem-solving behaviors (Viglione, 1999). of responses and his new measure, XQUAL, is that these Ror- 4 Dawes (1999, p. 298) stated that "the authors considered the average schach variables are easier to score and involve less projection by rating for up to three diagnoses and the sum of the ratings for up to three 3 the clinician. There is no established body of research that sup- diagnoses." Meyer (personal communication, March 6, 2000) calculated ports the number of responses to predict impairment; therefore, the average and sum of diagnoses, but he did not recall considering them there is no a priori reason to justify it as a predictor of impairment to be valid criterion variables and did not include them in his analyses. 460 VIGLIONE AND HILSENROTH

With respect to the Perry et al. data set, the only relevant statistic Incremental Validity and Clinical Utility is the univariate correlation relating the EII to the Social Compe- The clinical utility of a test emerges within individual cases that tence Index (r= .31, p < .05). Also, on the basis of the additional pose idiosyncratic questions and require idiographic decisions measures included in the Perry data set, the EII was correlated with (Kleinmuntz, 1990; Meehl, 1957; Meyer et al., 1998). Another the Schedule for Positive Symptoms global score (r = .367, p = complicating factor is that the "yield" from other sources of data .007) and with a thought disorder subscale of the Brief Psychiatric and tests varies from case to case. For example, a Rorschach may Rating scale (r = .334, p = .014). However, as expected, it was be too sparse to adequately address interpersonal perception or not correlated with the Schedule for Negative Symptoms global thought disorder. In one case, the MMPI may be too exaggerated score (r = -.092, ns). to address depression. For these and other reasons, the usefulness The Meyer data set allows us to address incremental validity. of a particular test varies greatly from case to case and question to For the reasons given above, the only defensible hierarchical question. "Field" incremental validity, therefore, only manifests in analytic strategy for this purpose is to add the EII and the other actual clinical practice in the form of clinical usefulness and the Rorschach variables after the MMPI variables. These analyses are impact of different tests in decision making within individual presented in Table 5. The correlation between EII and DxMax is cases. Weiner (1999) and Strieker and Gold (1999) provided .35. Given the limitations of DxMax as a measure of impairment, compelling rationales and case examples that clearly demonstrate the incremental Rs for the EII and the Rorschach are quite impres- the clinical "field" incremental validity of the Rorschach in indi- sive and may approach the maximum validity of DxMax as a vidual cases. measure of impairment. Conclusion regarding Dawes (1999). Thus, Dawes's (1999) Future Research analyses support the incremental and criterion validity of the Rorschach. The EII provided significant unique predictive power As with all personality measures, more research that demon- over MMPI variables for psychological impairment or disturbance strates incremental validity is needed. More sophisticated research as grossly estimated by the maximum severity of diagnosis. Ana- studies might address complexes of Rorschach variables with real lytic decisions by Dawes reduce the sensitivity of the analyses and, life behavioral and cognitive criteria. It would also be helpful to as a result, minimize the potential to support the Rorschach. look at response styles, response sets, and contextual moderator Re-analyses of these data sets provide clear empirical support for variables that might influence incremental validity. In other words, criterion validity and incremental validity of the EII and the Rorschach variables may generalize best to contexts that resemble Rorschach. the Rorschach problem, such as complex situations in which there is little external guidance so that the individual has to make his or her own way. Also, the adequacy of criterion variables and statis- Challenges to Incremental Validity of the Rorschach tical power present technical challenges to such research.

Recent claims (Garb, 1999; Garb et al., 2001; Hunsley & Bailey, Two Applications of Special Interest to Practitioners 1999) that the Rorschach has poor incremental validity are based largely on old studies that confound clinical judgment with incre- Thought Disorder and Implications for the Course of mental validity (Garb, 1994; see Garb, 1998, for a reproduction of Schizophrenic Spectrum Disorder these studies), a point we return to later. These studies also do not A variety of Rorschach summary scores have traditionally been have the benefit of the CS scoring system and established inter- used to assess and conduct research on thought disorder. Early pretive routines, in addition to lessons learned from clinical judg- research in this area used the Delta Index of Thought Disorder ment studies (Exner, 1993; Garb, 1994, 1998; Meyer et al., 1998). from which the Thought Disorder Index (Johnston & Holzman, Accordingly, these old experiments now lack ecological validity. 1979) was developed. Both indexes share many features with CS cognitive special scores and their weighted sum, the WSUM6. There are abundant data that support the irrefutable conclusion that Table 5 these scales, as well as similar summary scores from the CS and Incremental Validity of Ego Impairment Index (EII) and other systems, are related to thinking disturbance, psychoses, and Rorschach From Dawes (1999) With Meyer and Resnick's schizophrenic spectrum disorders (e.g., Acklin, 1999; Cadenhead, (1996) Data Set Predicting Ratings of Severity of Impairment of Perry, & Braff, 1996; Exner, 1991, 1993; Hilsenroth, Fowler, & Dx (DxMax) Padawer, 1998; Kleiger, 1999; Perry & Braff, 1994; Perry, Geyer, & Braff, 1999; Perry, Moore, & Braff, 1995; Viglione, 1999; Incremental r Weiner, 1966: Perry, Viglione, & Braff, 1992). In addition, re- 2 Predictor R Change in R over MMPI only search summarized below also associates Rorschach measures MMPI .366 .134 with schizophrenic spectrum disorders, genetic and environmental MMPI + EII .424 .058 .241 risk factors, psychophysiological processes in schizophrenia, and MMPI + Rorschach .492 .108 .329 subsequent outcome. Empirical findings. The Thought Disorder Index is associated Note. The Minnesota Multiphasic Personality Inventory (MMPI) and Ror- schach variables for analysis were those selected by Dawes: For the MMPI, with multiple criteria related to biological aspects of schizophrenia mean elevation clinical scales and Goldberg Index; and for the Rorschach, and psychosis proneness. Two studies associated the Thought EII, number of responses, and XQUAL. Disorder Index with the Chapman Psychosis Proneness Scales SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 461

(Coleman, Levy, Lenzenweger, & Holzman, 1996; Edell & Chap- thought disorder, with and outside of the CS, clearly reveals that man, 1979), a measure of magical thinking, perceptual aberration, the Rorschach can produce data related to the biological aspects of and social anhedonia (Chapman, Chapman, & Kwapil, 1995; thought disorder and psychosis proneness. Although the Thought Chapman, Chapman, & Raulin, 1976,1978; Eckblad, Chapman, & Disorder Index administration typically differs from the CS ad- Chapman, 1982). In another study, similar Thought Disorder Index ministration and conclusive data need to be collected in regard to elevations were produced by unaffected relatives of schizophrenics the impact of these administrative differences, similar findings (Shenton, Solovay, et al., 1989). The Thought Disorder Index also across the many Rorschach summary scores of thought disorder predicted the development of psychotic and psychotic-like symp- suggest that these positive findings should generalize to the EII toms over a 2 to 3 year period in a sample of Axis II and affective and other CS-based variables measuring disordered thinking. Of disorder adult patients over and above information from clinical course, research is needed to confirm this logical expectation. interviews (O'Connell, Cooper, Perry, & Hoke 1989). Initial In addition, further research should be conducted to specify the Thought Disorder Index scores also demonstrated clinical utility to association between the Thought Disorder Index and CS summary predict psychotic and psychotic-like symptoms at follow-up. The scores of thought disturbance, using protocols collected according Thought Disorder Index was about three times greater among to CS administrative procedures. Other research goals might be to psychotic children and high-risk children compared with normal conduct prospective, longitudinal studies to confirm that these children (Arboleda & Holzman, 1985). In this study, Thought Rorschach variables can predict (a) the emergence or return of Disorder Index scores in nonpsychotic, hospitalized children were psychoses, (b) negative outcomes in the course of schizophrenic not different from those of normal children. In a family adoptive spectrum disorders, and (c) the development of these disorders study, Wahlberg et al. (1997) found that genetic and environmental among high-risk children. It would be useful to know whether factors interacted to produce elevated Thought Disorder Index these associations are moderated by Rorschach variables and col- scores. lateral non-Rorschach information. For example, simple or preser- Empirical data also associate Rorschach measures of thought vative records may be associated with false-negative Rorschach disorder with more direct biological measures of brain functioning. thought disorder findings, given that the perseveration evident in For example, Daniels, Shenton, et al. (1988) found that patients Rorschach records is poorly captured by existing scores. Improve- with right hemisphere damage displayed fragmented thinking, ments in the measurement of thought disorder within the CS are manic patients displayed playful thinking (as measured by sub- possible (e.g., Kleiger, 1999; Perry, Potterat, Auslander, Kaplan, & components of the Thought Disorder Index), and schizophrenic Jeste, 1996), including the possibility of incorporating aspects of patients used idiosyncratic thinking. A series of studies (Perry & the Thought Disorder Index, adding the EII to the CS, developing Braff, in press; Perry & Braff, 1994; Perry et al., 1999) demon- more meaningful measures of perseveration, and refining proce- strates relationships between (a) the EII and its subcomponents and dures for coding ALOG and DR so that they more selectively (b) neurophysiological criterion measures of "gating." (Gating is identify problem-solving failures. This research should also ad- the ability to filter out irrelevant information in the environment, dress the sensitivity of Rorschach variables to problem-solving and it is considered to be a deficit in the neural circuitry of limitations in relatively well-functioning individuals. schizophrenia.) In one study, these information-processing crite- rion variables and Rorschach responses were measured in a near- Suicide simultaneous computer paradigm (Perry et al., 1999). Information- processing deficits were not correlated with symptom rating scales The CS S-CON is an actuarial variable that yields a single score but were correlated with form quality minus (X— %) and cognitive to assess suicide risk from a composite of 12 variables and ratios. special scores (WSUM6). X—% independently accounted for 57% On the basis of all the available data and the cost-benefit of (p < .001) of the variance in the neurophysiological measure and false-positive versus false-negative errors, continued use of the contributed 35% (p < .01) of the unique variance beyond the S-CON is in order (Viglione, 1999). It should not be used to rule clinician-rated Schedule for Positive Symptoms Delusion subscale out suicide risk, but it can be used to increase awareness about score. The Schedule for Positive Symptoms Delusion scale was self-destructive behavior and suicide. selected among five symptom scales, because it had the greatest The Exner and Wylie (1977) study still qualifies as one of the correlation with the neurophysiological measure. Thus, X—% more methodologically rigorous studies of suicide prediction. In demonstrated incremental validity over a variety of symptom the original study of 59 completed adult suicides, the authors found scales. In this study the EII was not calculated because of the a total S-CON score of 8 positive indices correctly identified 75% restriction of the number of responses, due to the demands of the of the patients who subsequently died as a result of their suicide computer-generated paradigm. Nevertheless, these results support attempt. Fowler, Piers, Hilsenroth, Hold wick, and Padawer (2001) the hypothesis that the EII and its subcomponents tap into the examined the predictive relationship between the CS S-CON and cognitive dysfunction that is observed among schizophrenia lethality of suicide attempts within 60 days following administra- patients. tion of the Rorschach. This study uses an ecologically valid and Challenges, conclusions, and future research. Garb et al. compelling behavioral criterion. Patient records were reliably rated (2001) took exception to Viglione's (1999) claim that Rorschach as nonsuicidal (n = 37), parasuicidal (n = 37), or near lethal (n = variables can be helpful in determining biological vulnerability for 30), based on the presence and lethality of self-destructive acts. schizophrenia and that these variables have been linked to biolog- The S-CON differentiated patients who would later engage in ical processes associated with schizophrenia. To do so, they iso- near-lethal suicide attempts from those who would engage in lated two individual studies from the mass of findings in the parasuicidal behavior and from those who did not engage in literature. The research with various Rorschach measures of self-destructive behaviors. An S-CON score of 7 or more success- 462 VIGLIONE AND HILSENROTH fully predicted which patients would engage in near lethal suicidal tactics and routines, particularly the interpretive search routines, activity relative to parasuicidal patients (overall correct classifica- were developed to minimize problems identified in clinical judg- tion [OCC] rate = .79), nonsuicidal inpatients (OCC = .79), and ment studies. college students (OCC = .89). Logistic regression analysis re- The current CS combination of actuarial and clinical methods vealed that an S-CON score of 7 or more was the sole predictor (Exner, 1991, 1993; Strieker & Gold, 1999; Weiner, 1999) is (p < .01) of emergency medical transfers due to suicide attempts consistent with thinking in the field of judgment and decision from among nine demographic, IQ, psychiatric, and Rorschach making (e.g., Garb, 1989). This clinical-actuarial method uses predictors. Some of these predictors included clinician-rated global procedures whose goal is to minimize cognitive judgment errors psychiatric severity, GAP scores, and the presence of DSM-IV and distortions.6 These procedures are probably ideal in addressing Axis I and II disorders. Thus, this study demonstrates the incre- unique questions in clinical contexts where no two cases are alike. mental validity of this Rorschach scale beyond clinician interview In contrast to the dated clinician judgment analogue research, in and clinician ratings. The total S-CON score also predicted which which the participant clinician interprets a limited, predefined, and patients would have a subsequent drug overdose (r = .32, p = idiosyncratic set of data, the clinician in the field uses data to .001) within 60 days of the Rorschach administration. develop, to reformulate, and to refine referral questions. The To challenge the S-CON, Garb et al. (2001) focused on two clinician also uses expertise to gather or to select accurate data studies that used postattempt suicide criteria (Meyer, 1993; Silberg relevant to those questions. In doing so, the examiner constantly & Armstrong, 1992) and one presentation that has been described interacts with emerging data sets, to change existing questions, only in reviews by Eyman and Eyman (1987, 1991, 1992). How- probabilities, and base rates (Meyer et al., 1998). Along the way, ever, direct examination of these reviews reveals empirical support the examiner brings extreme base rates nearer to levels in which for the efficacy for two single sign suicide predictor variables— testing data would be more efficient in decision making. In that transparency responses and color-shading blends. These scores are sense, the single event, low base analogy found in the literature, for incorporated into shading, form dimension, vista, and color- example, "Is this client thought disordered?" as addressed by 5 shading subcomponents of the S-CON. The Rorschachs in the Hunsley and Bailey (1999, p. 272), is an oversimplified analogue Eyman and Eyman study were administered in the Rapaport- to clinical decision making in the field. Schafer system (Rapaport, Gill, & Schafer, 1946; Schafer, 1954) In actual clinical practice, decisions with low base rate phenom- but recoded in the CS. The Rapaport-Schafer system does not ena rest on the consideration and synthesis of many small data contain all the same variables as the CS so that inquiry questions points. An example of such a situation may be the evaluation of a do not elicit all relevant information. Translation of these protocols schizophrenic-like thought disorder in outpatient or forensic con- to CS variables on an actuarial basis with specific cutoffs may not texts. Synthesizing such data as history, symptoms, observations of be justified without further research. thought disorder, and related characteristics can eventually lead to As a whole, the empirical data support the continued use of the optimal base rates closer to 50% (Hunsley & Bailey, 1999). In S-CON. The decision-making utility of the S-CON may not have other words, in a patient without the history and observations that been exhaustively evaluated, but based on substantial positive might lead to a sufficiently high probability of diagnosing a results one cannot ignore the S-CON data if the false-negative schizophrenic-like thought disorder, the clinician with positive error means losing a life. The most enlightened clinical interpre- evidence of confused thinking on the Rorschach might search for tation of an elevated S-CON is to assert that the Rorschach historical or behavioral information consistent with (a) excessively indicates additional suicide risk that might not be recognized elaborated, idiosyncratic, and overly abstract thinking (Franklin & through other means. Practice ethics and an S-CON of 7, 8, or Cornell, 1997; Gallucci, 1989) or (b) thinking problems associated more should lead clinicians to consider suicidal risk and to collect with intrusions of traumatic imagery (Holaday & Whittenberg, additional information about self-destructive risk. Of course, the 1994; Sloan, Arsenault, Hilsenroth, Harvill, & Handler, 1995). S-CON may produce false positives in some contexts because of These are alternative conditions that may lead to indices of con- the low base rates of actual suicide (Finn & Kamphuis, 1995). In fused thinking on the Rorschach that have been supported by real-life clinical decision making, which takes into consideration a research. wide range of contextual and collateral information, the cost of this false-positive error is minimal compared with the cost of false- Such a clinical-actuarial model is consistent with the threshold negative error (i.e., missing an imminent suicide). model described by Rogers (1997), which mitigates errors in evaluating malingering. It also explains why configurational, syn- thetic approaches, as described by Strieker and Gold (1999) and as Clinician Judgment demonstrated by Weiner (1999), are fundamental to the clinical- actuarial method. In this form, this method minimizes the possi- Ecologically Valid Clinical-Actuarial Judgment bility of a false-positive actuarial interpretation. This clinical- The goal of a large portion of the decisions in formulating the administration procedures of the CS was to minimize examiner 5 The Eyman and Eyman study was not included in the journals identi- and contextual effects identified in earlier research (Exner, 1974). fied by Meyer (1999) and was not published as a research report. Thus, Interrater reliability and temporal consistency data summarized Viglione (1999) did not consider it according to the predetermined, objec- earlier in this article, as well as other research data (e.g., Brinder- tive inclusion criteria established for his review. son, 1995; Cheyette, 1992; Perry, Sprock, Schaible, McDougall, et 6 There is no a priori reason that this actuarial-clinical—nomothetic- al., 1995; Viglione & Exner, 1983), suggest that these problems idiographic approach would have to be confined to CS variables. Other have been minimized. Similarly, a large portion of the interpretive variables with a sound research basis can certainly be added. SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 463 actuarial method with the Rorschach uses an ecologically valid and validity is to be addressed quantitatively, an additional challenge is informed understanding of interactive probabilities to increase power. Sample sizes must be very large or multiple clinical deci- accuracy of in vivo decision making in applied psychology. sions must be evaluated to be able to rule out false-negative results. Meehl and Rosen (1955) addressed this issue in the form of the "successive-hurdles" approach. Their example considered separate Utility, Cost-Benefit, and the Rorschach tests for a brain tumor decision, and they described an increase in hit rates. Meehl and Rosen reported small relative increases in hit The Rorschach allows us to study samples of behavior collected rates over base rates and large relative drops in false-negative under similar conditions in different native languages around the errors with this procedure. For example, successive tests with (a) world (Erdberg & Schaffer, 1999). There is an efficiency to 90% true positive and 15% false positive for Test 1, followed by sampling behaviors with a single technique to develop one's (b) Test 2 with 80% true positive and 10% false positive would understanding and appreciation of developmental changes across increase hit rates from 90% to 96%. In this example, overall errors the age span and across all types of disorders and problems. are cut by more than half, and false-negative errors are reduced by To make cost-benefit comparisons ecologically valid, one would more than three times, from 10% to 3%. The improvement in have to envision a full range of costs and benefits equivalent to the clinical utility is a function of the cost of these false-positive and Rorschach. The alternative to the Rorschach would have to address false-negative errors. In psychological evaluation, this process a wide range of clinical, personality, forensic, developmental, contains many more interactive steps. Thus, illness course, symp- cognitive, and neuropsychological constructs and applications tom report, Rorschach scores for thought disorder, observations of across the life span and around the world. The Rorschach allows thought-disordered behavior in interview, and other Rorschach one to develop a common, experientially based database for the variables and observations come together. On the basis of each of problem-solving practices and predilections of clients from age 5 these "tests," the examiner chooses to observe more data to con- through old age.8 One could identify many other single purpose tinue to test the hypotheses. Both a yes-no decision and an scales within a specific content-construct area that might compete idiographic, qualitative description of the problem emerge in the well with the Rorschach, but to produce the same benefits, one typical clinical or forensic Rorschach application. would have to master and monitor developments in perhaps 50 The clinical-actuarial judgment method is further complicated alternative instruments. Another cost is the expense and hassle of when one adds the strengths, weaknesses, and predilections of the purchasing kits, test blanks, and computer programs and paying individual examiner (Strieker & Gold, 1999). Along these lines, per use fees. Also, the Rorschach provides an efficient way to and in response to criticisms by Wood, Lilienfeld, Garb, and collect a behavior sample outside of interview behavior. Psychol- Nezworski (2000), Garfield (2000) emphasized the skill of the clinician as an important determinant when using the Rorschach in 7 applied practice. To study these issues, more creative and ecolog- In addressing clinical utility, Garb et al. (2001) presented a factually ically valid ways of studying the clinician and improving in vivo incorrect and misleading picture of Viglione's (1999) review of the liter- ature on clinician judgment: "Half of the studies cited by Viglione were applied judgment need to be developed both within the quantita- published in the 1960s and 1970s, with the remainder of the studies tive tradition, but also in terms of qualitative research (e.g., published in the 1980s" (p. 437). Viglione cited five research reports, Schaich, 1999). This approach is consistent with the APA Psycho- which met a priori inclusion and exclusion criteria, providing evidence that 7 logical Assessment Work Group report (Meyer et al., 1998.) clinicians can make valid and useful judgments from Rorschach data (Bilett, Jones, & Whitaker, 1982; Cerney, 1984; Dana & Back, 1983; Clinical Judgment: Future Research Dudek & Marchand, 1983; Holzman et al., 1974). Viglione cited three additional works, not as research reports, but merely as descriptions for Further research on clinical judgment is needed not only for the scales used in the five cited studies: (a) Urist (1977) for the Mutuality of Rorschach but also for other types of decision making. It might be Autonomy Scale, (b) Johnston and Holzman (1979) for the Thought illuminating to conduct new studies with the CS and non-CS Disorder Index, and (c) Masling, Rabid, and Blondheim (1967) for the Rorschach Oral Dependency Scale. Garb et al. (2001) argued that Viglione variables, using clinicians who are aware of errors and pitfalls in (1999) consistently ignored negative findings and then listed seven studies clinical judgment. As Garb (1998) stated, "most of the judgment supposedly ignored by Viglione (Albert, Fox, & Kahn, 1980; Bilett et al., studies on the use of projective tests are over 20 years old" (p. 1982; Cochrane, 1972; Gadol, 1969; Golden, 1964; Perez, 1976; Turner, 232), with virtually no studies with the modern CS and other 1966). Only the first two of the studies were among the 445 studies selected Rorschach variables. Such analogue clinical judgment research for review by the editor of this Special Series. These studies addressed the would have to meet ecological validity challenges, namely, how to incremental validity of judgments and the relationship between the validity reproduce the clinical-actuarial method in the lab, how to motivate of judgments and confidence or training, rather than simply the issue of clinicians and enforce the real life cost-benefit ratios for accurate judgment. Garb et al. (2001) also asserted that "in fact, positive results have versus inaccurate judgments, and how to make the questions and never been obtained for the Rorschach in studies on clinical judgment and decisions sufficiently idiographic, meaningful, and malleable. De- [italics added] incremental validity, confidence and [italics added] validity, and training and [italics added] validity" (p. 437). To clarify this statement, signs must also incorporate better criterion measures. Reproducing no single Rorschach study simultaneously offers (a) a demonstration of the cogwheeling interactive approach (Meyer et al., 1998), vicar- valid clinician judgment with incremental validity techniques, (b) a dem- ious functioning of the respondent and interpreter in facing un- onstration of an association between judges' avowed confidence and ac- bounded problems (Hammond, 1955; Levy, 1963) in an ecologi- curacy of judgments, and (c) a demonstration of an association between cally valid way is critically important. judges' training and accuracy of judgment. Lack of ecological validity, poor criteria, and the failure to 8 The Rorschach has applications below age 5 for testing developmental motivate clinicians all reduce potential effect sizes. If incremental issues (Leichtman, 1996). 464 VIGLIONE AND HILSENROTH ogists' time is valuable, but so is the client's time; therefore, it is the following: "Whether viewed from the perspective of research another cost to be entered into cost-benefit analysis. Mastering one attention or practical usage, the Rorschach inkblot technique con- test, the Rorschach, is efficient because it eliminates the need for tinues to be among the most popular personality assessment meth- many content specific tests. As a result, we can pick and choose ods and predictions about the technique's demise would appear more judiciously and master select instruments within an assess- both unwarranted and unrealistic" (Butcher & Rouse, 1996, p. 91). ment battery, rather than misusing a large number of tests for all the potential purposes that the Rorschach addresses. Forensic Use

Use of the Rorschach: Past, Present, and Future Weiner, Exner, and Schiara (1996) reviewed responses from 93 practioners who testified in over 4,000 criminal cases, over 3,000 Training, Research, and Clinical Use custody cases, and 858 personal injury cases, for a total of almost 8,000 cases. In only 6 of these cases (0.08%) did these respondents Repeated surveys of psychological test use over the past 40 report that the integrity of the test was challenged and in only 1 years have shown a substantial, consistent, and sustained use of the case (0.01%) was it ruled inadmissible. Although Weiner's sam- Rorschach in academic training, research, and clinical settings. pling technique may have limitations, these data provide a "good Surveys consistently indicate that over 80% of graduate programs empirical basis for concluding that, contrary to whatever opinion teach the Rorschach and that students regard this training as may be voiced, the Rorschach is welcome in the courtroom" (pp. important in developing other clinical skills and in understanding 423-424). their patients better, and as useful in their practicum-intemship To address the question of the legal weight, or the effect the training (e.g., since 1995, Camara, Nathan, & Puente, 2000; Rorschach has on court judgments, Meloy, Hansen, and Weiner Hilsenroth & Handler, 1995; Watkins, Campbell, Nieberding, & (1997) reviewed court of appeals citations from 1945 to 1995. In Hallmark, 1995). Although some surveys report that the Rorschach 90% of the 194 cases, "the admissibility and weight of Rorschach takes more of the clinician's time than many other tests, they also data were not questioned by either the appellant or the respondent report that it is one of the most frequently used (e.g., Camara et al., and were important enough to be mentioned and discussed in the 9 2000). legal ruling by the court of appeal" (p. 60). Typically when Also, 90% of clinical practitioners working in the field ex- questions arose, the practitioner's interpretation or findings de- pressed a belief that clinical students should be competent in rived from the test, rather than the test itself, were challenged. In Rorschach assessment (Watkins et al., 1995). This attitude was only two cases (0.8%) was the test criticized as a psychological shared by internship directors (Durand, Blanchard, & Mindell, measure. These data demonstrate that the Rorschach "has author- 1988). In a recent but relatively small survey of 84 (19%) of the ity, or weight, in higher courts of appeal in the United States" (p. 445 APA-approved internship programs in 1997 (Piotrowski & 61). hi a systematic, comprehensive legal analysis of the admissi- Belter, 1999), internship directors reported that the MMPI-2/ bility of the Rorschach, McCann (1998) agreed with the courts. He MMPI-A, WAIS-R/WAIS-III, and Rorschach were the top three concluded that a focus on the structural summary from the CS assessment instruments used in internship programs and were "meets professional and legal standards for admissibility of psy- considered essential for practicing psychologists. Twenty-seven chometric evidence and expert testimony" (p. 140). training directors reported decreased emphasis on projective as- sessment, whereas 12 reported an increased emphasis. Also, hours of seminar time required by internship programs were virtually Challenges identical for projective and self-report assessment measures. Fi- In an earlier article in this Special Series, Dawes (1999) referred nally, the Rorschach was used more than all five of the behavioral to the "deficiency" (p. 301) of the Rorschach in relation to the assessment instruments listed by Piotrowski and Belter (1999). A criterion of reasonable certainty used in forensic applications. His recent survey of practitioners' test use with adolescents suggests argument, concerning the forensic use of the Rorschach, appears to that testing has declined because of managed care constraints, but be based on some elusive juxtaposition of incremental validity that the Rorschach has retained its status as the second most used statistical analysis with the forensic notion that expert testimony instrument (Archer & Newsom, 2000). should be "incremental" in the forensic sense of aiding the trier of The two most comprehensive recent surveys of predoctoral facts. Contrary to his argument and his own data, the evidence internships, one including 329 (Stedman, Hatch, & Schoenfeld, from the courts suggests that more direct self-report assessment 2000) and another including 324 (Clemence & Handler, 2001) methods may not be as valuable in forensic adversarial evaluation Association of Psychology Postdoctoral and Internship Centers contexts; therefore, the incremental value of the Rorschach in sites (most of which were programs accredited by the APA), terms of assisting the trier of facts increases. In fact, Dawes's own revealed that internship training directors greatly value the Ror- findings indicated that whatever deficiency the Rorschach pos- schach as well as integrated test batteries. Again, training directors sessed, the MMPI also possessed because the Rorschach improved reported a desire for incoming interns to have had courses or a on predictions made by the MMPI. good working knowledge of the Rorschach. Academic depart- Given these data, the recent task force report by Division 12 ments that are making decisions about the place of the Rorschach (Clinical) of the APA (1999), concerning the development of a in their curriculum should closely attend to these data. The Rorschach is the second most frequently researched per- sonality assessment instrument. This consistently high rate of 9 Citations before 1995 that support the assertions in this section are empirical investigation led the authors of a recent review to state excluded for the sake of brevity. SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 465 model assessment curriculum for graduate training in assessment, turn, improve the validity and utility of the test. Others have also is confusing and misguided. In this report, Rorschach training was repeatedly offered methodological recommendations and stan- ranked 95th in importance out of the total 105 topics. The available dards for Rorschach research, and they have used these standards reliability and validity data, coupled with other contemporary to reject findings from individual studies supportive of the Ror- survey material reported earlier, suggest that there is vast discrep- schach (Garb, 1999; Garb et al., 2001; Hunsley & Bailey, 1999; ancy between the Division 12 Task Force future model curriculum Wood et al., 2000; Wood, Nezworski, & Stejskal, 1996a, 1996b). for assessment and actual graduate training needs. Should these These same standards are not fully applied when evaluating studies changes in graduate assessment training occur, students and future cited as yielding negative evidence for the Rorschach. Also, im- clients will have to pay for the misconceptions of graduate faculty. partiality would require applying the same standards to all tests, Also Hunsley and Bailey (1999) cited a dated survey (Wade & rather than reserving them specifically for the Rorschach (Garfield, Baker, 1977) of practicing clinicians to claim that practitioners 2000). In the first set of publications for this Special Series in disregard scientific evidence. Actually, there is empirical evidence Psychological Assessment, Viglione (1999) met a number of the to show quite the contrary. Beutler, Williams, Wakefield, and challenges posed by Rorschach critics. Nevertheless, the positive Entwistle (1995) recently found that clinicians in private practice research findings summarized by Viglione (1999) were met with value research and put forth effort to understand scientific find- the same repeated methodological criticisms (Garb et al., 2001). ings. There is no empirical evidence that indicates that the manner Similarly, other research reports (e.g., Meyer, 1997a; Hilsenroth, in which psychologists typically use the Rorschach is different Fowler, Padawer, & Handler, 1997) have not been recognized as from the standardized methods in which they were trained, and no positive responses to criticisms. In the section below, we examine evidence that psychologists who are using the Rorschach are Garb, Wood, and their colleagues' applications of these criticisms unaware of its research findings. about methodology in Rorschach research.

Conclusion Within and Across Diagnoses The data on the use and perceived value of the Rorschach Viglione's (1999) synthesis of the empirical research supports clearly demonstrate that psychologists find the Rorschach to be of the conclusion that "the Rorschach is useful in examining many value in routine clinical practice. Some psychologists ask why so clinically relevant criteria and behavior within and across diag- many practitioners persist in their use of the Rorschach despite noses" (p. 260). However, in reaction to Strieker and Gold's current time pressures and limited fees by third party payers. This (1999) summary of the literature, endorsing the usefulness of the becomes even more perplexing when self-report questionnaires Rorschach in evaluating depressed, schizophrenic, or acutely sui- and structured interviews may require less of the clinician's time. cidal patients, Garb et al. (2001) asserted that "empirical research Considering the persistent use of the Rorschach in the context of suggests that the Rorschach is not useful for all these tasks" (p. the available reliability and validity information leads to some 433) and cited only Wood et al. (1996a, 1996b) to bolster this obvious conclusions: One must speculate that clinicians and those statement. This claim overlooks a great deal of evidence in support in charge of clinical training find it helpful because of its reliability of the usefulness of the Rorschach in these disorders and associ- and validity relevant to challenges in applied settings. In other ated phenomena. There are numerous examples of research that words, clinicians continue to use the test despite its time require- support the validity of the Rorschach applied to various problems ments because of its clinical utility. within and across diagnoses (e.g., see Viglione, 1999). A substantial body of research demonstrates that individuals exhibit a defensive bias and manipulate their responses when asked to self-report aspects of their personality or psychopathol- Empirical Data in Peer-Reviewed Journals, Accessible to ogy. For example, individual differences in self-serving biases All have been demonstrated in relation to attributions, self- Some critics have maintained that too much of the empirical descriptions, inferences, personality traits, and avoidance of neg- support for the Rorschach contained within studies in Exner's ative affect. Rorschach research supports the view that the Ror- volumes is not published in peer-reviewed journals and is therefore schach is most useful in (a) contexts in which the respondent may not available for review, In response to this criticism, Viglione be unwilling or unable to self-report problems and (b) those in (1999) synthesized research findings from peer-reviewed journals which clinicians are trying to predict real-life behavior (Hiller et only. He reported considerable empirical support for the test. al., 1999; Viglione, 1999). In addition to routine clinical contexts, Viglione focused on the Methods and Results sections, rather than the data suggest that the Rorschach may be especially useful under on the theorizing of the authors. Thus, a careful review of the conditions that might induce response manipulation in self-report. empirical data published in peer-reviewed journals supports the These might include employment, forensic, and custody evalua- utility of the Rorschach. tions, and other settings that involve adversarial examiner- respondent relationship components (Rogers, 1997). Selection of Studies Methodological Challenges to Validity of the Rorschach Rorschach critics (e.g., Garb et al., 2001; Wood et al., 2000; Challenges Wood, Nezworski, Stejskal, Garven, & West, 1999) have claimed that Rorschach researchers selectively present findings. Meyer Rorschach researchers (Exner, 1995; Viglione, 1997) have of- (1999) requested that the contributors of this Special Series ad- fered recommendations to improve Rorschach research and, in dress 445 articles in five targeted journals published between 1977 466 VIGLIONE AND HILSENROTH and 1997. In response, Viglione (1999) applied objective, meta- intention to give readers an informed appreciation for the substan- analytic type selection criteria to select 128 research reports. In tial but often overlooked research basis for the reliability, validity, contrast, Hunsley and Bailey (1999) confined their review to 28 and utility of the Rorschach. From the broadest point of view, we studies, whereas Garb et al. (2001) reviewed only 44 of these same have demonstrated that examining the empirical findings can only 445 articles. In addition, Viglione's selection criteria resulted in lead to the conclusion that the Rorschach is a valuable test (Vigli- the exclusion of supportive Rorschach evidence found not only in one, 1999; Weiner, 1996). Interrater reliability by well-trained Exner's texts, but also in review articles, meta-analyses, and books coders both in the field and in the lab are very strong. Temporal (e.g., Blatt et al., 1994; Exner, 1993; Gacono & Meloy, 1994; consistency data are good, as are the currently available normative Kleiger, 1999; Masling, 1986; Weiner, 1966). Most of the negative data, though both could be improved. Criterion validity for the test studies selected by Garb, which Viglione was said to have ignored, is very strong, and incremental validity and meta-analytic data are were not among those identified by the series editor. On the other present with more data emerging. The current sweeping criticisms hand, Garb et al. uncovered virtually no research supportive of the of the Rorschach are largely without merit, uninformed, and biased Rorschach, even though they were provided with the same 445 against the test. Thus, the Rorschach, by virtue of the accumulated target articles as the other series contributors. Viglione included research, is now "better prepared to stand the tests of reliability and 10 negative findings when they occurred. validation" (Exner, 1974, p. 16). Clinicians persist in using the test because it continues to pro- Experiment-Wise Alpha Levels vide comprehensive and valuable information about their patients. Thus, these clinicians are living proof of the application of these Garb et al. (2001) maintained that the Rorschach literature fails research findings. In turn, the research findings mandate that the to control experiment-wise alpha levels. For example, Garb et al. test continue to be used. We have broadened the scope of the (2001) viewed Lemer and St. Peter's (1984a, 1984b) positive cost-benefit issues and utility analysis in the application of the test findings about borderline personality structure and Rorschach hu- to be more consistent with clinical practice. In doing so, we have man representations as a flawed study because of the failure to identified other obvious benefits to the Rorschach. These include control for false-positive findings. Garb et al. (2001) did not report the use of a single test for many applications and the opportunity that 41 of 64 analyses were significant at p < .05, and 33 of 64 n to collect quantifiable data while simultaneously collecting a be- were significant at p < .01. Mathematically, these positive havioral sample in a context that maximizes individual differences results cannot be explained by chance. All other analyses were in problem solving. The Rorschach serves those who are interested legitimate, post hoc group contrasts for the 41 significant in having a valuable tool to help understand an individual com- comparisons. prehensively and to synthesize information across many domains (Strieker & Gold, 1999). Undoubtedly, the behavior sampling Criterion Contamination and Diagnosis opportunity that occurs during administration promotes these goals and provides another foundation to generalize outside of the con- Critics have faulted Rorschach researchers for using clinician sulting room. diagnosis as a criterion (Garb et al., 2001; Wood et al., 2000). For Acclaim for the Rorschach has elicited criticism (Wood & example, Garb (1998) questioned the use of clinicians as criterion Lilienfeld, 1999). Recent critical commentaries of the Rorschach judges: "Historically, the clinicians making the indicator ratings have identified some areas (e.g., normative data and methodolog- have been called criterion judges and their ratings criterion ratings, ical problems in some studies) in which the Rorschach can be but these terms are inaccurate because the ratings are fallible" (p. improved. Like other tests, the Rorschach has its weaknesses. 14). Nonetheless, these same critics have emphasized studies by However, these recent criticisms are undermined by methodolog- Archer and his colleagues (Archer & Gordon, 1988; Archer & ical double standards, confirmatory bias, incomplete coverage of Krishnamurthy, 1997) to argue against the incremental validity of the literature, failure to integrate positive contributions over the the Rorschach. The Archer and Krishnamurthy (1997) study used last 5 years that have specifically addressed earlier criticisms from the clinical diagnosis from just one clinician as the sole basis for these authors, and a lack of any original data to support their the diagnosis. Moreover, Archer and Gordon (1988) noted that "psychological test results were frequently available and poten- positions. For example, calls for a moratorium on the use of the tially involved in the diagnostic decisions" (p. 279) so that crite- Rorschach (Garb, 1999; Garb et al., 2001) clearly contradict em- rion contamination and bias are possible. Both studies lack reli- pirical findings. Some clinical utility criteria (Hunsley & Bailey, ability statistics for their diagnostic criteria. These studies do not 1999) set such a high and unrealistic standard that implementing meet the methodological standards imposed by the critics on them would result in abandoning all testing, interviews, and ob- studies with evidence supportive of the Rorschach. Yet, these servation for assessment. Useful criticism and reasonable clinical studies are cited as negative evidence for the validity of the validity criteria challenge future research rather than mandate our Rorschach. This is clearly evidence of a double standard used by the critics of Rorschach research. 10 Viglione (1999) cited 19 studies on pages 254-259 with negative findings. Implications and Conclusions 11 Viglione (1999) is criticized for not including Lerner and St. Peter's (1984a) first article. To be factual, the earlier article was published in We have addressed the Rorschach's strengths and weaknesses Psychiatry, a journal excluded from the series editor's targeted list. It was and offered research recommendations in 10 focal areas relevant to never considered for inclusion, according to the objective inclusion criteria the clinical utility of the test. In addressing these issues, it is our adopted in this review. SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 467 forsaking psychological and neuropsychological assessment alto- Bornstein, R. F. (1996). Construct validity of the Rorschach Oral Depen- gether. However, an important contribution from critical reviews dency scale: 1967-1995. Psychological Assessment, 8, 200-205. has been to improve Rorschach research (Acklin, 1999), some- Bornstein, R. F. (1999). Criterion validity of objective and projective thing that is happening much more quickly than was the case after dependency tests: A meta-analytic assessment of behavioral prediction. Rorschach researchers called for similar improvements a few years Psychological Assessment, 11, 48-57. ago (Exner, 1995; Viglione, 1997). Bornstein, R. F., Bowers, K. S., & Robinson, K. J. (1997). Differential relationships of objective and projective dependency scores to self- The informed use of the Rorschach will lead to a more sensitive reports of interpersonal life events in college student subjects. Journal of and helpful understanding of our clients as well as more effective Personality Assessment, 65, 255-269. interventions. Continuing research and development of the test as Brinderson, C. (1995). Apparent nonpatient psychopathology on the Ror- well as the provision for sound training and continuing education schach: Disturbance or disinhibition? Unpublished doctoral disserta- in the Rorschach method, as with any assessment measure, are also tion, California School of Professional Psychology, San Diego, CA. essential. Nevertheless, the extant empirical research findings ne- Burns, B., & Viglione, D. J., Jr. (1996). The Rorschach Human Experience cessitate that any scholar must accept the fact that the Rorschach Variable, interpersonal relatedness, and object representation in nonpa- test method, its administration procedures, and its objective rules tients. Psychological Assessment, 8, 92-99. for quantifying behavior produce valid data and observations. Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A. M., & Kaemmer, B. (1989). MMPI-2: Manual for administration and scoring. References Minneapolis: University of Minnesota Press. Butcher, J. N., & Rouse, S. (1996). Personality: Individual differences and Acklin, M. W. (1999). Behavioral science foundations of the Rorschach clinical assessment. Annual Review of Psychology, 47, 87-111. test: Research and clinical applications. Assessment, 6, 319-324. Cadenhead, K. S., Perry, W., & Braff, D. L. (1996). The relationship of Acklin, M. W., McDowell, C. J., & VerscheU, M. S. (2000). Interobserver information-processing deficits and clinical symptoms in schizotypal agreement, intraobserver reliability, and the Rorschach Comprehensive personality disorder. Biological Psychiatry, 40, 853-858. System. Journal of Personality Assessment, 74, 15—47. Camara, W., Nathan, J., & Puente, A. (2000). Psychological test usage: Adair, H. E., & Wagner, E. E. (1992). Stability of unusual verbalizations Implications in professional use. Professional Psychology: Research on the Rorschach for outpatients with schizophrenia. Journal of Clinical and Practice, 31, 141-154. Psychology, 48, 250-256. Ceraey, M. S. (1984). One last response to the Rorschach test: A second Albert, S., Fox, H. M., & Kahn, M. W. (1980). Faking psychosis on the chance to reveal oneself. Journal of Personality Assessment, 48, 338- Rorschach: Can expert judges detect malingering? Journal of Person- 344. ality Assessment, 44, 115-119. Chapman, L. J., Chapman, J. P., & Kwapil, T. R. (1995). Scales for the American Psychiatric Association. (1994). Diagnostic and statistal manual measurement of schizotype. In A. Raine. T. Lencz, & S. A. Mednick of mental disorders (4th ed.). Washington, DC: Author. (Eds.), Schizotypal personality (pp. 79-106). New York: Cambridge American Psychological Association, Division 12, Presidential Task Force University Press. (1998). Assessment for the twenty-first century: A model curriculum. Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1976). Scales for The Clinical Psychologist, 52, 10-15. physical and social anhedonia. Journal of Abnormal Psychology, 85, Anastasi, A., & Urbina, S. (1996). Psychological testing (7th ed.). New 374-382. York: Macmillan. Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1978). Body-image Arboleda, C., & Holzman, P. S. (1985). Thought disorder in children at risk aberration in schizophrenia. Journal of Abnormal Psychology, 87, 399- for psychosis. Archives of General Psychiatry, 42, 1004-1013. 407. Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices of Cheyette, L. (1992). The effects of examiner style on Rorschach test results schizophrenic and depressive diagnoses among adolescent inpatients. of children with borderline psychopathology. Unpublished doctoral dis- Journal of Personality Assessment, 52, 276-287. sertation, California School of Professional Psychology, San Diego, CA. Archer, R. P., & Krishnamurthy, R. (1997). MMPI-A and Rorschach Clemence, A., & Handler, L. (2001). Psychological assessment on intern- indices related to depression and conduct disorder An evaluation of the ship: A survey of training directors and their expectations for students. incremental validity hypotheses. Journal of Personality Assessment, 69, Journal of Personality Assessment, 76, 18-47. 517-533. Cochrane, C. T. (1972). Effects of diagnostic information on empathic Archer, R. P., & Newsom, C. R. (2000). Psychological test usage with understanding by the therapist in a psychotherapy analogue. Journal of adolescent clients: Survey update. Assessment, 7, 227-235. Consulting and Clinical Psychology, 38, 359-365. Atkinson, L., Quarrington, B., Alp, I. E., & Cyr, J. J. (1986). Rorschach Coleman, M. J., Levy, D. L., Lenzenweger, M. F., & Holzman, P. S. validity: An empirical approach to the literature. Journal of Clinical (1996). Thought disorder, perceptual aberrations, and schizotypy. Jour- Psychology, 42, 360-362. Beutler, L. E., Williams, R. E., Wakefield, P. J., & Entwistle, S. R. (1995). nal of Abnormal Psychology, 105, 469-473. Bridging scientist and practitioner perspectives in clinical psychology. Colligan, R. C., Osbome, D. M. S. W., & Offord, K. P. (1984). The aging American Psychologist, 50, 984-994. MMPI: Development of contemporary norms. Mayo Clinic Proceed- Bilett, J. L., Jones, N. F., & Whitaker, L. C. (1982). Exploring schizo- ings, 59, 377-390. phrenic thinking in older adolescents with the WAIS, Rorschach and Cooper, S. H., Perry, J. C., & O'Connell, M. (1991). The Rorschach WISC. Journal of Clinical Psychology, 38, 232-243. Defense Scales: II. Longitudinal perspectives. Journal of Personality Blais, M., Hilsenroth, M., Castlebury, F., Fowler, C., & Baity, M. (2001). Assessment, 56, 191-201. Predicting DSM-W Cluster B personality disorder criteria from MMPI-2 Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. (1975). A MMPI and Rorschach data: A test of incremental validity. Journal of Person- handbook: Vol. 11: Research application. Minneapolis: University of ality Assessment, 76, 150-168. Minnesota Press. Blatt, S. J., Ford, R. Q., Berman, W. H., Jr., Cook, B., Cramer, P., & Dana, R. H., & Back, B. R. (1983). The concurrent validity of child Robins, C. E. (1994). Therapeutic change: An object relations perspec- Rorschach interpretations. Journal of Personality Assessment, 47, 3-6. tive. New York: Plenum. Daniels, E. K., Shenton, M. E., et al. (1988). Patterns of thought disorder 468 VIGLIONE AND HILSENROTH

associated with right cortical damage, schizophrenia, and mania. Amer- Yufit (Eds.), Assessment and prediction of suicide (pp. 183—201). New ican Journal of Psychiatry, 145, 944-949. York: Guilford Press. Dawes, R. M. (1999). Two methods for studying incremental validity of a Eyman, S. K., & Eyman, J. R. (1987, August-September). An investigation Rorschach variable. Psychological Assessment, 11, 297-302. of Exner's Suicide Constellation. Paper presented at the 95th Annual Dudek, S. Z., & Marchand, P. (1983). Artistic style and personality in Convention of the American Psychological Association, New York. creative painters. Journal of Personality Assessment, 47, 139—142. Finn, S. E., & Kamphuis, i. H. (1995). What a clinician needs to know Durand, V., Blanchard, E., & Mindell, J. (1988). Training in projective about baserates. In J. N. Butcher (Ed.), Clinical personality assessment testing: Survey of clinical training directors and internship directors. (pp. 224-235). New York: Oxford University Press. Professional Psychology: Research and Practice, 19, 236-238. Rske, D. W., Cartwright, D. S., & Kirtner, W. L. (1964). Are psychother- Eckblad, M. L., Chapman, L. J., & Chapman, J. P. (1982). The Revised apeutic changes predictable? Journal of Abnormal and Social Psychol- Social Anhedonia Scale. Unpublished manuscript. ogy, 69, 413-426. Edell, W. S., & Chapman, L. J. (1979). Anhedonia, perceptual aberration, Fowler, C., Hilsenroth, M. J., & Handler, L. (1996). A multimethod and the Rorschach. Journal of Consulting and Clinical Psychology, 47, approach to assessing dependency: The early memory dependency 377-384. probe. Journal of Personality Assessment, 67, 399-413. Erdberg, P., & Schaffer, T. W. (1999, July). International symposium on Fowler, C. J., Piers, C. C., Hilsenroth, M. J., Holdwick, D., & Padawer, R. Rorschach nonpatient data: Findings from around the world. Paper (2001). Assessing risk factors various degrees of suicidal activity: The presented at the International Congress of Rorschach and Projective Rorschach Suicide Constellation. Journal of Personality Assessment, 76, Methods, Amsterdam, the Netherlands. 333-351. Erstad, D. (1996). An investigation of older adults' less frequent human Franklin, K. W., & Cornell, D. G. (1997). Rorschach interpretation with movement and color responses on the Rorschach. Unpublished doctoral high-ability adolescent females: Psychopathology or creative thinking? dissertation, Marquette University. Journal of Personality Assessment, 68, 184-196. Exner, J. E. (1974). The Rorschach: A Comprehensive System (Vol. 1). Frueh, B. C., & Kinder, B. N. (1994). The susceptibility of the Rorschach New York: Wiley. inkblot test to malingering of combat-related PTSD. Journal of Person- Exner, J. E. (1991). The Rorschach: A Comprehensive System: Vol. 2. ality Assessment, 62, 280-298. Interpretation (2nd ed.). New York: Wiley. Gacono, C. B., & Meloy, J. R. (1994). The Rorschach assessment of Exner, J. E. (1993). The Rorschach: A Comprehensive System: Vol. 1. aggressive and psychopathic personalities. Mahwah, NJ: Erlbaum. Basic foundations (3rd ed.). New York: Wiley. Gadol, I. (1969). The incremental and predictive validity of the Rorschach test in personality assessment of normal, neurotic, and psychotic sub- Exner, J. E. (Ed.). (1995). Issues and methods in Rorschach research. jects. Dissertation Abstracts International, 29, 3482-B. (University Mi- Hillsdale, NJ: Erlbaum. crofilms No. 3469-4469) Exner, J. E. (1999). The Rorschach: Measurement concepts and issues of Gallucci, N. T. (1989). Personality assessment with children of superior validity. In S. E. Embretson & S. L. Hershberger (Eds.), The new rules intelligence: Divergence versus psychopathology. Journal of Personal- of measurement: What every psychologist and educator should know ity Assessment, 53, 749-760. (pp. 159-183). Mahwah, NJ: Erlbaum. Garb, H. N. (1989). Clinical judgment, clinical training, and professional Exner, J. E. (2000). A primer for Rorschach interpretation. Asheville, NC: experience. Psychological Bulletin, 105, 387-396. Rorschach Workshops. Garb, H. N. (1994). Cognitive heuristics and biases in personality assess- Exner, J. E., Armbruster, G. L., & Viglione, D. (1978). The temporal ment. In L. Heath, R. S. Tindale, J. Edwards, E. J. Posavac, F. B. Bryant, stability of some Rorschach features. Journal of Personality Assess- E. Henderson-King, Y. Suarez-Balcazar, & J. Myers (Eds.), Applications ment, 42, 474-482. of heuristics and biases to social issues. New York: Plenum Press. Exner, J. E., Colligan, S. C., Hillman, L. B., Metis, A. S., Ritzier, B., Garb, H. N. (1998). Studying the clinician: Judgment research and psy- Rogers, K. T., Sciara, A., D., & Viglione, D. J. (2001). A Rorschach chological assessment. Washington, DC: American Psychological workbook for the Comprehensive System (5th ed.). Asheville, NC: Ror- Association. schach Workshops. Garb, H. N. (1999). Call for a moratorium on the use of the Rorschach Exner, J. E., Colligan, S. C., Hillman, L. B., Ritzier, B., Sciara, A. D., & Inkblot Test in clinical and forensic settings. Assessment, 6, 313-315. Viglione, D. J. (1995). A Rorschach workbook for the Comprehensive Garb, H. N., Florio, C. M., & Grove, W. M. (1998). The validity of the System (4th ed.). Asheville, NC: Rorschach Workshops. Rorschach and the Minnesota Mulu'phasic Personality Inventory: Re- Exner, J. E., Thomas, E. A., & Mason, B. J. (1985). Children's Rorschachs: sults from meta-analyses. Psychological Science, 9, 402-404. Description and prediction. Journal of Personality Assessment, 49, 13- Garb, H. N., Wood, J. M., Nezworski, M. T., Grove, W. M., & Stejskal, 20. W. J. (2001). Toward a resolution of the Rorschach controversy. Psy- Exner, J. E., & Weiner, I. B. (1982). The Rorschach: A Comprehensive chological Assessment, 13, 433-448. System: Vol. 3. Assessment of children and adolescents. New York: Garfield, S. L. (2000). The Rorschach test in clinical diagnosis—A brief Wiley. commentary. Journal of Clinical Psychology, 56, 431-434. Exner, J. E., & Weiner, I. B. (1995). The Rorschach: A Comprehensive Golden, M. (1964). Some effects of combining psychological tests on System: Vol. 3. Assessment of children and adolescents (2nd ed.). New clinical inferences. Journal of Consulting Psychology, 28, 440-446. York: Wiley. Greenwald, D. F. (1991). Personality dimensions reflected by the Ror- Exner, J. E., Weiner, I. B., & Schuyler, W. (1976). A Rorschach workbook schach and the 16PF. Journal of Clinical Psychology, 47, 708-715. for the Comprehensive System. Bayville, NY: Rorschach Workshops. Haller, N., & Exner, J. E. (1985). The reliability of Rorschach variables for Exner, J. E., & Wylie, J. (1977). Some Rohrschach data concerning inpatients presenting symptoms of depression and/or helplessness. Jour- suicide. Journal of Personality Assessment, 41, 339-348. nal of Personality Assessment, 49, 516-521. Eyman, J. R., & Eyman, S. K. (1991). Personality assessment in suicide Hamel, M., Shaffer, T. W., & Erdberg, P. (2000). A study of nonpatient prediction. Special Issue: Assessment and prediction of suicide. Suicide preadolescent Rorschach protocols. Journal of Personality Assess- and Life Threatening Behavior, 21, 37-55. ment, 75, 280-294. Eyman, J. R., & Eyman, S. K. (1992). Personality assessment in suicide Hammond, K. R. (1955). Probabilistic functioning and the clinical method. prediction. In R. W. Maris, A. L. Berman, J. T. Maltsberger, & R. I. Psychological Review, 62, 255-262. SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 469

Hathaway, S. R., McKinley, J. C, Butcher, J. N., Dahlstrom, W. G., Form. Unpublished doctoral dissertation, University of Minnesota, Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota Mul- Minneapolis. tiphasic Personality Inventory-2: Manual for administration and scor- McCann, J. T. (1998). Defending the Rorschach in court: An analysis of ing. Minneapolis: University of Minnesota Press. admissibility using legal and professional standards. Journal of Person- Miller, J. B., Rosenthal, R., Bornstein, R. F., & Berry, D. T. (1999). A ality Assessment, 70, 125-144. comparative meta-analysis of Rorschach and MMPI validity. Psycho- McDowell, C. J., & Acklin, M. W. (1996). Standardizing procedures for logical Assessment, 11, 278-296. calculating Rorschach interrater reliability: Conceptual and empirical Hilsenroth, M. J., Fowler, C. J., & Padawer, J. R. (1998). The Rorschach foundations. Journal of Personality Assessment, 66, 308-320. Schizophrenia Index (SCZI): An examination of reliability, validity, and Meehl, P. E. (1957). When shall we use our heads instead of the formula? diagnostic efficiency. Journal of Personality Assessment, 70, 514-534. Journal of Counseling Psychology, 4, 268-273. Hilsenroth, M. J., Fowler, J. C., Padawer, J. R., & Handler, L. (1997). Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the effi- Narcissism in the Rorschach revisited: Some reflections on empirical ciency of psychometric signs, patterns, or cutting scores. Psychological data. Psychological Assessment, 9, 113-121. Bulletin, 52, 171-190. Hilsenroth, M. J., & Handler, L. (1995). A survey of graduate students' Meloy, J. R., Hansen, T. L., & Weiner, I. B. (1997). Authority of the experiences, interests, and attitudes about learning the Rorschach. Jour- Rorschach: Legal citations during the past 50 years. Journal of Person- nal of Personality Assessment, 64, 243—257. ality Assessment, 69, 53—62. Hilsenroth, M. J., Handler, L., Toman, K. M., & Padawar, J. R. (1995). Meyer, G. J. (1993). The impact of response frequency on the Rorschach Rorschach and MMPI-2 indices of early psychotherapy termination. constellation indices and on their validity with diagnostic and MMPI-2 Journal of Consulting and Clinical Psychology, 63, 956—965. criteria. Journal of Personality Assessment, 60, 153-180. Holaday, M., & Whittenberg, T. (1994). Rorschach responding in children Meyer, G. J. (1997a). Assessing reliability: Critical correlations for a and adolescents who have been severely burned. Journal of Personality critical examination of the Rorschach Comprehensive System. Psycho- Assessment, 62, 269-279. logical Assessment, 9, 480-489. Holzman, P. S., et al. (1974). Eye-tracking dysfunctions in schizophrenic Meyer, G. J. (1997b). Thinking clearly about reliability: More critical patients and their relatives. Archives of General Psychiatry, 31, 143- correlations regarding the Rorschach Comprehensive System. Psycho- 151. logical Assessment, 9. 495-498. Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach: Meyer, G. J. (1999). Introduction to the Special Series on the utility of the Unfulfilled promises and an uncertain future. Psychological Assess- Rorschach for clinical assessment. Psychological Assessment, 11, 235- ment. 11, 266-277. 239. Hunsley, J., Hanson, R. K., & Parker, K. C. (1988). A summary of the Meyer, G. J. (2000). Incremental validity of the Rorschach Prognostic reliability and stability of MMPI scales. Journal of Clinical Psychol- Rating Scale over the MMPI Ego Strength Scale and IQ. Journal of ogy, 44, 44—46. Personality Assessment, 74, 356-370. Johnston, M. H., & Holzman, P. S. (1979). Assessing schizophrenic think- Meyer, G. J. (2001). Evidence to correct misperceptions about Rorschach ing: A clinical and research instrument for measuring thought disorder. norms. Manuscript submitted for publication. San Francisco: Jossey-Bass. Meyer, G. J. (in press). Exploring possible ethnic differences and bias in Kates, J. (1994). A validation study of the Rorschach Ego Impairment Rorschach Comprehensive System scores. Journal of Personality As- Index. Unpublished doctoral dissertation, California School of Profes- sessment. sional Psychology, San Diego, CA. Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Kubiszyn, T. W., Kleiger, J. H. (1999). Disordered thinking and the Rorschach: Theory, Moreland, K. L., Eisman, E. J., & Dies, R. R. (1998). Benefits and costs research, and differential diagnosis. Hillsdale, NJ: Analytic Press. of psychological assessment in healthcare delivery: Report of the Board Kleinmuntz, B. (1990). Why we still use our heads instead of formulas: of Professional Affairs Psychological Assessment Work Group, Part 1. Toward an integrative approach. Psychological Bulletin, 107, 296-310. Washington, DC: American Psychological Association. Krall, V., Sachs, H., Langer, B., Rayson, B., & Growe, G. (1983). Ror- Meyer, G. J., & Handler, L. (1997). The ability of the Rorschach to predict schach norms for inner city children. Journal of Personality Assess- subsequent outcome: A meta-analysis of the Rorschach Prognostic Rat- ment, 47, 155-157. ing Scale. Journal of Personality Assessment, 69, 1-38. Leichtman, M. (1996). The Rorschach: A developmental perspective. Hills- Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E., Jr., Fowler, C. J., dale, NJ: Analytic Press. Piers, C. C., & Resnick, J. (in press). An examination of interrater Lerner, H. D., & St. Peter, S. (1984a). Patterns of object relations in reliability for scoring the Rorschach Comprehensive System in eight neurotic, borderline and schizophrenic patients. Psychiatry, 47, 77-92. data sets. Journal of Personality Assessment. Lemer, H. D., & St. Peter, S. (1984b). The Rorschach H response and Meyer, G. J., & Resnick, G. D. (1996, February). Assessing ego impair- object relations. Journal of Personality Assessment, 48, 345-350. ment: Do scoring procedures make a difference? Paper presented at the Levy, L. H. (1963). Psychological interpretation. New York: Holt, Rine- 15th International Research Conference, Boston. hart & Winston. Milott, S. R., Lira, F. T., & Miller, W. C. (1977). Psychological assessment Masling, J. M. (1986). Empirical studies of psychoanalytic theories. Hills- of the burned patient. Journal of Clinical Psychology, 33, 425-430. dale, NJ: Erlbaum. Netter, B. E. C., & Viglione, D. J. (1994). An empirical study of malin- Masling, J. M., Rabid, L., & Blondheim, S. H. (1967). Obesity, level of gering schizophrenia on the Rorschach. Journal of Personality Assess- aspiration, and Rorschach and TAT measures of oral dependence. Jour- ment, 62, 45-57. nal of Consulting Psychology, 31, 233-239. O'Connell, M., Cooper, S., Perry, C., & Hoke, L. (1989). The relationship Mattlar, C. E., Carlsson, A., & Forsander, C. (1993). The issue of the between thought disorder and psychotic symptoms in borderline person- popular response: Definition and universal vs. culture-specific popular ality disorder. Journal of Nervous and Mental Disease, 177, 273-278. responses. Special Issue: International Rorschach Congress 1993. British Paolo, A. M., Troster, A. I., & Ryan, J. J. (1997). Test-retest stability of the Journal ofProjective Psychology, 38, 53-62. California Learning Test in older persons. Neuropsychology, 11, 613- Mauger, P. A. (1972). The test-retest reliability of persons: An empir- 616. ical investigation utilizing the MMPI and the Personality Research Parker, K. C., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorschach, and 470 VIGLIONE AND HILSENROTH

WAIS: A meta-analytic comparison of reliability, stability, and validity. dures involved with the assessment of malingering: An applied study. Psychological Bulletin, 103, 367-373. Unpublished doctoral dissertation, California School of Professional Parker, K. C. H., Hunsley, J., & Hanson, R. K. (1999). Old wine from old Psychology, San Diego, CA. skins sometimes taste like vinegar: A response to Garb, Florio, and Schuerger, J. M., Zarrella, K. L., & Holtz, A. S. (1989). Factors that Grove. Psychological Science, 10, 367-373. influence the temporal stability of personality by questionnaire. Journal Perez, F. I. (1976). Behavioral analysis of clinical judgment. Perceptual of Personality and Social Psychology, 56, 777-783. and Motor Skills, 43, 711-718. Schwartz, N. S., Mebane, D. L., & Malony, H. N. (1990). Effects of Perry, W., & Braff, D. L. (1994). Information-processing deficits and alternate modes of administration of Rorschach performance of deaf thought disorder in schizophrenia. American Journal of Psychiatry, 151, adults. Journal of Personality Assessment, 54, 671-683. 363-367. Shaffer, T. W., Erdberg, P., & Haroian, J. (1999). Current nonpatient data Perry, W., & Braff, D. L. (in press). The simultaneous measurement of for the Rorschach, WAIS-R, and MMPI-2. Journal of Personality attention and information processing and disturbed thinking in male Assessment, 73, 305-316. schizophrenic patients. Archives of General Psychiatry. Shapiro, J. P., Leifer, M., Martone, M. W., & Kassem, L. (1990). Multi- Perry, W., Geyer, M. A., & Braff, D. L. (1999). Sensorimotor gating and method assessment of depression in sexually abused girls. Journal of thought disturtence measured in close temporal proximity in schizo- Personality Assessment, 55, 234-248. phrenic patients. Archives of General Psychiatry, 56, 277—281. Shenton, M. E., Solovay, M. R., et al. (1989). Thought disorder in the Perry, W., Moore, D., & Braff, D. L. (1995). Gender differences on thought relatives of psychotic patients. Archives of General Psychiatry, 46, disturbance measures in schizophrenia. American Journal of Psychiatry, 897-901. 152, 1298-1301. Shrout, P. E., & Fliess, J. L. (1979). Intraclass correlations: Uses in Perry, W., Potterat, E. G., Auslander, L., Kaplan, E., & Jeste, D. (19%). A assessing rater reliability. Psychological Bulletin, 86, 420-428. neuropsychological approach to the Rorschach in patients with dementia Silberg, J. L., & Armstrong, J. G. (1992). The Rorschach test for predicting of the Alzheimer type. Assessment, 3, 351-363. suicide among depressed adolescent inpatients. Journal of Personality Perry, W., Sprock, J., Schaible, D., McDougall, A., et al. (1995). Amphet- Assessment, 59, 290-303. amine on Rorschach measures in normal subjects. Journal of Personality Sines, L. K., Silver, R. J., & Lucero, R. J. (1961). The effect of therapeutic Assessment, 64, 456-465. intervention by untrained "therapists." Journal of Clinical Psychol- Perry, W., & Viglione, D. J. (1991). The Ego Impairment Index as a ogy, 17, 394-396. predictor of outcome in melancholic depressed patients treated with Skelton, M. D., Boik, R. ;., & Madero, J. N. (1995). Thought disorder on tricyclic antidepressants. Journal of Personality Assessment, 56, 487- the WAIS-R relative to the Rorschach: Assessing identity disorder 501. adolescents. Journal of Personality Assessment, 65, 533-549. Perry, W., Viglione, D., & Braff, D. L. (1992). The Ego Impairment Index Sloan, P., Arsenault, L., Hilsenroth, M., Handler, L., & Harvill, L. (1996). and schizophrenia: A validation study. Journal of Personality Assess- Rorschach measures of posttraumatic stress in Persian Gulf War veter- ment, 59, 165-175. ans. Journal of Personality Assessment, 66, 54-64. Peterson, C., Samel, A., von Baeyer, C., Abramson, L. H., Metalsky, G. I., Sloan, P., Arsenault, L., Hilsenroth, M., Harvill, L., & Handler, L. (1995). & Seligman, M. E. P. (1982). The Attributional Style Quesionnaire. Rorschach measures of posttraumatic stress in Persian Gulf War veter- Cognitive Theory and Research, 6, 289-299. ans. Journal of Personality Assessment, 64, 397-414. Piotrowski, C., & Belter, R. W. (1999). Internship training in psychological Spiro, A., Butcher, J. N., Levenson, M. R., Aldwin, C. M., & Bosse, R. assessment: Has managed care had an impact? Assessment, 6, 381-389. (1993, August). Personality change and stability over five years: The Presley, G., Smith, C., Hilsenroth, M., & Exner, J. (in press). Rorschach validity with African Americans. Journal of Personality Assessment. MMPI-2 in older men. Paper presented at the 101st Annual Convention The Psychological Corporation. (1997). Wechsler Adult Intelligence Scale, of the American Psychological Association, Toronto, Ontario, Canada. technical manual (3rd ed.). San Antonio, TX: The Psychological Stedman, J., Hatch, J., & Schoenfeld, L. (2000). Preintemship preparation Corporation. in psychological testing and psychotherapy: What internship directors Putnam, S. H., Kurtz, J. E., & Houts, D. C. (1996). Four-month test-retest say they expect. Professional Psychology: Research and Practice, 31, reliability of the MMPI-2 with normal male clergy. Journal of Person- 321-326. ality Assessment, 67, 341-353. Stone, L. A. (1965). Test-retest stability of the MMPI scales. Psycholog- Rapaport, D., Gill, M., & Schafer, R. (1946). Diagnostic psychological ical Reports, 16, 619-620. testing (Vols. 1 & 2). Chicago: Yearbook Publishers. Strieker, G., & Gold, J. R. (1999). The Rorschach: Toward a nomotheti- Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency cally based, idiographically applicable configurational model. Psycho- of personality traits from childhood to old age: A quantitative review of logical Assessment, 11, 240-250. longitudinal studies. Psychological Bulletin, 126, 3—25. Turner, D. R. (1966). Predictive efficiency as a function of amount of Rogers, R. (Ed.). (1997). Clinical assessment of malingering and deception information and level of professional experience. Journal of Projective (2nd ed.). New York: Guilford Press. Techniques and Personality Assessment, 30, 4-11. . Russ, S. W. (1980). Primary process integration on the Rorschach and Urist, J. (1977). The Rorschach Test and the assessment of object relations. achievement in children. Journal of Personality Assessment, 44, 338- Journal of Personality Assessment, 41, 3-9. 344. Viglione, D. J. (1989). Rorschach science and art. (Review of the Ror- Russ, S. W. (1981). Primary process integration on the Rorschach and schach: A Comprehensive System, Vol. 1, 2nd ed.). Journal of Person- achievement in children: A follow-up study. Journal of Personality ality Assessment, 53, 195-197. Assessment, 45, 473-477. Viglione, D. J. (1997). Problems in Rorschach research and what to do Ryan, J. J., Dunn, G. E., & Paolo, A. M. (1995). Temporal stability of the about them. Journal of Personality Assessment, 68, 590-599. MMPI-2 in a substance abuse sample. Psychotherapy in Private Prac- Viglione, D. J. (1999). A review of recent research addressing the utility of tice, 14, 33-41. the Rorschach. Psychological Assessment, 11, 251-265. Schafer, R. (1954). Psychoanalytic interpretation in Rorschach testing. Viglione, D. J. (2001). Challenges and solutions in Rorschach coding and New York: Grune & Stratton. administration. Manuscript in preparation. Schaich, D. (1999). A synthesis of the conceptual and inferential proce- Viglione, D. J., & Exner, J. E. (1983). The effects of state anxiety and SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 471

limited social-evaluative stress on the Rorschach. Journal of Personality Weiner, I. B. (2001b). Considerations in correcting Rorschach reference Assessment, 47, 150-154. date. Journal of Personality Assessment, 77, 122—127. Viglione, D. J., Gaudiana, S., & Gowri, A. (1997, April). Two questions Weiner, I. B., Exner, J. E., Jr., & Scbiara, A. (1996). Is the Rorschach about Rorschach norms: Cultural issues and divergence. Paper pre- welcome in the courtroom? Journal of Personality Assessment, 67, sented at the Society for Personality Assessment, San Diego, CA. 422-424. Wade, T. C., & Baker, T. B. (1977). Opinions and use of psychological Wood, J. M., & Lilienfeld, S. O. (1999). The Rorschach Inkblot Test: A tests: A survey of clinical psychologists. American Psychologist, 32, case of overstatement? Assessment, 6, 341-349. 874-882. Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000). Wahlberg, K. E., Wynne, L. C., Oja, H., Keskitalo, P., Pykalainen, L., The Rorschach test in clinical diagnosis: A critical review with a Lahti, I., Moring, J., Naarala, M., Sorri, A., Seitamaa, M., Laksy, K., backward look at Garfield (1947). Journal of Clinical Psychology, 56, Kolassa, J., & Tienari, P. (1997). Gene-environment interaction in 395-430. vulnerability to schizophrenia: Findings from the Finnish adoptive fam- Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996a). The Compre- ily study of schizophrenia. American Journal of Psychiatry, 154, 355- hensive System for the Rorschach: A critical examination. Psychological 362. Science, 7, 3-10. Watkins, C., Campbell, V., Nieberding, R., & Hallmark, R. (1995). Con- Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996b). Thinking temporary practice of psychological assessment by clinical psycholo- critically about the Comprehensive System for the Rorschach: A reply to gists. Professional Psychology: Research and Practice, 26, 54—60. Weiner, I. B. (1966). Psychodiagnosis in schizophrenia. New York: Wiley. Exner. Psychological Science, 7, 14-17. Weiner, I. B. (1996). Some observations on the validity of the Rorschach Wood, J. M., Nezworski, M. T., Stejskal, W. J., Garven, S., & West, S. G. Inkblot Method. Psychological Assessment, 8, 206-213. (1999). Methodological issues in evaluating Rorschach validity: A com- Weiner, I. B. (1998). Principles of Rorschach interpretation. Mahwah, NJ: ment on Burns and Viglione (1996), Weiner (19%), and Ganellen Erlbaum. (1996). Assessment, 6, 115-129. Weiner, I. B. (1999). What the Rorschach can do for you: Incremental validity in clinical applications. Assessment, 6, 327-338. Weiner, I. B. (2001a). Advancing the science of psychological assessment: Received July 27, 2000 The Rorschach Inkblot Method as exemplar. Psychological Assess- Revision received February 26, 2001 ment, 13, 423-432. Accepted July 3, 2001

Low Publication Prices for APA Members and Affiliates

Keeping you up-to-date. All APA Fellows, Members, Associates, and Student Affiliates receive—as part of their annual dues—subscriptions to the American Psychologist and APA Monitor. High School Teacher and International Affiliates receive subscriptions to the APA Monitor, and they may subscribe to the American Psychologist at a significantly reduced rate. In addition, all Members and Student Affiliates are eligible for savings of up to 60% (plus a journal credit) on all other APA journals, as well as significant discounts on subscriptions from cooperating societies and publishers (e.g., the American Association for Counseling and Development, Academic Press, and Human Sciences Press).

Essential resources. APA members and affiliates receive special rates for purchases of APA books, including the Publication Manual of the American Psychological Association, and on dozens of new topical books each year.

Other benefits of membership. Membership in APA also provides eligibility for competitive insurance plans, continuing education programs, reduced APA convention fees, and specialty divisions.

More information. Write to American Psychological Association, Membership Services, 750 First Street, NE, Washington, DC 20002-4242.