Analysis, Interpretation, and Visual Presentation of Experimental Data
Total Page:16
File Type:pdf, Size:1020Kb
Chapter to appear in Stevens' Handbook of Experimental Psychology, Volume IV C h a p t e r 16 ANALYSIS, INTERPRETATION, AND VISUAL PRESENTATION OF EXPERIMENTAL DATA Geoffrey R. Loftus, University of Washington The Linear Model 3 Null Hypothesis Significance Testing 4 Problems with The LM and with NHST 4 Pictorial Representations 9 The Use of Confidence Intervals 11 Planned Comparisons (Contrasts) 23 Percent Total Variance Accounted for ( 2) 29 Model Fitting 31 Equivalence Techniques to Investigate Interactions 34 References 38 Following data collection from some experi- techniques best instill understanding in the inves- ment, there arise two goals which should guide tigator to begin with are generally also optimal for subsequent data analysis and data presentation. conveying the data’s meaning to the data’s even- The first goal is for the data collector him or her- tual consumers. self to understand the data as thoroughly as possi- So what are these data-analysis and data- ble in terms of (1) how they may bear on the spe- presentation techniques? It is not possible in a sin- cific question that the experiment was designed to gle chapter or even in a very long book to describe address, (2) what if any surprises the data may them all, because there are an infinite number of have produced, and (3) what, if anything, such them. Although most practicing scientists are surprises may imply about the original questions, equipped with a conceptual foundation with re- related questions, or anything else. The second spect to the basic tools of data analysis and data goal is to determine how to present the data to the presentation, such a foundation is far from suffi- scientific community in a manner that is as clear, cient: it is akin to an artist’s foundation in the tools complete, and intuitively compelling as possible. of color mixing, setting up an easel, understanding This second goal is intimately entwined with the perspective, and the like. To build on this analogy, first: Whatever data analysis and presentation a scientist analyzing any given experiment is like an artist rendering a work of art: Ideally the tools The writing of this chapter was supported by NIMH comprising the practitioner’s foundation should be grant MH41637 to G. Loftus. I thank the late Merrill used creatively rather than dogmatically to pro- Carlsmith for introducing me to numerous of the techniques described in this chapter and David duce a final result that is beautiful, elegant, and Krantz for a great deal of more recent conceptual interesting, instead of ugly, convoluted, and pro- enlightenment about some of the more subtle aspects saic. of hypothesis-testing, confidence intervals and My goal in this chapter is to try to demonstrate planned comparisons. how a number of data-analysis techniques may be STEVENS HANDBOOK OF EXPERIMENTAL PSYCHOLOGY Page 2 used creatively in an effort to understand and con- clearly conveyed to the intended recipients. vey to others the meaning and relevance of a data Somewhere in this chapter is an answer to ap- set. It is not my intent to go over territory that is proximately 70% of these complaints. It is my traditionally covered in statistics texts. Rather, I hope that, among other things, this chapter will have chosen to focus on a limited, but powerful, provide a reference to which I can guide authors arsenal of techniques and associated issues that are whose future work passes across my desk—as an related to, but are not typically part of a standard alternative, that is, to trying to solve what I believe statistics curriculum. I begin this chapter with an to be the world’s data analysis and presentation overview of data analysis as generically carried problems one manuscript at a time. out in psychology, accompanied by a critique of some standard procedures and assumptions, with FOUNDATIONS: THE LINEAR particular emphasis on a critique of null hypothe- sis significance testing (NHST). Next, I discuss a MODEL AND NULL HYPOTHESIS collection of topics that represent some supple- SIGNIFICANCE TESTING ments and/or alternatives to the kinds of standard Suppose that a memory researcher were inter- analysis procedures about which I will have just ested in how stimulus presentation time affects complained. These discussion include (1) a de- memory for a list of words as measured in a free- scription of various types of pictorial representa- recall paradigm. In a hypothetical experiment to tions of data, (2) an overview of the use of confi- answer this question, the investigator might select dence intervals which, I believe, constitutes an J = 5 presentation times consisting of 0.5, 1.0, 2.0, attractive alternative to NHST, (3) a review of the 4.0, and 8.0 sec/word and carry out an experiment benefits of planned comparisons which entail an using a between-subjects design in which n = 20 analysis of percent between-conditions variance subjects are assigned to each of the 5 word- accounted for, (4) a description of techniques in- duration conditions—hence, N = 100 subjects in volving percent total variance accounted for, (5) a all. Each subject sees 20 words, randomly selected brief set of suggestions about presentation of re- from a very large pool of words. For each subject, sults based on mathematical models (meant to the words are presented sequentially on a com- complement the material in the Myung & Pitt puter screen, each word presented for its appropri- chapter) and finally, (6) a somewhat evangelical ate duration. Immediately following presentation description of what I have termed equivalence of the last word, the subject attempts to write techniques. down as many of the words as possible. The in- vestigator then calculates the proportion correct My main expositional strategy is to illustrate number of words (out of the 20 possible) for each through example. In most instances, I have in- subject. vented experiments and associated data to use in the examples. This strategy has the disadvantage The results of this experiment therefore con- that it is somewhat divorced from the real world of sist of 100 numbers: one for each of the 100 sub- psychological data, but has the dominating ad- jects. How are these 100 numbers to be treated in vantage that the examples can be tailored specifi- order to address the original question of how cally to the illustration of particular points. memory performance is affected by presentation time? There are two steps to this data- The logic and mathematical analysis in this interpretation process. The first is the specification chapter is not meant to be formal or complete. For of a mathematical model1, within the context of proofs of various mathematical assertions that I which each subject’s experimentally observed make, it is necessary to consult a mathematically number results from assumed events occurring oriented statistics text. There are a number of such within the subject. There are an infinite number of texts; my personal favorite is Hays (1973), and ways to formulate such a mathematical model. The where appropriate, I supply references to Hays most widely used formulation, on which I will fo- along with specific page numbers. cus on in this chapter, is referred to as the linear My choice of material and the recommenda- model or LM. tions that I selected to include in this chapter have The second step in data interpretation is to been strongly influenced by 35 years of experience carry out a process by which the mathematical in reviewing and journal editing. In the course of model, once specified, is used to answer the ques- these endeavors I have noticed an enormous num- ber of data-analysis and data-presentation tech- niques that have been sadly inimical to insight and 1 I have sometimes observed that the term "mathemati- clarity—and conversely, I have noticed enormous cal model" casts fear into the hearts of many research- numbers of missed opportunities to analyze and ers. However, if it is numbers from an experiment that present data in such a way that the relevance and are to be accounted for, then the necessity of some importance of the findings are underscored and kind of mathematical model is logically inevitable. DATA ANALYSIS, INTERPRETATION AND PRESENTATION Page 3 Table 1. Types of Models. The response measure is R, and the values of independent variables are la- beled X and Y. The model parameters are indicated by Greek letters, and . All models listed are linear models except for the last which is not linear because it includes the product of three parame- ters, i j. Model Model Name R = a + b X + gY Multiple Regression (Additive) R = a + b X + gY + dXY Multiple Regression (Bilinear) R = a + b X + gY + dY2 Multiple Regression (Quadratic in Y) R = a + bi + gj Two-Way ANOVA (Additive) R = a + b i + gj + dij Two-Way ANOVA with Interaction R = a + b X + gj One-Way ANACOVA (Additive) R = a + b i + gj + dbigj Tukey’s one-degree-of-freedom interaction model tion at hand. Note that there are numerous possi- 3. Across the subjects x words population there bilities for how this can be done. The process that is a “grand mean,” denoted m, of the depend- is the most widely used is that of null hypothesis ent variable measured in the experiment. The significance testing, or NHST. grand mean is a theoretical entity, but roughly Most readers of this chapter are probably fa- it can be construed as the number that would miliar with both the LM and the process of NHST. result if all individuals in the target population Nonetheless, to insure a common conceptual and were run in the experiment for an infinite notational foundation, I will describe both of them number of times in all conditions, using in the briefly in the next two sections.