Contrast Analysis: a Tutorial. Practical Assessment, Research & Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
Contrast analysis Citation for published version (APA): Haans, A. (2018). Contrast analysis: A tutorial. Practical Assessment, Research & Evaluation, 23(9), 1-21. [9]. https://doi.org/10.7275/zeyh-j468 DOI: 10.7275/zeyh-j468 Document status and date: Published: 01/01/2018 Document Version: Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication: • A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement: www.tue.nl/taverne Take down policy If you believe that this document breaches copyright please contact us at: [email protected] providing details and we will investigate your claim. Download date: 28. Sep. 2021 A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic and database forms. Volume 23 Number 9, May 2018 ISSN 1531-7714 Contrast Analysis: A Tutorial Antal Haans, Eindhoven University of Technology Contrast analysis is a relatively simple but effective statistical method for testing theoretical predictions about differences between group means against the empirical data. Despite its advantages, contrast analysis is hardly used to date, perhaps because it is not implemented in a convenient manner in many statistical software packages. This tutorial demonstrates how to conduct contrast analysis through the specification of the so-called L (the contrast or test matrix) and M matrix (the transformation matrix) as implemented in many statistical software packages, including SPSS and Stata. Through a series of carefully chosen examples, the main principles and applications of contrast analysis are explained for data obtained with between- and within-subject designs, and for designs that involve a combination of between- and within-subject factors (i.e., mixed designs). SPSS and Stata syntaxes as well as simple manual calculations are provided for both significance testing and 2 contrast-relevant effect sizes (e.g., η alerting). Taken together, the reader is provided with a comprehensive and powerful toolbox to test all kinds of theoretical predictions about cell means against the empirical data. The statistical analysis of empirical data serves a (teacher wearing sunglasses or not) by 4 (student sitting single function: to answer research questions about a in first, second, third, or fourth row of the lecturing hall) population on the basis of observations from a sample. between-subject experiment with performance on a In experimental psychology, these questions usually subsequent retention test as the dependent variable. involve specific theoretical predictions about differences Their theoretical prediction is that seating location has a between group or cell means. Researchers, however, do negative linear relation with retention, but only in the not always transform their research question into the condition in which the teacher did not wear sunglasses. proper statistical question, which, among other things, The rationale is that wearing sunglasses effectively involves the application of the correct statistical method decreases any effect of eye contact regardless of where (Hand, 1994). Since any statistical test answers a specific in the classroom the student is located, so that no effect question (also Haans, 2008), conducting an incorrect test of seating location on retention is expected for these results in what Hand (1994) called an error of the third groups. The research questions thus are: Is their specific kind: “giving the right answer to the wrong question” (p. 317). theory-based prediction supported by the data? And if To illustrate the prevalence of such Type III errors, so, how much of the empirical or observed variance consider the following example. A group of researchers associated with the experimental manipulations can be wants to test a specific theory-driven explanation for the explained by the theory? observation that student retention of the discussed Assume that the researchers in our example materials decreases more or less linearly with the consulted a standard textbook to determine how they distance between the student and the teacher in the should analyze their empirical data. Their textbook lecturing hall. This effect of seating location, the would most likely insist on using a factorial ANOVA in researchers theorized, could be explained by the which differences between observed cell means are decreasing frequency of eye contact over distance. To explained by two main effects and an interaction effect. test this theoretical explanation, they conducted a 2 Doing so, the researchers, as expected, found a Practical Assessment, Research & Evaluation, Vol 23 No 9 Page 2 Haans, Contrast Analysis: A Tutorial statistically significant interaction between the seating statistical techniques, however, often produce the most location and sunglasses factors. This interaction effect meaningful and robust results (Cohen, 1990). signifies that the effect of seating location was indeed Yet another compelling explanation for the different with as compared to without the teacher unpopularity of contrast analysis is that the method is wearing sunglasses. However, one should not be lured not implemented, at least not in a convenient point-and- into making a Type III error: Neither the significant click manner, in most statistical software packages. In interaction effect, nor any of the main effects provide an for example SPSS, some contrasts (e.g., Helmert and answer to the researchers’ question of interest. The linear contrasts) are implemented and accessible through interaction effect in a 2 by 4 design has three degrees of the user interface of the various procedures, but these freedom in the numerator of the F-test, and thus is an are generally of limited use when finding answers to omnibus test. In contrast to focused tests (with a single theory-driven questions (i.e., only infrequently do these degree of freedom in the numerator), omnibus tests are contrasts answer a researcher’s question of interest). not directly informative regarding the observed pattern Custom contrasts may be defined in the user interface of the effect. There are multitudes of ways in which the window of the ONEWAY procedure, but they are not effect of seating location may be different between the available for more complex designs (e.g., for designs that different levels of the Sunglasses factor. Yet, only one is include a within-subject factor). To unleash the full predicted by our researchers’ theory. As a result, power of contrast analysis, many statistical software eyeballing a graph depicting observed group means in packages require researchers to specify manually the so- combination with conducting a series of post-hoc called contrast or test matrix L and / or the comparisons will be needed to validate whether the transformation matrix M. In SPSS, this is done through significant interaction is consistent with their the LMATRIX and MMATRIX subcommands of the expectations. None of these post-hoc analyses will General Linear Model (GLM) procedure (IBM Corp., answer directly the researchers’ question of interest, let 2013). In Stata, they can be specified in the alone be informative about the extent to which their MANOVATEST postestimation command of the theory was able to explain the observed or empirical MANOVA procedure (StataCorp LP, 2015). differences between group means (i.e., as reflected in an effect size). Would it not be more fruitful if one could In the present paper, I will demonstrate how to test theoretical expectations directly against the observed specify the L and M matrices to test a wide variety of or empirical data? focused questions in a simple and time efficient manner. First, I will discuss contrast analysis for hypothesis One such method that allows researchers to test testing and effect size estimation in between-subject theory-driven expectations