Statistical Suppression in Psychological Research 1

Incidence and Interpretation of Statistical Suppression

in Psychological Research

Naomi Martinez Gutierrez & Robert A. Cribbie

Quantitative Methods Program

Department of Psychology

York University

Statistical Suppression in Psychological Research 2

Abstract

Suppressors are third variables that increase the predictive power of one or more predictors by suppressing their irrelevant when included in a regression model. Although theoretically and statistically useful, no research has addressed the frequency or interpretation of statistical suppression (SS) in the psychological literature. Two studies explored the nature and interpretation of SS. In the first study, regression analyses were reviewed to determine the frequency with which SS occurs in psychological articles published in 2017. Results indicate that approximately one-third of articles showed evidence of SS, although researchers did not acknowledge or attempt to interpret the SS. The second study reviewed articles containing the keyword ‘suppression’ to assess the interpretations provided by researchers that identified SS.

Results indicate that most researchers do not attempt to classify or interpret SS. Therefore, although SS is common in psychology, scarcely any attempts are made to identify, classify, and/or interpret it.

Statistical Suppression in Psychological Research 3

Incidence and Interpretation of Statistical Suppression

in Psychological Research

Suppressors are variables that remove irrelevant variance from other predictors included in a model, thereby increasing the predictive validity of the variable(s) which have had their irrelevant variance suppressed (Conger, 1974). In other words, suppressors unmask relationships between predictor(s) and outcomes, increasing each suppressed variable’s predictive power.

Despite their usefulness, Pandey and Elliot (2010) note that statistical suppression (SS) remains misunderstood and underreported. However, there are no previous studies that have investigated the frequency of SS within psychology or, subsequently, its interpretation (or lack thereof). Thus, the purpose of this study is to report on the occurrence and interpretation of SS within psychological research. Given the paucity of research on SS, a better understanding of the frequency with which SS occurs and the nature of the interpretations of SS provided by researchers will help clarify the extent to which SS is an issue warranting further investigation.

Quantifying Statistical Suppression

There are a variety of statistical models used in the field of psychology (e.g., multiple regression, structural equation modeling, hierarchical linear modeling), with most being able to easily produce the statistical information necessary to identify the presence of SS. For example, imagine a researcher interested in the partial effects of predictors X1 and X2 on outcome y who adopts the model:

yi = b0 + b1X1i + b2X2i + ei, where yi is the outcome value for individual i, b0 is the predicted value of yi when X1 and X2 are

0, b1 and b2 are the partial regression coefficients (slopes) for predicting y from X1 and X2, respectively, and ei is the portion of yi not explained by b0 + b1X1i + b2X2i. Statistical Suppression in Psychological Research 4

The important for exploring SS include raw sample correlations (r, which are identical to standardized regression coefficients in single predictor models with standardized variables), standardized partial regression weights (β) and partial (pr)/semipartial (sr) correlations, all of which indicate the strength of the relationship between a predictor and an outcome. β refers to the partial regression coefficient from a model in which all variables have been transformed (standardized) to have a mean of 0 and of 1, pr refers to the correlation between a predictor and a criterion when a third variable’s variance is removed from both the predictor and the criterion, and sr refers to the correlation between a predictor and a criterion when a third variable’s variance is removed from the predictor. pr2/sr2 provide an estimate of the proportion of variance in an outcome variable that can be explained by a predictor when other predictors are partialled out (e.g., sr2 represents the increase or decrease in the model

R2 if a predictor is added or removed, respectively, from the model). When there are only two predictors and an outcome (y) in a model, the formula for sr can be written as:

푟푦1 − 푟푦2푟12 푟 = , (1) 푦(1.2) 2 √1 − 푟12

wherein ry(1.2) reflects the sr between the outcome y and predictor X1 when predictor X2 is controlled only from predictor X1. ry1 is the correlation between y and X1, ry2 is the correlation between y and X2, and r12 is the correlation between X1 and X2.

Similarly, the formula for pr is:

푟푦1 − 푟푦2푟12 푟(푦1).2 = . (2) 2 2 √1 − 푟12√1 − 푟푦2 Statistical Suppression in Psychological Research 5

The standardized coefficient β is defined as the standard deviation change expected in the outcome per standard deviation change in the respective predictor. In a model with two predictors and an outcome, the formula for β is:

푟푦1 − 푟푦2푟12 훽푦1.2 = 2 (3) 1 − 푟12 wherein βy1.2 is the partial regression coefficient for X1, partialling out the effects of X2.

Equations 1 and 3 are related in that they share the same numerator. The denominators, however, differ such that (combining Equations 1 and 3):

푟푦(1.2) 훽 = . (4) 푦1.2 2 √1 − 푟12

Equations 2 and 3 are also similar but differ in denominators in that (combining Equations 2 and 3):

2 √1 − 푟푦2 푟(푦1).2 훽 = . (5) 푦1.2 2 √1 − 푟12

In a SS effect, pr, sr or 훽푦1.2 will either be larger or different in sign relative to the respective raw correlation or regression coefficient (ry1/훽푦1,Velicer, 1968). In a two-predictor regression model, this occurs because one predictor is suppressing or explaining irrelevant variance within the other predictor, therefore making the suppressed predictor’s relationship with the outcome stronger. This can be further explained by using models where all variables have been standardized. Both equations 6 and 7 below represent simple models.

Equation 8, on the other hand, is the multiple regression model that includes both of the Statistical Suppression in Psychological Research 6

predictors from equation 6 and 7. Here we use X1 and X2 to represent the standardized predictor variables and y to represent the standardized outcome variable.

푌푖 = 훽0 + 훽1푋1푖 + 휀푖 (6)

푌푖 = 훽0 + 훽2푋2푖 + 휀푖 (7)

푌푖 = 훽0 + 훽3푋1푖 + 훽4푋2푖 + 휀푖 (8)

Statistical suppression occurs when |β3| is greater than |β1| and/or |β4| is greater than |β2|

(or the signs of the coefficients are reversed, e.g.. β3 = .2, β1 = -.3). From the previous discussion, the partial β coefficients could also be converted to pr or sr, wherein |푟(푦1).2| or |푟푦(1.2)| would be compared to |푟푦1|. In summary, derivatives of multiple regression models consist of similar statistics (e.g., β, sr, pr) that allow researchers to determine the allocation of variance amongst the variables involved. When the β, pr and/or sr associated with a particular predictor is greater in magnitude or opposite in sign relative to its raw correlation, this is called a SS effect. There are, however, different subtypes accompanying this effect; these will be discussed in the following section.

Types of Statistical Suppression

Absolute Suppression. Absolute SS was first explained by Horst et al. (1941). Using a two predictor model as an example, absolute SS occurs when one predictor suppresses irrelevant variance in the other and thus the magnitude of the β/sr/pr associated with the suppressed predictor becomes larger than its raw correlation with the outcome (e.g., β3= -.4, β1 = -.2, β4 =

.2, β2 =.3). In other words, β3 is greater in magnitude relative to β1, but retains the same sign. In this situation, X2 acts as the suppressor. A more restrictive version of absolute SS is classical suppression, wherein the suppressor does not correlate with the outcome. Statistical Suppression in Psychological Research 7

A valuable example of absolute SS was provided by Horst (1966). During World War II, the success of pilot training, the outcome, was predicted by paper-delivered tests measuring mechanical, numerical, and spatial ability. Although verbal ability did not predict pilot training, it did correlate with the three abilities being measured, since verbal skills were needed to read and comprehend the tests in the first place. The relationship between the other abilities and the success of pilot training was strengthened when verbal skills was included in the model (i.e., controlled for). Here, verbal ability is an absolute suppressor. Accordingly, it is useful to include it in the regression model so that it can suppress the irrelevant variance in the other predictors related to test taking ability, therefore enhancing their predictive power.

Following Horst et al. (1941), negative SS was discussed by Lubin (1957). Continuing with the two predictor model example, negative SS occurs when the addition of a suppressor variable into a model causes the sign of the other predictor’s β/sr/pr to be reversed. For example, negative SS would be visible if the sign of β3 was opposite to that of β1 (e.g., β3 = -.2, β1= .2). In this scenario, X2 acts as the suppressor.

An example of negative SS comes from a study on sleep deprivation by Tagler, Stanko and Forbey (2017). The raw correlation between perceived behavioural control and actigraphy sleep duration was positive (r = .18), however the standardized partial coefficient was negative

(β = -0.07). Given that only two predictors were included in the multiple , it is possible to pinpoint the second predictor (intention) as the negative suppressor.

Lastly, mutual SS was introduced by Conger (1974). In this situation, both predictors in the two-predictor model obtain a larger β/sr/pr in in magnitude relative to their respective raw correlations with the outcome. For example, β3 = .4, β1 = .2, β4 = .3, β2 = .1). In other words, both X1 and X2 simultaneously act as suppressor and suppressed variables. The sign of the Statistical Suppression in Psychological Research 8 relationship between the predictors is usually opposite that of the correlation between each predictor and the outcome (Conger).

A noted case of mutual SS occurs when two positively correlated dimensions of perfectionism, namely perfectionistic strivings and perfectionistic concerns, are simultaneously added to a regression model that predicts psychological adjustment or maladjustment (Stoeber,

Kobori, & Brown, 2014). Specifically, including perfectionistic concerns cancels out the irrelevant variance in perfectionistic strivings, which makes it a stronger positive predictor of psychological adjustment and a stronger negative predictor for maladjustment. Similarly, perfectionistic striving suppresses the irrelevant variance in perfectionistic concerns, making it a stronger negative predictor of adjustment and a stronger positive predictor of maladjustment

(Stoeber, Kobori, & Brown, 2014).

Although these definitions are widely accepted, some researchers disagree with the characteristics outlined above. For example, Paulhus, Robins, Trzesniewski, and Tracy (2004) argue that all three types of SS are special cases of mutual SS because in all SS effects that they analyzed, not only did the suppressed variable’s β increase in magnitude but the suppressor’s β increased in magnitude as well. However, the definitions provided above for absolute, negative and mutual suppression have gained support in the literature over the past couple decades (e.g.,

Akinwande, Dikko, & Samson, 2015).

Caution/Concern Regarding Suppressor Variables in Applied Research

One reason that has been proposed for why researchers should not take advantage of suppressors is suspicion regarding the lack of replicability of SS. For example, in the field of personality, it has been proposed that researchers should add socially desirable responding as an accompanying predictor to a regression model since it could act as an absolute suppressor Statistical Suppression in Psychological Research 9

(Paulhus, 1991). However, the results of doing this are mixed; some studies find no SS effect, whereas others do. For example, Goldberg, Rorer, and Greene (1970) reported that inclusion of this response bias did not improve overall predictive power, whereas Rosse, Stecher, Miller, and

Levin (1998) found that socially desirable responding acts as a valuable suppressor.

Recently, some studies have found SS effects that consistently replicate. Paulhus, Robins,

Trzesniewski, and Tracy (2004), for example, managed to replicate two SS situations in the field of personality using different samples. Specifically, they were able to show that narcissism acts as a suppressor to enhance the relationship between self-esteem and antisocial behaviour, and that shame and guilt act as mutual suppressors. These findings suggest that some suppressor effects can be reliably replicated and should, therefore, be considered statistically and theoretically beneficial.

Another reason why researchers are wary of SS effects is related to the selection of predictors to include in models. Often, before running primary statistical models, researchers reduce the number of predictors by removing variables that do not correlate highly with the criterion (Horst et al., 1941) to increase the validity of the regression model. Because potential suppressors need not correlate with any given criterion, however, researchers are suspicious that such variables may only reduce the validity and increase collinearity concerns in a model.

However, this wrongly assumes that any particular correlation between a predictor and an outcome variable is an accurate reflection of its respective partial coefficient in a model containing other predictors. The primary reason for running a multiple regression is to explore meaningful partial relationships. Also, although predictors are often removed due to fears of collinearity, collinearity does not affect a model’s overall predictive power even though it affects the standard error of a single predictor (Horst et al., 1941; Pandey & Elliot, 2010). A loss of Statistical Suppression in Psychological Research 10 potential suppressors, on the other hand, does affect a model’s predictive power; it lowers the predictive power by decreasing certain predictors’ partial coefficients relative to their raw coefficients.

The discussion above highlights that while methodologists are slowly starting to grasp the complexities of SS, very little is known about the frequency with which SS occurs in applied psychology research or how well researchers are able to interpret SS effects when they occur.

Thus, this paper utilizes two related studies to explore the incidence and elucidation of SS in the psychological literature. Study One explores articles that have evidence of SS effects, enumerating the frequency with which researchers are aware of the presence of SS effects in their studies. Study Two investigates the interpretation of SS by researchers who have acknowledged the occurrence of SS, noting how many attempt to label and interpret these effects, as well as whether or not such labels and interpretations were accurate.

Study One

Method

The first study reviewed articles from sixteen psychology journals from various subdisciplines to explore SS acknowledgement, classification and interpretation practices by researchers conducting linear models.

Google Scholar was used to identify studies published in the year 2017 that included the terms ‘regression’, ‘correlation’ and at least one of ‘β’, ‘beta’, ‘partial’ or ‘semipartial’ somewhere in the article. The journals chosen to represent a range of specializations in psychology were: Computers in Human Behavior; Applied Psychology; Abnormal Child

Psychology; Personality and Social Psychology; Research in Personality; Personality and

Social Psychology Bulletin; Personality; Applied Developmental Psychology; Experimental Statistical Suppression in Psychological Research 11

Psychology: Applied; Educational Psychology; Abnormal Psychology; European Journal of

Personality; Personality Assessment; Clinical Psychological Science; Journal of Consulting and

Clinical Psychology; Personality Disorders. Only studies that contained both raw correlations and either β, pr or sr were included in the study as this information is necessary in order to identify and interpret the nature of the SS effect. If there was evidence for SS, it was categorized as absolute, negative, or mutual, in accordance with the criteria previously discussed.

Results

Frequency of Statistical Suppression. The number of articles with evidence of SS is reported in Figure 1. The results indicate that 101 (32.90%) of the 307 articles reviewed had evidence of SS; that is, at least one predictor in a regression model within a study had a β, pr or sr that was larger in magnitude (but the same sign as) its respective raw correlation with the outcome or the sign of β, pr or sr was different than that of the raw correlation with the outcome.

Of the articles that had evidence of SS, only two (1.98%) actually acknowledged it. Of the studies that showed evidence of SS, 76 (75.25%) could be easily classified as one of the three types of SS because either (1) only two predictors were involved in the model; or (2) of all the variables involved in a multiple predictor model, only one predictor met the conditions for SS.

More specifically, a greater magnitude β relative to the raw correlation with the outcome was labeled absolute, while a change in the sign of β relative to r was classified as negative. Note that it is possible that multiple predictors combined to suppress irrelevant variability in the suppressed variable since many models had more than two predictors.

The rest of the studies that had evidence of SS (25 or 24.75%) could not be classified because the multi-predictor models had more than one suppressed predictor (e.g., at least two predictors had a β that was larger in magnitude relative to the raw correlation with the outcome). Statistical Suppression in Psychological Research 12

It would be tempting to label this situation mutual SS, however the distribution of variance amongst the variables would not be clear enough to make for a confident assessment. For example, Landers, Bauer, and Callan (2017) conducted a hierarchical regression to test whether the relationship between the use of leaderboards and task performance (the dependent variable) was moderated by goal commitment. In the second step of the regression model, 5 predictors and

4 interaction terms were simultaneously added. Three significant predictors showed an increase in the magnitude of β: leaderboard vs Do your best goal condition (r = -.27, β = -0.46); leaderboard vs Easy goal condition (r = -.29, β = -0.46); and goal commitment (r = .29, β =

0.63). Because all three predictors were suppressed, a specific classification is not possible; were these three instances of absolute SS, or one case of absolute SS and a separate case of mutual

SS? If the latter option, which two predictors are mutual suppressors and which predictor is being absolutely suppressed? Given that multiple predictors were simultaneously added to the model, how the variance is shared amongst the variables is unclear.

Frequency of the Types of Statistical Suppression. The results in Figure 1 indicate that of the studies that showed classifiable evidence of SS, the majority were absolute SS (56.44%), followed by negative SS (21.78%) and mutual SS (21.78%). The total number of SS effects

(101) exceeds the total number of articles that exhibit SS (76) because a few of these studies (17 or 22.37%) had more than one SS effect, each occurring within different statistical models. For example, Tagler, Stanko, and Forbey (2017) conducted two separate multiple regression analyses, wherein one model had sleep diary duration as the outcome and the other actigraphy sleep duration as the outcome. The former model only had two predictors, intention and perceived behavioral control, both of which exhibited a SS effect (r = .38, β = 0.47 for intention; r = .18, β = -0.14 for perceived behavioral control), classified as mutual SS. Likewise, the latter Statistical Suppression in Psychological Research 13 model also had the same two predictors, but only perceived behavioral control was negatively suppressed (r =.18, β = -0.07).

Discussion

In this study it was found that approximately one third of articles with enough information to determine whether or not SS occurred showed evidence of SS. This demonstrates that SS is a common event that needs further investigation. It is important to note that the majority of the articles analyzed (532 or 63.41%) did not provide sufficient information (r and at least one of sr, pr and β) for the reader to make a decision about whether SS was present or not.

This highlights deficiencies in reporting that impact the ability of a reader to understand the relationships present in the data. It is encouraged for all researchers to provide standardized partial coefficients in addition to raw coefficients so that SS effects can be identified.

We found that more than three quarters of SS effects, even without the raw data, could be classified into absolute, negative or mutual suppression. Understanding the type of SS is valuable for gaining a better understanding of the SS effect and variability is being distributed in the model. In order for researchers to capitalize on the benefits of including suppressors in their models, they first must gain an understanding of the nature of the SS effects and relationships.

Possibly the most important finding is that only two of the studies identified the presence of SS effects in their results. This suggests that either researchers are not familiar with the concept of SS or they deem it unimportant to address. Again, as discussed above, in order for researchers to understand and make use of statistical suppressors it is important for authors to explore and report on SS effects in their results (along with an attempt at interpreting the nature of the SS). Statistical Suppression in Psychological Research 14

One potential limitation of the study was that we only used the information from the published articles to assess SS. As it becomes commonplace for researchers to make their raw data available, it will be much easier to explore SS (including the specific nature and types of

SS).

To summarize, our results indicate that psychology researchers frequently encounter SS but do not identify or interpret the SS in their articles. This is an important practice both in terms of understanding the nature of the present results as well as for assisting future researchers in identifying important suppressors to include in their models.

Study Two

The goal of this study was to analyze articles in which the occurrence of SS is acknowledged, and to explore the extent to which authors interpreted and labeled these SS effects. Specifically, how many attempted to label these effects? Of those attempts, how many were correct? Were the effects interpreted? The results of this study will give insight into how well SS is understood in the field.

Method

The search engine Google Scholar was used to search recently published (2000 – 2017) articles within psychology journals that included the keywords ‘statistical suppression’ or

‘suppressor’ anywhere in the document. For a study to be included in the analysis, the raw correlation between the predictors and the outcome and either the β, pr, or sr for the relationship between the predictors and outcome must have been provided. If SS was verified, each SS situation was categorized as absolute, negative, or mutual, and this label was compared against the label provided by the author (if one was provided). Furthermore, all articles were checked for whether an interpretation of the effect was provided. Statistical Suppression in Psychological Research 15

Results

Frequency of the Types of Suppression. The results in Figure 2 indicate that of the 44 studies that mentioned SS, 36 (81.81%) had enough information to deduce what type of SS occurred. That is, the raw correlation and standardized regression coefficient or semi- partial/partial r were reported. Of these studies, the most frequent type of SS was mutual (31 or

49.21%), followed by absolute (11 or 17.46%) and negative (13 or 20.63%). Note that these percentages do not add to 100% because the rest of the SS cases lacked sufficient information to assess the type of suppression (8 or 12.70%).

Frequency of Attempts at Labelling the Type of SS. Of the sixty-three cases of SS, forty-eight (76.19%) were not labeled. Of the 31 mutual SS effects, only nine (29.03%) were labeled while the rest (70.96%) were not. Similarly, of the thirteen negative SS effects, most (10 or 76.92%) were also not given any label. Lastly, of the eleven absolute SS effects, only one was given a label.

Of the fourteen out of 63 SS effects that were labeled (22.22%), most (7 or 50%) of the classifications were incorrect. One issue was that mutual SS was misidentified for classical SS because only one predictor’s increased |β| was taken into consideration. Another issue was that the wrong variable would be incorrectly labeled as the suppressor where, in fact, it was the variable being suppressed.

Frequency of Interpretation. A study was coded as including an interpretation of an SS effect if it addressed the question ‘what does this effect imply?’. In other words, interpretation is not just identifying a SS effect; it involves explaining what the effect means, be it in terms of the future research and/or application. Of a total of 63 SS effects, only 25 (39.68%) contained an acceptable interpretation. An example of a valid interpretation comes from a study analyzing the Statistical Suppression in Psychological Research 16 relationship between social Facebook use, social anxiety and anxiety on Facebook (McCord,

Rodebaugh & Levinson, 2014). In this study, the relationship between social Facebook use and social anxiety is strengthened by the suppressor ‘anxiety on Facebook’. The authors explain that the effect implies that people with high social anxiety, but unexpectedly low levels of anxiety on

Facebook, are more likely to use Facebook for specific social and interactive functions.

Discussion

The finding that the most common type of SS detected by researchers was mutual SS might suggest that researchers are more likely to notice SS if more than one predictor experiences an increase in the magnitude of β relative to r. However, given the frequency of SS found in Study One, it also makes sense that mutual suppression occurs frequently given that many different variables are influenced by the effects of SS.

That most lacked an attempt at labeling the SS suggests that researchers either are unaware of the types of SS or they are unsure how to go about applying an appropriate label. The latter seems more plausible, since researchers that did attempt to label the type of SS struggled with labeling the type of SS correctly.

Similarly, most studies also failed to provide an interpretation of the SS effect. This may suggest that researchers do not know how to interpret SS effects in-context, be it in terms of how it can explain previous findings in the literature, what the effect means in terms of real-life application and/or what future studies should do about the effect. As discussed in Study One, it is important for researchers to provide as much detail as possible regarding their SS effects in order to aid the reader in understanding the observed effects as well as to help future researchers identify valuable suppressor variables. Statistical Suppression in Psychological Research 17

It is also important to acknowledge that a definite contributor to the lack of interpretation of SS effects is that the effects are often very complex and difficult (or impossible) to precisely explain. If it were the case that understanding SS effects was straightforward, many more researchers would be making use of the beneficial effects of including theoretically important suppressors in statistical models.

Limitations

In this study, only the keywords ‘statistical suppression’ and ‘suppressor’ were included in the search engine. This is a limitation in that other terms may have been used by researchers to define SS. Davis (1985), for example, defined SS as an inconsistent system when the effect occurred in a mediation model. Similarly, the term negative effects is used when describing SS, to contrast it to positive confounding (which is simply the usual confounding effect) (MacKinnon, Krull, & Lockwood, 2000). It is possible that researchers identified SS effects in their research but used terms that the search would not detect (even so, it is unlikely that these extra articles would change the findings from the analysis given the size of the effects).

Another limitation is the small sample size. However, a small sample size was simply a function of the fact that many researchers over an eight-year period did not identify the presence of SS and hence our recommendations above (for researchers to be more cognizant of the occurrence of SS) remain valuable.

Conclusion

In summary, about one-third of the articles published in the psychology journals we reviewed had evidence of SS. This implies that SS is a relatively common phenomenon in psychological research and should therefore garner further investigation. Despite it being Statistical Suppression in Psychological Research 18 common-place, most of the authors in the Study One did not address the nature of the SS in their articles.

Of the more than 40 authors that identified evidence of SS in their models, most failed to label the effect (or labeled it incorrectly), suggesting that researchers may not have a good grasp of the different types/forms of SS. We encourage researchers to actively investigate the presence of SS in their research as this is beneficial for both explaining the nature of the effects found in the current study as well as helping to identify potential replicable suppressors for use in future studies.

To summarize, researchers are encouraged to provide standardized partial coefficients including , sr and pr along with their respective raw correlations when reporting the results from their statistical models. This will provide both the researcher and reader with the tools necessary to identify SS. If the researcher finds evidence of SS, they should attempt to classify and interpret the effect in-context. It is also important to highlight that identifying and understanding SS falls under the umbrella of estimation, not null hypothesis significance testing.

The goal is to better understand the coefficients and variables of a model and therefore paying attention to SS is completely consistent with the movement towards estimation and away from solely adopting null hypothesis significance testing.

It is hoped that the results and recommendations of this research will help create an environment where SS is actively discussed, investigated and that researchers, where appropriate, make use of potential suppressors to help expose important relationships among variables within the field of psychology.

Statistical Suppression in Psychological Research 19

Statistical Suppression in Psychological Research 20

References

Akinwande, M.O., Dikko, H.G. and Samson, A. (2015) Variance Inflation Factor: As a Condition for

the Inclusion of Suppressor Variable(s) in Regression Analysis. Open Journal of Statistics, 5,

754-767. http://dx.doi.org/10.4236/ojs.2015.57075

Conger, A. J. (1974). A Revised Definition for Suppressor Variables: A Guide To Their Identification

and Interpretation. Educational and Psychological Measurement, 34, 35–46.

https://doi.org/10.1177/001316447403400105

Davis, J. A., & Weber, R. P. (1985). The logic of causal order (Vol. 55). Sage.

Goldberg, L. R., Rorer, L. G. & Green, M. M. (1970). The usefulness of “stylistic” scales as potential

suppressor or moderator variables in predictions from the CPI. Oregon Research Institute

Research Bulletin, 10, (3). https://doi.org/10.1037/e301622005-001

Horst, P., Wallin, P., Guttman, L., Wallin, F. B., Clausen, J. A., Reed, R., & Rosenthal, E. (1941). The

prediction of personal adjustment. Social Science Research Council.

Horst, P. (1966). Psychological measurement and prediction. Wadsworth, Oxford.

Landers, R. N., Bauer, K. N., & Callan, R. C. (2017). Gamification of task performance with

leaderboards: A goal setting experiment. Computers in Human Behavior, 71, 508–515.

https://doi.org/10.1016/j.chb.2015.08.008

Lubin, A. (1957). Some Formulae for Use With Suppressor Variables. Educational and Psychological

Measurement, 17, 286–296. https://doi.org/10.1177/001316445701700209

MacKinnon, D. P., Krull, J. L., & Lockwood, C. M. (2000). Equivalence of the mediation,

confounding and suppression effect. Prevention Science, 1, 173-181.

https://doi.org/10.1023/A:1026595011371

Statistical Suppression in Psychological Research 21

McCord, B., Rodebaugh, T. L., & Levinson, C. A. (2014). Facebook: Social uses and anxiety.

Computers in Human Behavior, 34, 23–27. https://doi.org/10.1016/j.chb.2014.01.020

Pandey, S., & Elliott, W. (2010). Suppressor Variables in Social Work Research: Ways to Identify in

Multiple Regression Models. Journal of the Society for Social Work and Research, 1, 28–40.

https://doi.org/10.5243/jsswr.2010.2

Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L.

S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp.17-59).

New York: Academic Press.

Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004). Two Replicable Suppressor

Situations in Personality Research. Multivariate Behavioral Research, 39, 303–328.

https://doi.org/10.1207/s15327906mbr3902_7

Rosse, J. G., Stecher, M. D., Miller, J. L., & Levin, R. A. (1998). The impact of response distortion on

preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83,

634–644. https://doi.org/10.1037/0021-9010.83.4.634

Stoeber, J., Kobori, O., & Brown, A. (2014). Examining Mutual Suppression Effects in the Assessment

of Perfectionism Cognitions: Evidence Supporting Multidimensional Assessment. Assessment,

21, 647–660. https://doi.org/10.1177/1073191114534884

Tagler, M. J., Stanko, K. A., & Forbey, J. D. (2017). Predicting sleep hygiene: a reasoned action

approach: Predicting sleep hygiene. Journal of Applied Social Psychology, 47, 3–12.

https://doi.org/10.1111/jasp.12411

Velicer, W. F. (1978). Suppressor Variables and the Semipartial . Educational

and Psychological Measurement, 38, 953–958. https://doi.org/10.1177/001316447803800415

Statistical Suppression in Psychological Research 22

839 total

532 did not meet 307 met criteria criteria

76 had identifiable 25 had unidentifiable 206 had no suppression suppression suppression

57 22 22 Absolute Negative Mutual

Figure 1. Flow chart organizing the analyzed studies from 16 journals that employed regression methods for evidence of suppression effects. The sum of the three types of suppression (n = 101) does not equal to the total number of articles (n = 76) because some articles had more than one type of identifiable suppression.

Statistical Suppression in Psychological Research 23

44 articles

discovered

36 had enough 8 lacked sufficient information to information to asses type of asses type of suppression suppression

22 Did not attempt to label suppression 31 Mutual 1 attempted to label suppression 9 attempted to label suppression (4 erroneous)

15 Interpreted the suppression effect 2 interpreted the suppression effect

10 Did not attempt to label suppression 11 Absolute 1 attempted to label suppression

6 Interpreted the suppression effect

10 Did not attempt to label suppression 13 Negative 3 attempted to label suppression (2 erroneous)

2 Interpreted the suppression effect

Figure 2. Flow chart organizing the analyzed studies that included the term ‘suppression’ in the body of text. The sum of the three types of suppression (n = 55) does not equal to the total number of articles (n = 36) because some articles had more than one type of identifiable suppression.