Review Article Visual Inference for the Funnel Plot in Meta-Analysis

Michael Kossmeier, Ulrich S. Tran, and Martin Voracek

Department of Basic Psychological Research and Research Methods, Faculty of Psychology, University of Vienna, Austria

Abstract: The funnel plot is widely used in meta-analyses to assess potential publication bias. However, experimental evidence suggests that informal, mere visual, inspection of funnel plots is frequently prone to incorrect conclusions, and formal statistical tests (Egger regression and others) entirely focus on funnel plot asymmetry. We suggest using the visual inference framework with funnel plots routinely, including for didactic purposes. In this framework, the type I error is controlled by design, while the explorative, holistic, and open nature of visual graph inspection is preserved. Specifically, the funnel plot of the actually observed data is presented simultaneously, in a lineup, with null funnel plots showing data simulated under the null hypothesis. Only when the real data funnel plot is identifiable from all the funnel plots presented, funnel plot-based conclusions might be warranted. Software to implement visual funnel plot inference is provided via a tailored R function.

Keywords: funnel plot, meta-analysis, publication bias, small-study effects, visual inference

The funnel plot is a widely used diagnostic plot in meta- (Hunter et al., 2014;Simmonds,2015;Tang&Liu,2000; analysis to assess small-study effects and publication bias Terrin, Schmid, & Lau, 2005). in particular (Light & Pillemer, 1984). It was one of the first Formal statistical tests (e.g., Egger regression test, and genuine plots proposed to visualize meta-analytic data and, others) based on funnel plot asymmetry are widely used nexttotheforestplot(Lewis&Clarke,2001), is the most to establish objectivity, while controlling for type I errors. iconic and popular display for this purpose (Schild & All these tests have in common that they are based on fun- Voracek, 2013). nel plot asymmetry quantified via the association of study In essence, the funnel plot is a scatter plot of study effects effects with study standard errors (or respective functions on the abscissa and a study precision measure (preponder- of these). On the other hand, visual inspection of funnel antly, the study standard error) on the ordinate (Sterne & plots allows incorporating a multitude of visually displayed Egger, 2001). Its main idea is that observed effects should statistical information in an exploratory fashion, in order to scatter randomly and symmetrically around the meta- assess the presence and severity of publication bias. Impor- analytic summary effect. As smaller studies are displayed tant questions that can be addressed by visual funnel plot toward the bottom and higher effect size variability is examinations include the following: Which role does statis- expected for these, this gives rise to the characteristic shape tical significance play (as indicated by significance-contour of an inverted funnel for this graphical display. funnel plots)? Is there an abundance of just-significant stud- Certain deviations from this expected funnel plot shape ies? Is asymmetry driven by single-study outliers or clusters are commonly taken as suggestive for publication bias, of studies? although they similarly emerge via true effect heterogeneity As prominently outlined in the Cochrane Collaboration or chance alone. In particular, a frequent observation is that handbook for systematic reviews, formal tests for funnel smaller studies on average report larger effects. This plot asymmetry should never be interpreted in isolation, so-called small-study effect, in turn, leads to asymmetric but rather always, and first of all, in the light of a visual funnel plots (see Figure 1). inspection of the funnel plot (Sterne, Egger, & Moher, https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 However, despite the popularity of the funnel plot, its 2008,p.317). Hence, visual inference might be the suitability to detect publication bias has been questioned sought-after bridge between both worlds: it allows research- (Lau, Ioannidis, Terrin, Schmid, & Olkin, 2006). Indeed, ers to formally safeguard against type I errors, while still empirical research suggests that subjective interpretations being in keeping with the more general diagnostic, open, of whether publication bias is present or absent, based on and explorative nature of visual examination of statistical mere visual funnel plot inspection, often times are wrong graphs.

Ó 2019 Hogrefe Publishing Zeitschrift für Psychologie (2019), 227(1), 83–89 https://doi.org/10.1027/2151-2604/a000358 84 M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis

Figure 1. Two examples of funnel plots including 95% confidence contours and significance contours at the .05 and .01 levels. Left: unsuspicious funnel plot with symmetrical scatter of studies around the summary effect (vertical black line). Right: funnel plot showing conspicuous small- study effects. Evidently, studies with larger standard errors on average observe larger effects, whereas studies with null or negative effects are missing. Hence, the funnel plot is asymmetric. R code to reproduce the figure: https://osf.io/drws7/.

Visual Inference for Statistical of the lineup). A natural choice for the number of plots in a lineup therefore is 20, corresponding to the conventional Graphs alpha level (5%). Just as with conventional statistical tests, a test statistic (the actual plot) is compared to the null Visual inference is a formal inferential framework (Buja distribution (the plots showing data simulated under the et al., 2009) which allows researchers to test whether graph- null hypothesis). At the same time, visual inference differs ically displayed data support a hypothesis or not. The princi- from conventional inference, in that the test statistic is pal idea is that if a suitable statistical plot of actually not compared to the entire null distribution, but rather to observed data indeed is visually distinguishable from corre- a finite number of realizations thereof (Majumder, sponding plots of data simulated under the null hypothesis, Hofmann, & Cook, 2013). then this constitutes evidence against the null hypothesis. Evaluations made by several independent viewers of a In this framework, valid inferences are drawn via the lineup, instead of by a single viewer, can be used for visual so-called lineup protocol. That is, a lineup of diagnostic inference as well. This extension of the basic procedure has plots is constructed for the actually observed data and the the potential to increase the power to correctly reject the null hypothesis a researcher wants to reject. The total null hypothesis. For either two, three, four, five, six, or lineup comprises k plots, of which k 1 plots show data seven viewers of the same lineup of 20 plots, it is sufficient simulated under the null hypothesis, and wherein the single that at least two viewers are able to identify the real-data plot with the actually observed data is positioned randomly. plot to reject the null hypothesis with the alpha level of This lineup then is inspected by a viewer unfamiliar with the procedure not exceeding 5% (for further details, see the actually observed data and their peculiarities with the Majumder et al., 2013). aim to identify the real-data plot. In practice, the primary In recent studies visual inference has shown promise researcher and the viewer of the lineup often times will under scenarios of different statistical plots and data be the same person, which is valid as long as the primary contexts (Chowdhury et al., 2015;Loy,Follett,&Hofmann, researcher is unfamiliar with the shape of the real-data plot. 2016; Loy, Hofmann, & Cook, 2017; Majumder et al., 2013). After the viewer has selected one of the plots in the lineup, the position of the true-data plot in the lineup is revealed. If the real-data plot, showing the actually observed data, indeed visually was noticeably different and therefore iden- Visual Inference for the https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 tifiable by the viewer from all the other plots in the lineup, Meta-Analytic Funnel Plot then the null hypothesis is rejected. If the actually observed data in fact are realizations of the We suggest evaluating meta-analytic funnel plots via visual null hypothesis, the probability to identify the real-data plot inference for two main reasons. First, by controlling for type and therefore to falsely reject the null hypothesis is 1/k. I errors, visual inference has the potential to increase the Hence, the alpha level is controlled by design (i.e., the size (often low) validity of conclusions based on mere visual

Zeitschrift für Psychologie (2019), 227(1), 83–89 Ó 2019 Hogrefe Publishing M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis 85

1. Data 2. Create lineup 3. Inspect lineup 4. Draw inference

ID d se Which funnel plot stands out? Reject H0 1 1.32 0.48 Typical evaluations 2 1.01 0.47 Yes 3 1.13 0.25 Small-study effects discernible? 4 0.69 0.32 Role of statistical significance? Real-data funnel 5 0.78 0.30 Abundance of just-significant plot identified? . . . study outcomes? . . . Conspicuous outliers or clustering No . . . of studies? Retain H0

Figure 2. Visual inference testing procedure using the lineup protocol with funnel plots. Starting from the effect sizes and their standard errors actually observed in the meta-analysis (step 1), a lineup is constructed, showing the real-data funnel plot randomly positioned among null funnel plots (step 2). A viewer visually inspects the funnel plots in the lineup and picks the one that seems most noticeably or eye-catching (step 3). If the

picked funnel plot indeed is the real-data funnel plot, the null hypothesis (H0) used for null plot simulation is rejected (step 4).

inspection of the funnel plot. The lineup protocol allows That is, the study effects yi are independent realizations researchers to safeguard against prematurely interpreting of normal distributions with the same shared expected 2 funnel plot patterns which might be perfectly plausible by value μ, but with study-specific variances σi ,whichare chance. Second, formal statistical tests for funnel plot asym- mainly due to different sample sizes. The FEM therefore metry exclusively focus on the association of study effects assumes that differences between study effects entirely with their standard errors, whereas visual inference pre- are due to sampling error. serves the explorative nature of diagnostic graph inspection. The REM allows the modeling of additional (unsystem- Using visual perception as a formal statistical test allows to atic) random variability between the study effects, which flexibly incorporate a multitude of visual information to exceeds the amount of variability expected under the assess the plausibility of the observed data under the null. FEM. The REM assumes that the observed effects yi can Specifically, we propose using visual funnel plot inference be modeled as ÀÁ as a pretest before drawing further conclusions from the 2 yi ¼ μ þ ui þ ei; with ui N 0; σi and visual inspection of a funnel plot. Only if the actually ÀÁ 2 observed data displayed in a funnel plot visually are distin- ei N 0; τ : guishable from random patterns, any further conclusions 2 2 based on mere visual inspection of the real-data funnel plot In the REM, the variance of each effect is σi þ τ ,and 2 might be warranted. therefore increased by the constant τ , as compared to The procedural details to conduct valid statistical the FEM. Based on these models, two straightforward ways inference using funnel plots are outlined in Figure 2.As toconstructnullplotsforvisualfunnelplotinferenceareas an illustrative example, Figure 3 shows a lineup for visual follows.

inference using a published meta-analytic funnel plot Given n actually observed effects yi,obs, with estimated (Shanks et al., 2015). standard errors σ^i, the estimated meta-analytic summary ^ effect μobs, and an optional estimate for the between-study ^τ2 variance obs, the effects displayed in each null plot are simulated using the following model: ÀÁ Null-Plot Simulation for Visual y ¼ μ^ þ u þ e ; u N 0; σ^2 i; simul obs i i with i i and Funnel-Plot Inference ÀÁ e N 0; ^τ2 : i obs Essential for visual inference is the null distribution used to simulate the data displayed in the null plots, as this directly That is, the effects in each null plot are randomly drawn corresponds to the null hypothesis one seeks to reject. For from normal distributions with expected value equal to the https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 μ^ meta-analysis, natural choices are the fixed-effect model actually observed meta-analytic summary effect obs and (FEM) and the random-effects model (REM). study-specific variances equal to the sum of the observed σ^2 The FEM assumes the observed effects yi can be variance i , and the estimated between-study variance ^τ2 modeled as obs from the actually observed data (REM). For the FEM, ÀÁ ^τ2 is simply set to zero. The null dataset for one null plot 2 obs y ¼ μ þ ui; ui N 0; σ : i with i is then given as the n simulated effects yi,simul and the

Ó 2019 Hogrefe Publishing Zeitschrift für Psychologie (2019), 227(1), 83–89 86 M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis

Figure 3. Example of a funnel plot lineup using data from a published meta-analysis on romantic priming (Shanks et al., 2015). One funnel plot shows the actually observed data, whereas the data in the 19 other funnel plots have been simulated under the null hypothesis of a random- effects meta-analytic model. Shown are 95% confidence contours (black lines), the summary effect (vertical line), and significance contours (dark area indicates the .05 and .01 levels). Only if the real-data funnel plot showing the actually observed data is distinguishable from the null plots and therefore is identifiable, the null hypothesis can be rejected and any further conclusions based on mere visual inspection of the real-data funnel plot might be warranted. The randomly positioned real-data funnel plot is at position 12. R code to reproduce the lineup figure and to conduct visual inference: https://osf.io/6qyg4/.

initially observed corresponding standard errors σ^i.To hypothesis via visual inference was solely due to an excess emphasize, effect sizes are randomly drawn from a null dis- of unsystematic between-study variation in the real-data tribution, with actually observed standard errors regarded plot. An exception to this rule would be when (paralleling as fixed. the Cochran Q procedure as the conventional meta-analytic Under both the FEM and REM scenarios, the null test) the excess of between-study variability itself is a target hypothesis tested against in visual inference is that study for visual inference. In this case, the FEM should be used effect sizes are independent realizations of normal distribu- for null-plot simulation. Finally, alternative models to simu- tions with the same expected value, but different (nonran- late the data for the null funnel plots, including Bayesian dom) variances. In the context of meta-analysis, null models, may well be proposed and used. https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 funnel plots simulated that way are well-behaved, symmet- Quite a number of variants of the classic meta-analytic ric random noise. funnel plot have been proposed, which differ in the statisti- Which of the two models above should be used for null- cal information conveyed and in their diagnostic purpose plot simulation? In most cases, the REM is a suitable default (Langan, Higgins, Gregory, & Sutton, 2012). These variants choice. This allows researchers to exclude the alternative include different choices for the ordinate, the display possibility, namely, that the reason for rejecting the null of study subgroups, confidence contours, significance

Zeitschrift für Psychologie (2019), 227(1), 83–89 Ó 2019 Hogrefe Publishing M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis 87

Figure 4. Example of a funnel plot lineup incorporating subgroups of studies. One funnel plot shows the actually observed data from a published meta-analysis on the Mozart effect (Pietschnig, Voracek, & Formann, 2010), whereas the other 19 funnel plots show data simulated under the null hypothesis of a meta-analytic random-effects model. Study subgroups are depicted with different plotting symbols (white squares: studies from one author group of interest; dark circles: studies from all other authors). Subgroup membership is randomly drawn without replacement from the actually observed data and randomly assigned for each study in each null plot. The randomly positioned real-data funnel plot is at position 9. R code to reproduce the lineup figure and to conduct visual inference: https://osf.io/mx9zy/.

contours, or the regression line from Egger’s test, and the Team, 2018), the package nullabor (Wickham, Chowdhury, Duval-Tweedie trim-and-fill method. The visual inference & Cook, 2014) is available, which provides general-purpose framework can be accommodated to include all these functions to conduct visual inference with arbitrary graphi- variants. As an example, Figure 4 shows a lineup for visual cal displays, including functionalities to reveal the position inference of a meta-analytic funnel plot, incorporating sub- of the real-data funnel plot in the lineup only after inspect- groups, and using data from a published meta-analysis ing a lineup. Building on this, we have developed and (Pietschnig, Voracek, & Formann, 2010). documented the R function funnelinf within the R package metaviz (Kossmeier, Tran, & Voracek, 2018) for specifically conducting visual funnel-plot inference. The funnelinf func- tion provides tailored features, which currently include: https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 Software for Visual Funnel-Plot (1) options for null-plot simulation under both FEM and Inference REM meta-analysis; (2) subgroup analysis; (3)graphical options specific to the funnel plot (significance and confi- For meta-analytic practitioners, an important question is dence contours, and choice of the ordinate); and (4)addi- how to conveniently conduct visual funnel-plot inference. tional options to display various statistical information Within the statistical computing environment R (R Core (Egger’s regression line, and imputed studies by, as well

Ó 2019 Hogrefe Publishing Zeitschrift für Psychologie (2019), 227(1), 83–89 88 M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis

as the adjusted summary effect from, the trim-and-fill empirical inquiry, visual inference with funnel plots is yet method). For further details, example code, and example at its beginning. data we refer to the documentation of package metaviz Visual funnel plot inference also holds potential to serve (https://CRAN.R-project.org/package=metaviz). didactic purposes in meta-analysis and research synthesis, by allowing students and users to collect experience with the manifold shapes and patterns appearing in funnel plots Conclusions and Implications just by chance. For this specific purpose, plot lineups entirely comprised of null plots, also known as the 2009 We propose to present, contemplate, and evaluate the Rorschach protocol of visual inference (Buja et al., ), funnel plot of the actually observed data simultaneously might be used. with null-hypothesis funnel plots. Only if the real-data Software to conduct all these forms of visual inference funnel plot is identifiable from null-plots, the null hypothe- with meta-analytic funnel plots is readily available in the sis is formally rejected and conclusions based on visual form of a tailored function within R package metaviz 2018 inspection of the real-data funnel plot might be warranted. (Kossmeier et al., ). We suggest using visual funnel-plot inference routinely, as it is a convenient way to increase the validity of conclusions based on funnel plots by saving investigators from inter- References preting funnel-plot patterns which might be perfectly plau- Buja, A., Cook, D., Hofmann, H., Lawrence, M., Lee, E. K., Swayne, sible by chance. D. F., & Wickham, H. (2009). Statistical inference for explora- Empirical experiments are suited to examine the power tory data analysis and model diagnostics. Philosophical Trans- of the procedure to reject the null hypothesis in different actions of the Royal Society of London A: Mathematical, scenarios. Using datasets simulated under an alternative Physical and Engineering Sciences, 367, 4361–4383. https:// doi.org/10.1098/rsta.2009.0120 hypothesis of interest, the proportion of corresponding line- Chowdhury, N. R., Cook, D., Hofmann, H., Majumder, M., Lee, E. K., ups leading to correctly rejecting the null hypothesis can be & Toth, A. L. (2015). Using visual statistical inference to better used as a direct estimate of the power of the lineup proce- understand random class separations in high dimension, low dure (Majumder et al., 2013). Ideally, for this purpose larger sample size data. Computational Statistics, 30, 293–316. https://doi.org/10.1007/s00180-014-0534-x numbers of independent viewers would work through Hunter, J. P., Saratzis, A., Sutton, A. J., Boucher, R. H., Sayers, funnel plot lineups from different experimental conditions. R. D., & Bown, M. J. (2014). In meta-analyses of proportion A difficulty in recruiting larger numbers of viewers is the studies, funnel plots were found to be an inaccurate method of likely effect of formal education in meta-analysis and assessing publication bias. Journal of Clinical Epidemiology, 67, 897–903. https://doi.org/10.1016/j.jclinepi.2014.03.003 expertise with funnel plots in particular. The magnitude Kossmeier, M., Tran, U. S., & Voracek, M. (2018). metaviz,. [R ofthiseffectisunknown,butatleastquestionstheuseof software package]. Retrieved from https://CRAN.R-project.org/ online recruitment systems like Amazon’s Mechanical package=metaviz Turk, which has been regularly used in visual inference Langan, D., Higgins, J. P., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of experiments in the past (e.g., Loy et al., 2016;Majumder additional evidence on a meta-analysis. Journal of Clinical 2013 et al., ). Hence, either innovative ways have to be Epidemiology, 65,511–519. https://doi.org/10.1016/j.jclinepi. found to recruit viewers with already existing funnel plot 2011.10.009 expertise or to train viewers unfamiliar with the funnel plot Lau, J., Ioannidis, J. P., Terrin, N., Schmid, C. H., & Olkin, I. (2006). Evidence based medicine: The case of the misleading funnel in an efficient way prior to the experiment. plot. British Medical Journal, 333, 597. https://doi.org/10.1136/ Questions for future experimental research include: bmj.333.7568.597 What is the power of the procedure in different scenarios, Lewis, S., & Clarke, M. (2001). Forest plots: Trying to see the wood for instance, for varying levels of publication bias or and the trees. British Medical Journal, 322, 1479–1480. https:// doi.org/10.1136/bmj.322.7300.1479 between-study heterogeneity? How does the procedure Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of compare to conventional statistical tests for funnel plot reviewing research. Cambridge, MA: Harvard University Press. asymmetry? Are there power differences to detect publica- Loy, A., Follett, L., & Hofmann, H. (2016). Variations of Q-Q plots: tion bias when using different graphical variants of the fun- The power of our eyes!. American Statistician, 70, 202–214. https://doi.org/10.1080/00031305.2015.1077728 nel plot for visual inference, for instance, by additionally https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25 Loy, A., Hofmann, H., & Cook, D. (2017). Model choice and ’ showing Egger s regression line? Which role do viewer diagnostics for linear mixed-effects models using statistics on characteristics and expertise with funnel plots play in suc- street corners. Journal of Computational and Graphical Statistics, cessfully conducting visual inference with funnel plots? 26, 478–492. https://doi.org/10.1080/10618600.2017.1330207 Majumder, M., Hofmann, H., & Cook, D. (2013). Validation of visual What is the power-wise benefit when basing the decision statistical inference, applied to linear models. Journal of the ’ to reject the null hypothesis on more than one viewer s American Statistical Association, 108, 942–956. https://doi.org/ evaluation per lineup? As a promising topic for future 10.1080/01621459.2013.808157

Zeitschrift für Psychologie (2019), 227(1), 83–89 Ó 2019 Hogrefe Publishing M. Kossmeier et al., Visual Inference for the Funnel Plot in Meta-Analysis 89

Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart Tang, J. L., & Liu, J. L. (2000). Misleading funnel plot for detection effect–Shmozart effect: A meta-analysis. Intelligence, 38, of bias in meta-analysis. Journal of Clinical Epidemiology, 53, 314–323. https://doi.org/10.1016/j.intell.2010.03.001 477–484. https://doi.org/10.1016/S0895-4356(99)00204-8 R Core Team. (2018). R: A language and environment for statistical Terrin, N., Schmid, C. H., & Lau, J. (2005). In an empirical computing. Vienna, Austria: R Foundation for Statistical evaluation of the funnel plot, researchers could not visually Computing. identify publication bias. Journal of Clinical Epidemiology, 58, Schild, A. H., & Voracek, M. (2013). Less is less: A systematic 894–901. https://doi.org/10.1016/j.jclinepi.2005.01.006 review of graph use in meta-analyses. Research Synthesis Wickham, H., Chowdhury, N. R., & Cook, D. (2014). nullabor,. [R Methods, 4, 209–219. https://doi.org/10.1002/jrsm.1076 software package]. Retrieved from https://CRAN.R-project.org/ Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., package=nullabor Hickin, N., ... Puhlmann, L. M. C. (2015). Romance, risk, and replication: Can consumer choices and risk-taking be primed History by mating motives? Journal of Experimental Psychology: Gen- Received February 28, 2018 eral, 144, e142–e158. https://doi.org/10.1037/xge0000116 Revision received September 27, 2018 Simmonds, M. (2015). Quantifying the risk of error when inter- Accepted October 22, 2018 preting funnel plots. Systematic Reviews, 4, 24. https://doi.org/ Published online March 29, 2019 10.1186/s13643-015-0004-8 Sterne, J. A., & Egger, M. (2001). Funnel plots for detecting bias in Michael Kossmeier meta-analysis: Guidelines on choice of axis. Journal of Clinical Department of Basic Psychological Research and Research Methods Epidemiology, 54, 1046–1055. https://doi.org/10.1016/S0895- Faculty of Psychology 4356(01)00377-8 University of Vienna Sterne, J. A., Egger, M., & Moher, D. (2008). Addressing reporting Liebiggasse 5 bias. In J. P. Higgins & S. Green (Eds.), Cochrane handbook for 1010 Vienna systematic reviews of interventions (pp. 297–333). Chichester, Austria England: Wiley. [email protected] https://econtent.hogrefe.com/doi/pdf/10.1027/2151-2604/a000358 - Thursday, May 28, 2020 10:19:31 AM Universidade de Sao Paulo IP Address:143.107.196.25

Ó 2019 Hogrefe Publishing Zeitschrift für Psychologie (2019), 227(1), 83–89