The Influence of Study-Level Inference Models and Study Set Size on Coordinate-Based Fmri Meta-Analyses
Total Page:16
File Type:pdf, Size:1020Kb
ORIGINAL RESEARCH published: 18 January 2018 doi: 10.3389/fnins.2017.00745 The Influence of Study-Level Inference Models and Study Set Size on Coordinate-Based fMRI Meta-Analyses Han Bossier 1*, Ruth Seurinck 1, Simone Kühn 2, Tobias Banaschewski 3, Gareth J. Barker 4, Arun L. W. Bokde 5, Jean-Luc Martinot 6, Herve Lemaitre 7, Tomáš Paus 8, Sabina Millenet 3 and Beatrijs Moerkerke 1 1 Department of Data Analysis, Ghent University, Ghent, Belgium, 2 Department of Psychiatry and Psychotherapy, University Clinic, Hamburg-Eppendorf, Germany, 3 Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany, 4 Department of Neuroimaging, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, United Kingdom, 5 Discipline of Psychiatry, School of Medicine and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland, 6 Institut National de la Santé et de la Recherche Médicale, INSERM Unit 1000 Neuroimaging & Psychiatry, University Paris Sud – Paris Saclay, University Paris Descartes; and Maison de Solenn, Paris, France, 7 Institut National de la Santé et de la Recherche Médicale, INSERM Unit 1000 “Neuroimaging & Psychiatry”, Faculté de médecine, Université Paris-Sud, Le Kremlin-Bicêtre; and Université Paris Descartes, Sorbonne Paris Cité, Paris, France, 8 Baycrest and Departments of Psychology and Psychiatry, Rotman Research Institute, University of Toronto, Toronto, ON, Canada Edited by: Jennifer L. Robinson, Auburn University, United States Given the increasing amount of neuroimaging studies, there is a growing need Reviewed by: to summarize published results. Coordinate-based meta-analyses use the locations Xin Di, of statistically significant local maxima with possibly the associated effect sizes to New Jersey Institute of Technology, aggregate studies. In this paper, we investigate the influence of key characteristics of United States Hui-Jie Li, a coordinate-based meta-analysis on (1) the balance between false and true positives Institute of Psychology, Chinese and (2) the activation reliability of the outcome from a coordinate-based meta-analysis. Academy of Sciences, China More particularly, we consider the influence of the chosen group level model at the study *Correspondence: Han Bossier level [fixed effects, ordinary least squares (OLS), or mixed effects models], the type of [email protected] coordinate-based meta-analysis [Activation Likelihood Estimation (ALE) that only uses peak locations, fixed effects, and random effects meta-analysis that take into account Specialty section: This article was submitted to both peak location and height] and the amount of studies included in the analysis (from 10 Brain Imaging Methods, to 35). To do this, we apply a resampling scheme on a large dataset (N 1,400) to create = a section of the journal a test condition and compare this with an independent evaluation condition. The test Frontiers in Neuroscience condition corresponds to subsampling participants into studies and combine these using Received: 11 August 2017 Accepted: 20 December 2017 meta-analyses. The evaluation condition corresponds to a high-powered group analysis. Published: 18 January 2018 We observe the best performance when using mixed effects models in individual studies Citation: combined with a random effects meta-analysis. Moreover the performance increases Bossier H, Seurinck R, Kühn S, Banaschewski T, Barker GJ, with the number of studies included in the meta-analysis. When peak height is not taken Bokde ALW, Martinot J-L, Lemaitre H, into consideration, we show that the popular ALE procedure is a good alternative in terms Paus T, Millenet S and Moerkerke B of the balance between type I and II errors. However, it requires more studies compared (2018) The Influence of Study-Level Inference Models and Study Set Size to other procedures in terms of activation reliability. Finally, we discuss the differences, on Coordinate-Based fMRI interpretations, and limitations of our results. Meta-Analyses. Front. Neurosci. 11:745. Keywords: coordinate-based meta-analysis, fMRI, group modeling, mixed effects models, random effects models, doi: 10.3389/fnins.2017.00745 reliability Frontiers in Neuroscience | www.frontiersin.org 1 January 2018 | Volume 11 | Article 745 Bossier et al. On Group Level Models and K in CBMA INTRODUCTION (CBMA, see e.g., Paus, 1996; Paus et al., 1998). When full images (and hence information in all voxels) are available, Over the past two decades, there has been a substantial increase methods designed for image-based meta-analysis (IBMA) can in the number of functional Magnetic Resonance Imaging (fMRI) be used (Salimi-Khorshidi et al., 2009; Radua and Mataix-Cols, studies, going from 20 publications in 1994 to over 5000 in 2012). 2015. Despite this vast amount of fMRI literature, it remains In this study, we focus on CBMA for which different challenging to establish scientific truth. algorithms exist (Wager et al., 2007; Radua et al., 2012). In First, fMRI studies tend to have small sample sizes to particular, we consider the popular Activation Likelihood detect realistic effect sizes (median estimated sample size in Estimation (ALE) (Turkeltaub et al., 2002, 2012) and effect 2015 28.5; Poldrack et al., 2017) as among other causes = size based methods such as seed based d-mapping (SBdM, scanning participants is costly and time consuming. The large formerly called effect size-signed differential mapping, multiple testing problem and ensuing corrections make statistical RRID:SCR_002554) (Radua et al., 2012). The ALE algorithm testing in fMRI conservative, thereby further reducing statistical considers a reported local maximum as a center of a spatial power or probability to detect true activation (Lieberman and probability distribution. As such, the method only requires Cunningham, 2009; Durnez et al., 2014). As a consequence, the location of the peak and then searches for brain regions the probability that a statistically significant effect reflects true where spatial convergence can be distinguished from random activation is reduced (Button et al., 2013). This can lead to clustering of peaks. Effect size based methods on the other more false negatives (missing true activation) as well as more hand transform t-values of reported local maxima into effect false positives (detecting activation where there is none) in size estimates and calculate a weighted average of the reported published fMRI studies. Second, the diversity of pre-processing evidence. The weights determine the underlying meta-analysis steps and analysis pipelines have made fMRI studies challenging model. For instance, the weights in seed based d-mapping to replicate (Carp, 2012a,b) even though researchers recognize include within-study and between-study variability which the value of both reproducibility (obtaining identical parameter corresponds to a random effects model. If the weights ignore estimates compared to the original experiment using the same the between-study variability one obtains a fixed effects analysis and data; Poldrack and Poline, 2015) and replicability model. (the ability of an entire experiment to be replicated by gathering In this paper, we evaluate the influence of study characteristics new data using the exact same materials and methods; Patil on the statistical properties of CBMA techniques for fMRI. et al., 2016). Roels et al. (2015) also showed there is variability Previous work by Eickhoff et al. (2016a) and Radua et al. (2012) in the number of significant features (i.e., peaks or clusters already evaluated statistical properties of CBMA algorithms of activity) depending on the data-analytical methods used. or tested software for implementation errors (Eickhoff et al., Several approaches have been offered to overcome these 2016b). However, these studies did not study the effect of input challenges. A first remediating step is to promote transparency, characteristics at the individual study level on the performance pre-registration and open science initiatives such as data of these CBMA algorithms. We investigate the influence of sharing or using standardized protocols in organizing and the group level model on the performance of various CBMA managing data (Poline et al., 2012; Pernet and Poline, 2015; procedures. More specifically, we test the effect of pooling Gorgolewski and Poldrack, 2016; Gorgolewski et al., 2016; subjects at the individual study level using either a fixed effects, Poldrack et al., 2017). A second approach to establish scientific ordinary least squares (OLS) or mixed effects group level model truth across studies, is to accumulate knowledge by scientifically on the outcome of the meta-analyses methods mentioned above. combining previous results using meta-analysis (Lieberman As in Eickhoff et al. (2016a) we also evaluate the effect of and Cunningham, 2009; Yarkoni et al., 2010). Combining the number of studies in the meta-analysis (K). Extending on findings across studies increases power to detect true effects, their work, we consider the case for K 10, 12, 14, 16, 18, = while false positives are not expected to replicate across studies, 20, 30, and 35 when using ALE as well as effect size based given a representative set of unbiased results. Furthermore, CBMA’s using a fixed and random effects model. We consider meta-analyses can generate new scientific questions two performance measures: the balance between false positives (Wager et al., 2009). and true positives