Reliability in Cognitive Neuroscience: a Meta-Meta-Analysis Books Written by William R
Total Page:16
File Type:pdf, Size:1020Kb
Reliability in Cognitive Neuroscience: A Meta-Meta-Analysis Books Written by William R. Uttal Real Time Computers: Techniques and Applications in the Psychological Sciences Generative Computer Assisted Instruction (with Miriam Rogers, Ramelle Hieronymus, and Timothy Pasich) Sensory Coding: Selected Readings (Editor) The Psychobiology of Sensory Coding Cellular Neurophysiology and Integration: An Interpretive Introduction. An Autocorrelation Theory of Form Detection The Psychobiology of Mind A Taxonomy of Visual Processes Visual Form Detection in 3-Dimensional Space Foundations of Psychobiology (with Daniel N. Robinson) The Detection of Nonplanar Surfaces in Visual Space The Perception of Dotted Forms On Seeing Forms The Swimmer: An Integrated Computational Model of a Perceptual-Motor System (with Gary Bradshaw, Sriram Dayanand, Robb Lovell, Thomas Shepherd, Ramakrishna Kakarala, Kurt Skifsted, and Greg Tupper) Toward A New Behaviorism: The Case against Perceptual Reductionism Computational Modeling of Vision: The Role of Combination (with Ramakrishna Kakarala, Sriram Dayanand, Thomas Shepherd, Jaggi Kalki, Charles Lunskis Jr., and Ning Liu) The War between Mentalism and Behaviorism: On the Accessibility of Mental Processes The New Phrenology: On the Localization of Cognitive Processes in the Brain A Behaviorist Looks at Form Recognition Psychomyths: Sources of Artifacts and Misrepresentations in Scientifi c Cognitive neuroscience Dualism: The Original Sin of Cognitivism Neural Theories of Mind: Why the Mind-Brain Problem May Never Be Solved Human Factors in the Courtroom: Mythology versus Science The Immeasurable Mind: The Real Science of Psychology Time, Space, and Number in Physics and Psychology Distributed Neural Systems: Beyond the New Phrenology Neuroscience in the Courtroom: What Every Lawyer Should Know about the Mind and the Brain Mind and Brain: A Critical Appraisal of Cognitive Neuroscience Reliability in Cognitive Neuroscience: A Meta-Meta-Analysis Reliability in Cognitive Neuroscience: A Meta-Meta-Analysis William R. Uttal The MIT Press Cambridge, Massachusetts London, England © 2013 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email [email protected] or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge, MA 02142. This book was set in Stone Sans and Stone Serif by Toppan Best-set Premedia Limited, Hong Kong. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Uttal, William R. Reliability in cognitive neuroscience : a meta-meta analysis / William R. Uttal. p. ; cm. Includes bibliographical references and indexes. ISBN 978-0-262-01852-4 (hardcover : alk. paper) I. Title [DNLM: 1. Mental Processes — physiology — Review. 3. Brain — physiology — Review. 3. Brain Mapping — Review. 4. Cognition — Review. 5. Meta-Analysis as Topic — Review. 6. Reproducibility of Results — Review. WL 337] 612.8' 233 — dc23 2012016250 10 9 8 7 6 5 4 3 2 1 Preface This book is a study of reliability, of replicability, and of consistency. As such, it is a test of the merit and validity of a body of scientifi c research that has been playing an increasingly central role in the fi eld of cognitive neuroscience. The general question is: Do macroscopic brain images pro- duced by a rapidly evolving technology refl ect, correlate, or represent cognitive processes? The specifi c question asked in this book is: Can this question be better answered by pooling the data from a number of experi- ments than by depending on the results from a single one? The main thesis presented in this book is that despite its unchallengeable utility in ana- tomic and physiological applications, brain imaging research has not pro- vided consistent answers to the general question for cognition. Furthermore, the answer to the specifi c question is that little is gained by pooling brain image data. These theses are based on a review of the empirical literature carried out at several levels of comparison. It is shown that there is great variability not only among individual subjects and experiments but also in the most recent approach to pooling data — meta-analysis. Because of the importance of reliability in scientifi c research, any lack of consistency that is exhibited by the empirical evidence in this fi eld has profound implications. This inconsistency suggests that cognitive neuro- scientists using this approach have not yet proven their interpretations and theories of the relations between this kind of brain activity and cogni- tive processes. What may have appeared to be correlations and relations might have only been illusions of association stimulated by a host of attrac- tive and seductive features of brain imaging technology. Underlying these “ illusions ” may be a proclivity on the part of cognitive neuroscientists to see order in what are essentially random responses and to have mistakenly accepted marginal statistical criteria as proofs. Although there have been For Mitchan We face an abundance of information. Our problem is to fi nd the knowledge in the information. We need methods for the orderly summarization of studies so that knowledge can be extracted from the myriad individual researches. — G. V. Glass (1976, p. 4) Mental faculties are notions used to designate extraordinarily involved complexes of elementary functions. One cannot think of their taking place in any other way than through an infi nitely complex and involved interaction and cooperation of numerous elementary activities, with the simultaneous functioning of just as many cortical zones, and probably of the whole cortex, and perhaps also including even subcortical centers. Thus, we are dealing with a physiological process extending widely over the whole cortical surface and not a localized function within a specifi c region. We must therefore reject as a quite impossible psychological concept the idea that an intellectual faculty or a mental event or a spatial or temporal quality or any other complex, higher psychic function should be represented in a single circumscribed cortical zone, whether one calls this an “ association centre ” or “ thought organ ” or anything else. — K. Brodmann (1909; translated and edited by Garey, 1994, p. 255, as located by Ross, 2010) It ’ s easy to sell simple stories with memorable take-home conclusions (the amygdala is the seat of fear, etc.), but it ’ s harder for people to understand and accept more complex models, I think. — Tor Wager (personal communication, 2011) Contents Preface ix Acknowledgments xv 1 Meta-analysis: The Idea 1 2 Meta-analysis: The Methodology 27 3 On the Reliability of Cognitive Neuroscience Data: An Empirical Inquiry 79 4 Macroscopic Theories of the Mind-Brain 139 5 Current Status and Future Needs 185 Notes 197 Bibliography 209 Name Index 223 Subject Index 229 x Preface indications in the past that all was not well in correlative studies of brain image and cognitive processes, the evidence for inconsistency and unreli- ability is increasing year by year. The reasons that associating brain images and cognitive processes have become such a popular methodology in cognitive neuroscience are both understandable and opaque. Most important of all of the understandable reasons is that the mind-brain relationship is the preeminent question of human existence, and the urge to solve it is intense. Of equally consequen- tial import is the fact that we still have no idea how the wonders of con- sciousness emerge from the electrochemical interactions among the innumerable neurons of the brain. Therefore, we reach out to anything that has even a remote chance of explaining this mysterious and wondrous process. Throughout their history, psychologists and neuroscientists have sought any new technology in the hope that we would be able to fi nd some entr é e into understanding how the brain and the mind are related. Of the basic premise that mind is a brain process, there is no controversy in scientifi c circles; how and at what level of analysis it happens, there are barriers to understanding galore. It is, for example, possible that we are asking the wrong questions. By attaching our research to macroscopic brain imaging procedures, we may be operating at the wrong level of analysis. Indeed, it is possible that the “ where ” question (i.e., in what parts of the brain do activations correlate with cognitive processes?) may be what philosophers would call a bad question. This would certainly be the case if mind emerged from the actions at the microscopic neuronal level — a highly plausible and likely possibility, but one that is still unproven. No matter how plausible the alternatives may be, there has never been an empirical test of this hypoth- esis that meets strict criteria of proof. Another reason that the fi eld has prospered in the face of inconsistent and contradictory data is that the complexities of the brain and the uncon- trolled nature of the cognitive processes generated by specifi c experimental conditions will always permit some sort of a differential response no matter how few or many the subjects or how many experiments are meta- analyzed. Indeed, it is extremely diffi cult to carry out a brain imaging