UNIVERSITY OF CALIFORNIA, SAN DIEGO How brain images reveal cognition: An ethnographic study of meaning-making in brain mapping practice A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Cognitive Science by Morana Alač Committee in charge: Professor Edwin Hutchins, Chair Professor Lisa Cartwright Professor Seana Coulson Professor Gilles Fauconnier Professor Charles Goodwin 2006 Copyright Morana Alač, 2006 All rights reserved. The dissertation of Morana Alač is approved, and it is acceptable in quality and form for publication on microfilm. ________________________________________ ________________________________________ ________________________________________ ________________________________________ ________________________________________ Chair University of California, San Diego 2006 iii DEDICATION To Josipa iv TABLE OF CONTENTS SIGNATURE PAGE .................................................................................................................iii DEDICATION ...........................................................................................................................iv TABLE OF CONTENTS ............................................................................................................v LIST OF FIGURES ...................................................................................................................vi ACKNOWLEDGMENTS .......................................................................................................... viii VITA ...........................................................................................................................ix ASTRACT OF DISSERTATION.................................................................................................x CHAPTER 1 Introduction ........................................................................................................ 1 CHAPTER 2 Negotiating Pictures of Numbers ..................................................................... 49 CHAPTER 3 From Trash To Treasure: Learning About Brain Images Through Multimodality ......................................................................................................................... 75 CHAPTER 4 I See What You Are Saying: Action As Cognition In fMRI Brain Mapping Practice ......................................................................................................... 107 CHAPTER 5 Seeing Digital Brain Images As Hybrid Objects To Work With...................... 146 CHAPTER 6 Conclusion: Distributed Cognition, Embodiment and Multimodal Interaction In MRI Practice............................................................................................... 184 v LIST OF FIGURES FIGURE 1-1 Representation of electrophysiological recordings (top) vs. fMRI……..…………5 FIGURE 1-2 Conceptual Integration Network ……………………………………………………19 FIGURE 2-1 Figure 2 from Sereno et al., 2001 ...................................................................... 55 FIGURE 2-2 Figure 2 from Sereno et al., 2001 ...................................................................... 61 FIGURE 3-1 An application of the thermal plastic mask ………………………..………………79 FIGURE 3-2 The computer screen displays brain slices over the time-course of the experiment ........................................................................................................ 83 FIGURE 3-3 Chapter3, Excerpt 2 ........................................................................................... 85 FIGURE 3-4 Chapter3, Excerpt 2 ........................................................................................... 86 FIGURE 3-5 Chapter3, Excerpt 2 ........................................................................................... 86 FIGURE 3-6 Chapter3, Excerpt 2 ........................................................................................... 86 FIGURE 3-7 Chapter3, Excerpt 2 .......................................................................................... 86 FIGURE 3-8 Chapter3, Excerpt 2 .......................................................................................... 86 FIGURE 3-9 Chapter3, Excerpt 2 .......................................................................................... 86 FIGURE 4-1 Phase map as viewed on the computer screen ............................................... 111 FIGURE 4-2 Phase map captured at a later stage of the practice after the expert and the novice had already inscribed on it borders of early visual areas…..…………113 FIGURE 4-3 The setting........................................................................................................ 115 FIGURE 4-4 Expanding ring stimulus shown at five moments in time (from Engel et al., 1997) ....................................................................................................................... 116 FIGURE 4-5 Chapter 4, Excerpt 1 ........................................................................................ 118 FIGURE 4-6 The expert’s chart............................................................................................. 121 FIGURE 4-7 Chapter 4, Excerpt 2 ........................................................................................ 122 FIGURE 4-8 Chapter 4, Excerpt 3 ........................................................................................ 127 FIGURE 4-9 Rotating wedge stimulus shown at five moments in time (from Engel et al., 1997) ....................................................................................................................... 132 FIGURE 5-1 Functional and structural images as they appear on the computer screen ..... 151 FIGURE 5-2 Chapter 5, Excerpt 1 ........................................................................................ 154 FIGURE 5-3 Chapter 5, Excerpt 1 ........................................................................................ 154 FIGURE 5-4 Chapter 5, Excerpt 1 ........................................................................................ 155 FIGURE 5-5 Chapter 5, Excerpt 1 ........................................................................................ 155 FIGURE 5-6 Chapter 5, Excerpt 1 ........................................................................................ 155 vi FIGURE 5-7 Chapter 5, Excerpt 2 ........................................................................................ 162 FIGURE 5-8 Chapter 5, Excerpt 2 ........................................................................................ 163 FIGURE 5-9 Chapter 5, Excerpt 2 ........................................................................................ 163 FIGURE 5-10 Chapter 5, Excerpt 2 ...................................................................................... 163 FIGURE 5-11 Detail of the computer screen during the shearing correction…..……..……165 FIGURE 5-12 Schematized representation of the four stages of moving correction............ 166 FIGURE 5-13 Chapter 5, Excerpt 3 ...................................................................................... 168 FIGURE 5-14 Chapter 5, Excerpt 3 ...................................................................................... 168 vii ACKNOWLEDGMENTS This dissertation consists of six chapters. Four of the chapters have been published. The dissertation author is the primary investigator and author for all four published chapters. Chapter 2, in full, is a reprint of Morana Alač: "Negotiating Pictures of Numbers", Journal of Social Epistemology, 18:2, 199-214, 2004. http://www.tandf.co.uk Chapter 3, in full, is a reprint of Morana Alač: "From Trash to Treasure: Learning about the Brain Images through Multimodality", Semiotica, 156-1/4, 177-202, 2005. Chapter 4, in full, is a reprint of Morana Alač & Edwin Hutchins: "I See What You are Saying: Action as Cognition in fMRI Brain Mapping Practice", Journal of Cognition and Culture, 4:3, 629-661, 2004. Permission to reprint was granted by the co-author. viii VITA 1997 B.A. (Hons.) Philosophy, University of Bologna, Italy 2002 Ph.D. in Semiotics, Department of Communication, University of Bologna, Italy 2004 M.S. Cognitive Science, University of California, San Diego, CA. 2006 Ph.D. Cognitive Science, University of California, San Diego, CA ix ABSTRACT OF THE DISSERTATION How brain images reveal cognition: An ethnographic study of meaning-making in brain mapping practice by Morana Alač Doctor of Philosophy in Cognitive Science University of California, San Diego, 2006 Professor Edwin Hutchins, Chair The study investigates the nature of multimodal communication as a process situated in socially organized and culturally rich environments of practice. To do so it describes ways in which neuroscientists make meaning of the visual fMRI evidence. It examines everyday laboratory practices (e.g., how a novice learns through interaction with an expert and the material world of practice, and how neuroimaging data are analyzed and presented to the larger scientific community), to describe how practitioners engage with digital brain images while they coordinate a variety of multimodal semiotic means such as gesture, talk, gaze, and body orientation. The study involves traditional ethnographic observations and digital video and audio recordings of practices conducted in three laboratories in the Cognitive Science department at UCSD and the Salk Institute. By acknowledging the mutual dependence of communication, signification, and inference the study also advances