Brain Informatics (2015) 2:181–195 DOI 10.1007/s40708-015-0020-4 Multimodal neuroimaging computing: the workflows, methods, and platforms Sidong Liu • Weidong Cai • Siqi Liu • Fan Zhang • Michael Fulham • Dagan Feng • Sonia Pujol • Ron Kikinis Received: 29 July 2015 / Accepted: 20 August 2015 / Published online: 4 September 2015 Ó The Author(s) 2015. This article is published with open access at Springerlink.com Abstract The last two decades have witnessed the 1 Introduction explosive growth in the development and use of noninva- sive neuroimaging technologies that advance the research Neuroimaging has profoundly advanced neuroscience on human brain under normal and pathological conditions. research and clinical care rapidly in the past two decades, Multimodal neuroimaging has become a major driver of prominently by magnetic resonance imaging (MRI), com- current neuroimaging research due to the recognition of the plemented positron emission tomography (PET), and clinical benefits of multimodal data, and the better access electroencephalography (EEG)/magnetoencephalography to hybrid devices. Multimodal neuroimaging computing is (MEG). The art of neuroimaging today is shaped by three very challenging, and requires sophisticated computing to concurrent, interlinked technological developments [1]: address the variations in spatiotemporal resolution and Data Acquisition The advances of imaging instrumen- merge the biophysical/biochemical information. We review tation have enabled digital image acquisition, as well as the current workflows and methods for multimodal neu- electronic data storage and communication systems, such roimaging computing, and also demonstrate how to con- as the picture archiving and communication system duct research using the established neuroimaging (PACS). These imaging systems, CT, MRI and PET computing packages and platforms. showed obvious clinical benefits by providing high contrast tissue differentiation. The previous film-based reading was Keywords Multimodal Á Neuroimaging Á Medical image replaced by the electronic displays (axial, coronal and computing sagittal planes of the volume) without losing diagnostic quality. Medical Image Computing The growth of neuroimaging has spurred a parallel development of neuroimaging com- puting methods and workflows, including bias correction, S. Liu (&) Á W. Cai Á S. Liu Á F. Zhang Á D. Feng registration, segmentation, information extraction and School of IT, The University of Sydney, Sydney, Australia visualization. We should note the difference between e-mail: [email protected] neuroimaging and neuroimaging computing. Neuroimag- F. Zhang Á S. Pujol Á R. Kikinis ing focuses on the image acquisition, capturing the snap- Surgical Planning Laboratory, Harvard Medical School, Boston, shot of the brain; whereas neuroimaging computing focuses USA on the computational analysis of the brain images, extracting and enhancing the information of relevance to M. Fulham Department of PET and Nuclear Medicine, Royal Prince Alfred best describe the brain anatomy and function. Hospital, Sydney Medical School, The University of Sydney, Package and Platform Development To fit into research Sydney, Australia and clinical timelines and facilitate translational medicine, the neuroimaging computing methods and workflows are D. Feng Med-X Research Institute, Shanghai Jiao Tong University, often integrated into software packages. Many such pack- Shanghai, China ages were added to imaging systems by the major vendors 123 182 S. Liu et al. of medical imaging equipment and many specialized Fig. 1 Overview of the current status and major components of c companies. However, a greater number of neuroimaging multimodal neuroimaging computing, including neuroimaging modalities, modality-specific computing workflows, multimodal computing packages and platforms are free and open- computing methods, algorithms, task-oriented packages, all-inte- source, designed and supported by the medical imaging grated platforms, and neuroimaging research communities research groups and communities. Multimodal neuroimaging, i.e., the simultaneous imag- ing measurement (EEG/fMRI [2], PET/CT [3]) or sum- including neuroimaging modalities, modality-specific mation of separate measurement (PET and sMRI [4], sMRI computing workflows (a series of tasks), multimodal com- and dMRI [5], fMRI and dMRI [6]), has become an puting methods, algorithms, packages, platforms and com- emerging research area due to better access to imaging munities. MRI, PET, EEG/MEG and their computing devices, especially the hybrid systems, such as PET/CT [7, workflows and methods are discussed in this review. A 8] and PET/MR [9]. The recent advances in neuroimaging neuroimaging computing task in an analysis workflow may computing methods also enabled joint analysis of the be fulfilled by multiple algorithms, and the most widely used multimodal data. The free and open-source software algorithms, e.g., voxel-based morphometry (VBM) [49], are (FOSS) packages and platforms for neuroimaging com- often integrated into software packages, e.g., Statistical puting further facilitate the translation of the multimodal Parametric Mapping (SPM)1, FMRIB Software Library neuroimaging research from the lab to better clinical care. (FSL)2, and Neurostat3. The new imaging tasks also demand Multimodal neuroimaging advances the neuroscience the refinement of existing algorithms and development of research by overcoming the limits of individual imaging new algorithms. Similar algorithms are often developed modalities and by identifying the associations of findings independently in different labs, sometimes with little from different imaging sources. Multimodal neuroimaging awareness of existing packages/platforms. has been used to investigate a multitude of populations and This paper is organized as follows. In Sect. 2,we disorders, such as Alzheimer’s disease (AD) [4, 10–12], elaborate the computing workflows, which consist of a schizophrenia [13–16], epilepsy [3, 17–19], obsessive- number of specific tasks, for individual modalities. In compulsive disorder (OCD) [20–22], bipolar disorder [23, Sect. 3, we review the major multimodal neuroimaging 24], attention-deficit hyperactivity disorder (ADHD) [25– computing methods, i.e., registration, segmentation, feature 27], Autism spectrum disorder (ASD) [28–30], traumatic integration, pattern analysis and visualization. In Sect. 4, brain injury (TBI) [31–34], stroke [35, 36], multiple scle- we introduce the task-oriented packages and platforms for rosis (MS) [37–39], and brain tumors [9, 40–42]. We have the tasks mentioned in previous sections. We focus on the recently reviewed advances in neuroimaging technologies free and open source software (FOSS) in this review, since and the applications of multimodal neuroimaging in these they could help to better realize the quickly evolved neuropsychiatric disorders [43]. Multimodal neuroimaging methods and workflows than their commercial counter- has also been used in many non-clinical applications, such parts, and thus accelerate translational medicine. For the as building brain machine interface (BMI) [44], tracing sake of clarity and precision, the algorithms, packages and neural activity pathways [45] and mapping mind and platforms are not described in detail, but we refer the behavior to brain anatomy [46–48]. interested readers to more specific papers instead. In Multimodal neuroimaging computing is a very chal- Sect. 5, we give one example of brain tumor surgical lenging task due to large inter-modality variations in spa- planning using the established packages and platforms. tiotemporal resolution, and biophysical/biochemical Lastly, we outline the future directions of multimodal mechanism. Compared to single imaging modality com- computing in Sect. 6. puting, it requires more sophisticated bias correction, co- registration, segmentation, feature extraction, pattern analysis, and visualization. Various methods for neu- 2 Modality-specific neuroimaging computing roimaging analysis have been proposed, and many have workflows been integrated into the task-oriented packages or inte- grated platforms. 2.1 Bias and artifacts correction In this paper, we review the state-of-the-art methods and workflows for both modality-specific neuroimaging com- Different neuroimaging techniques have different spa- puting and multimodal neuroimaging computing, and tiotemporal resolutions, and biophysical/biochemical demonstrate how to conduct multimodal neuroimaging research using the established packages and platforms. 1 www.fil.ion.ucl.ac.uk/spm. Fig. 1 provides an overview of the current status and illus- 2 www.fmrib.ox.ac.uk/fsl. trates the major components of neuroimaging computing, 3 http://128.208.140.75/*Download/. 123 Multimodal neuroimaging computing... 183 Modalities sMRI dMRI fMRI PET EEG / MEG Modality-Specific Bias and Artifacts Correction Computing Workflows Tensor / ODF ERP / ERF Skull Stripping Spatial Normalization Estimation Analysis Tissue / ROI Parametric Map Time / Frequency Spatial Smoothing Segmentation Reconstruction Analysis Surface Source Fiber Tracking Parameter Estimation Reconstruction Reconstruction Brain Connec Connectome Functional Brain Mapping Morphometry tome Registration Segmentation Features Fusion Pattern Analysis Visualization Multimodal Rigid Manual / Semi- Feature Multi-Dimensional Computing Methods Classification Registration Auto Outlining Selection Visualization Affine Atlas Inverse Feature Interactive Regression Registration Mapping Embedding Visualization