1 Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network Anmol Sharma, Student Member, IEEE, Ghassan Hamarneh, Senior Member, IEEE Abstract—Magnetic resonance imaging (MRI) is being increas- tissue, and certain diagnosis are best performed when a ingly utilized to assess, diagnose, and plan treatment for a variety particular sequence is observed. For example, T1 and T2flair of diseases. The ability to visualize tissue in varied contrasts sequences provide clear delineations of the edema region of in the form of MR pulse sequences in a single scan provides T valuable insights to physicians, as well as enabling automated tumor in case of glioblastoma, 1c provides clear demarcation systems performing downstream analysis. However many issues of enhancing region around the tumor used as an indicator like prohibitive scan time, image corruption, different acquisition to assess growth/shrinkage, and T2flair sequence is used to protocols, or allergies to certain contrast materials may hinder detect white matter hyperintensities for diagnosing vascular the process of acquiring multiple sequences for a patient. This dementia (VD) [2]. poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is In clinical settings, however, it is common to have MRI lost. In this paper, we propose a variant of generative adversarial scans acquired using varying protocols, and hence varying network (GAN) capable of leveraging redundant information sets of sequences per patient. Sequences which are routinely contained within multiple available sequences in order to gen- acquired may be unusable or missing altogether due to scan erate one or more missing sequences for a patient scan. The corruption, artifacts, incorrect machine settings, allergies to proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse certain contrast agents and limited available scan time [3]– sequences and synthesizes the missing ones in a single forward [5]. This phenomenon is problematic for many downstream pass. We demonstrate and validate our method on two brain MRI data analysis pipelines that assume presence of a certain set datasets each with four sequences, and show the applicability of of pulse sequences to perform their task. For instance, most the proposed method in simultaneously synthesizing all missing of the segmentation methods [6]–[10] proposed for brain MRI sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach scans depend implicitly on the availability of a certain set of with competing unimodal and multi-modal methods, and show sequences in their input in order to perform the task. Most that we outperform both quantitatively and qualitatively. of these methods are not designed to handle missing inputs, Index Terms—generative adversarial networks, multi-modal, and hence may fail in the event where some or most of the missing modality, pulse sequences, MRI, synthesis. sequences may be absent. Modifying existing pipelines in order to handle missing sequences is hard, and may lead to performance degradation. I. INTRODUCTION Also, the option of redoing a scan to acquire the miss- EDICAL imaging forms the backbone of the modern ing/corrupted sequence is impractical due to the expensive healthcare systems, providing means to assess, diag- nature of the acquisition, longer wait times for patients with nose,M and plan treatments for a variety of diseases. Imaging non-life-threatening cases, need for registration between old techniques like computed tomography (CT), magnetic reso- and new scans, and rapid changes in anatomy of area in- nance imaging (MRI), X-Rays have been in use for over many between scan times due to highly active abnormalities such decades. Magnetic resonance imaging (MRI) out of these is as glioblastoma. Hence there is a clear advantage in retrieving particularly interesting in the sense that a single MRI scan is a any missing sequence or an estimate thereof, without having arXiv:1904.12200v3 [eess.IV] 2 Oct 2019 grouping of multiple pulse sequences, each of which provides to redo the scan or changing the downstream pipelines. varying tissue contrast views and spatial resolutions, without To this end, we propose a multi-modal generative adver- the use of radiation. These sequences are acquired by varying sarial network (MM-GAN) which is capable of synthesizing the spin echo and repetition times during scanning, and are missing sequences by combining information from all avail- widely used to show pathological changes in internal organs able sequences. The proposed method exhibits the ability to and muscoskeletal system. Some of the commonly acquired synthesize, with high accuracy, all the required sequences sequences are T1-weighted, T2-weighted, T1-with-contrast- which are deemed missing in a single forward pass through enhanced (T1c), and T2-fluid-attenuated inversion recovery the network. The term “multi-modal” simply refers to the fact (T2flair), though there exist many more [1]. that the GAN can take multiple-modalities of available infor- A combination of sequences provide both redundant and mation as input, which in this case represents different pulse complimentary information to the physician about the imaged sequences. Similar to the input being multi-modal, our method generates multi-modal output containing synthesized versions This work was partially supported by the NSERC-CREATE Bioinformatics of the missing sequences. Since most of the downstream 2018-2019 Scholarship. analysis pipelines commonly target C = 4 pulse sequences S Anmol Sharma and Ghassan Hamarneh are with the Medical Image = T , T , T , T as their input [7], [11], [12], we design Analysis Laboratory, School of Computing Science, Simon Fraser University, { 1 1c 2 2flair} Canada. e-mail: asa224, hamarneh @sfu.ca our method around the same number of sequences, although { } 2 we note that our method can be generalized to any number T1. The synthesis step is encoded as a maximum a-posterior C and set S of sequences. The input to our network is a 4- (MAP) estimate of Gaussian distribution parameters built from channel (corresponding to C = 4 sequences) 2D axial slice, a learnt regression tree. Nguyen et al. [27] was one of the first where a zero image is imputed for channels corresponding to to employ DL in the form of location-sensitive deep network missing sequences. The output of the network is a 4-channel (LSDN) for sequence synthesis. LSDN predicts the intensity 2D axial slice, in which the originally missing sequences are value of the target voxel by using voxel-centered patches synthesized by the network. extracted from an input sequence. The network models the The rest of the paper is organized as follows: Section II responses of hidden nodes as a product of feature and spatial presents a review of the MR sequence synthesis literature. responses. Similarly, Bowleset et al. [2] generate “pseudo- Section III provides an overview of the key contributions of healthy” images by performing voxel-wise kernel regression this work. Section IV presents the proposed method in detail. instead of deep networks to learn local relationships between Section V provides details about the method implementation, intensities in T1 and T2flair sequences of healthy subjects. datasets used, as well as outlines experimental setup for the Since most of the methods were based on local features in current work. Section VI discusses the results and observations the form of patches and did not leverage global features of for the proposed method, and finally the paper is concluded the input sequence, Sevetlidis et al. [21] proposed an encoder- in Section VII. decoder style deep neural network trained layer-wise using restricted Boltzmann machine (RBM) based training. The II. RELATED WORK method utilized global context of the input sequence by taking There has been an increased amount of interest in devel- a full slice as input. Recently, Jog et al. [22] propose a random oping methods for synthesizing MR pulse sequences [2], [5], forest based method that learns intensity mapping between [13]–[26]. We present a brief overview of previous work in input patches centered around a voxel extracted from a single this field by covering them in two sections: Unimodal, where pulse sequence, and the intensity of corresponding voxel in both the input and output of the system is a single pulse target sequence. The method utilized multi-resolution patches sequence (one-to-one); and multimodal, where methods are by building a Gaussian pyramid of the input sequence. Yu et able to leverage multiple input sequences to synthesize a single al. [26] propose a unimodal GAN architecture to synthesize (many-to-one) or multiple sequences (many-to-many). missing pulse sequences in a one-to-one setting. The approach uses an edge detection module that tries to preserve the high-frequency edge features of the input sequence, in the A. Unimodal Synthesis synthesized sequence. Recently, Ul Hassan Dar et al. [28] In unimodal synthesis (one-to-one), a common strategy propose to use a conditional GAN to synthesize missing MR includes building an atlas or a database that maps intensity pulse sequences in a unimodal setting for two sequences T1 values between given sequences. Jog et al. [15] used a bagged and T2. ensemble of regression trees trained from an atlas. The training data ( , ) consisted of multiple image patches around A1 A2 A1 a voxel i in a source sequence, and a single intensity value B. Multimodal Synthesis at the same voxel in a target sequence, as . The use of Multimodal synthesis has been a relatively new and un- A2 image patches to predict the intensity value of a single voxel explored avenue in MR synthesis literature. One of the first in output sequence allows representing many-to-many relation- multi-input, single-output (many-to-one) method was proposed ship between intensity values of input and target sequences.