Complex-valued Deep Learning with Applications to Magnetic Resonance Image Synthesis Patrick Virtue Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2019-126 http://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-126.html August 16, 2019 Copyright © 2019, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. Complex-valued Deep Learning with Applications to Magnetic Resonance Image Synthesis By Patrick M. Virtue A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Engineering – Electrical Engineering and Computer Sciences in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Michael Lustig, Co-chair Professor Stella X. Yu, Co-chair Professor Trevor Darrell Professor Bruno Olshausen Summer 2019 Complex-valued Deep Learning with Applications to Magnetic Resonance Image Synthesis Copyright 2019 by Patrick M. Virtue 1 Abstract Complex-valued Deep Learning with Applications to Magnetic Resonance Image Synthesis by Patrick M. Virtue Doctor of Philosophy in Engineering – Electrical Engineering and Computer Sciences University of California, Berkeley Professor Michael Lustig, Co-chair Professor Stella X. Yu, Co-chair Magnetic resonance imaging (MRI) has the ability to produce a series of images that each have different visual contrast between tissues, allowing clinicians to qualitatively assess pathologies that may be visible in one contrast-weighted image but not others. Unfortu- nately, these standard contrast-weighted images do not contain quantitative values, produc- ing challenges for post-processing, assessment, and longitudinal studies. MR fingerprinting is a recent technique that produces quantitative tissue maps from a single pseudorandom acquisition, but it relies on computationally heavy nearest neighbor algorithms to solve the associated nonlinear inverse problem. In this dissertation, we present our deep learning methods to speed up quantitative MR fingerprinting and synthesize the standard contrast- weighted images directly from the same MR fingerprinting scan. Adapting deep learning methodologies to MR image synthesis presents two specific chal- lenges: 1) complex-valued data and 2) the presence of noise while undersampling. MRI signals are inherently complex-valued, as they are measurements of rotating mag- netization within the body. However, modern neural networks are not designed to support complex values. As an example, the pervasive ReLU activation function is undefined for complex numbers. This limitation curtails the impact of deep learning for complex data applications, such as MRI, radio frequency modulation identification, and target recogni- tion in synthetic-aperture radar images. In this dissertation, we discuss the motivation for complex-valued networks, the changes that we have made to implement complex backpropa- gation, and our new complex cardioid activation function that made it possible to outperform real-valued networks for MR fingerprinting image synthesis. In Fourier-based medical imaging, undersampling results in an underdetermined system, in which a linear reconstruction will exhibit artifacts. Another consequence is lower signal- to-noise ratio (SNR) because of fewer acquired measurements. The coupled effects of low SNR and underdetermined system during reconstruction makes it difficult to model the 2 signal and analyze image reconstruction algorithms. We demonstrate that neural networks trained only with a Gaussian noise model fail to process in vivo MR fingerprinting data, while our proposed empirical noise model allows neural networks to successfully synthesize quantitative images. Additionally, to better understand the impact of noise on undersampled imaging systems, we present an image quality prediction process that reconstructs fully sampled, fully determined data with noise added to simulate the SNR loss induced by a given undersampling pattern. The resulting prediction image empirically shows the effects of noise in undersampled image reconstruction without any effect from an underdetermined system, allowing MR pulse sequence and reconstruction developers to determine if low SNR, rather than the underdetermined system, is the limiting factor for a successful reconstruction. i For Caitlin and Molly ii Contents List of Figuresv List of Tables xiii Acknowledgments xiv 1 Introduction1 1.1 MRI Overview...................................2 1.2 Complex-valued Deep Learning.........................3 1.3 Deep Learning for MR Fingerprinting......................4 1.4 Noise in Undersampled Reconstruction.....................5 1.5 Organization and Previously Published Work..................6 2 Complex-valued Deep Learning7 2.1 Complex numbers.................................8 2.2 Motivation.....................................9 2.2.1 Where Do Complex Values Occur in Deep Learning Applications?..9 2.2.2 Why Not Use a Real Network with Two Channels?.......... 11 2.2.3 Why Not Directly Extend Real-valued Layers to Complex?...... 14 2.3 Complex Calculus Background.......................... 17 2.3.1 Complex-differentiable.......................... 17 2.3.2 Wirtinger/CR Calculus.......................... 18 2.3.3 Gradient Descent............................. 20 2.4 Complex-valued Neural Networks........................ 21 2.4.1 Backpropagation with Real-valued Loss................. 23 2.4.2 Hybrid Real/Complex Networks..................... 25 2.4.3 Complex Activation Functions...................... 28 2.4.4 Well-defined Layers............................ 35 2.4.5 Software.................................. 39 2.5 Discussion..................................... 41 Contents iii 3 Deep Learning for MR Fingerprinting 43 3.1 Background for MR Fingerprinting....................... 44 3.1.1 Measuring Net Magnetization...................... 44 3.1.2 Scanner Parameters............................ 45 3.1.3 Tissue Parameters............................ 46 3.1.4 Modeling the MR System........................ 47 3.1.5 Parameter Mapping and Contrast-weighted Imaging.......... 47 3.1.6 MR Fingerprinting............................ 49 3.2 Complex-valued Deep Learning......................... 51 3.2.1 Methods.................................. 52 3.2.2 Results................................... 54 3.2.3 Discussion................................. 56 3.3 MRF Training Data Synthesis.......................... 59 3.3.1 Introduction................................ 59 3.3.2 Methods.................................. 63 3.3.3 Results................................... 68 3.3.4 Discussion................................. 71 3.4 Direct Contrast Synthesis............................ 73 3.4.1 Methods.................................. 74 3.4.2 Results................................... 80 3.4.3 Discussions and conclusions....................... 83 4 Empirical Effect of Gaussian Noise in Undersampled MRI Reconstruction 87 4.1 Introduction.................................... 87 4.1.1 Measurement Time and SNR...................... 91 4.1.2 Undersampling and Expected Measurement Time........... 92 4.1.3 Image Quality Prediction......................... 93 4.1.4 Weighted Least Squares Reconstruction................. 94 4.2 Methodology................................... 96 4.2.1 Effect of Measurement Time Distribution................ 96 4.2.2 Effects of Measurement Noise and Undersampling Rate........ 98 4.2.3 Image Quality Prediction......................... 98 4.2.4 Regularization Parameter........................ 100 4.3 Results....................................... 100 4.3.1 Effect of Measurement Time Distribution................ 101 4.3.2 Effects of Measurement Noise and Undersampling Rate........ 103 4.3.3 Image Quality Prediction......................... 104 4.3.4 Regularization Parameter........................ 106 4.4 Discussion..................................... 106 Bibliography 109 Contents iv A Chapter 2 Derivations 118 A.1 Complex Normalization.............................. 118 A.2 Phase....................................... 119 A.3 Separable Sigmoid................................ 120 A.4 Siglog....................................... 122 A.5 iGaussian..................................... 123 A.6 Cardioid...................................... 124 A.7 Batch Normalization............................... 125 A.8 Euclidean Loss.................................. 128 B Chapter 4 Derivations 132 B.1 Variance of Added Gaussian Noise........................ 132 B.2 Weighted Least Squares............................. 132 v List of Figures 1.1 Illustration of how a complex-valued neural network could learn a unique trans- form function, despite an under-specified training dataset. Even though the train- ing data lie on a single line, left, the complex identity theorem implies that this is sufficient to define a unique complex-valued holomorphic transform function, cen- ter [1]. However, there is no such theorem when considering real-valued transform functions, right....................................3 1.2 Our proposed direct contrast synthesis (DCS) uses a trained neural network to transform the MR fingerprinting