Deep Learning based 3D Image Segmentation Methods and Applications A dissertation presented to the faculty of the Russ College of Engineering and Technology of Ohio University In partial fulfillment of the requirements for the degree Doctor of Philosophy Yani Chen May 2019 © 2019 Yani Chen. All Rights Reserved. 2 This dissertation titled Deep Learning based 3D Image Segmentation Methods and Applications by YANI CHEN has been approved for the School of Electrical Engineering and Computer Science and the Russ College of Engineering and Technology by Jundong Liu Associate Professor of Electrical Engineering and Computer Science Dennis Irwin Dean, Russ College of Engineering and Technology 3 Abstract CHEN, YANI, Ph.D., May 2019, Computer Science Deep Learning based 3D Image Segmentation Methods and Applications (115 pp.) Director of Dissertation: Jundong Liu Medical image segmentation is the procedure to delineate anatomical structures and other regions of interest in various image modalities. While crucial and often a prerequisite step for other analysis tasks, accurate automatic segmentation is difficult to obtain, especially for three dimensional (3D) data. Recently, deep learning techniques have revolutionized many domains of artificial intelligence (AI) including image search, speech recognition and 2D/3D natural image/video segmentation. When it comes to 3D image segmentation, however, the majority of deep learning solutions either treat 3D volumes as stacked 2D slices, overlooking the adjacent information between slices, or directly perform 3D convolutional operations with isotropic kernels that are inconsistent with the anisotropic dimensions in 3D medical data. Neural networks based on 3D convolutions tend to be computationally costly, as well as require much more training data to account for the increased number of parameters that need to be tuned. The scarcity of annotated data in medical imaging also adds up the difficulty. To remedy the aforementioned drawbacks of existing solutions, we propose two works for 3D volume segmentation in this dissertation. The first work is multi-view ensemble convolutional neural network (CNN) framework in which multiple decision maps gener- ated along different 2D views are integrated. The second work is a novel end-to-end deep learning architecture that combines CNN and Recurrent Neural Network (RNN) to bet- ter leverage the dimensional anisotropism in 3D medical data. Our model is designed with the aim to take advantage of CNNs remarkable power in capturing multi-scale 2D features, while rely on multi-view ensemble learning or inter-slice sequential learning to ensure certain level of output consistency through inter-slice contextual constraints. Ex- 4 periments conducted on hippocampus magnetic resonance imaging (MRI) data for both work demonstrate that the multi-view solution and the joint CNN-RNN model achieve sig- nificant improvements over single-view approaches, and outperform many state-of-the-art solutions in hippocampus segmentation. Our work also show better results when compar- ing with 3D CNN method. In addition, we further validate our proposed work on other neuroimage segmentation task, i.e., multiple-class segmentation for brain tumor (glioma) using pre-operative multi-modal MRI scans. Also, experimental results demonstrate that our proposed solutions can effectively improve the accuracy and consistency of the tumor segmentation, and show a very comparative performance when compared to the state-of- the-art solutions. 5 Dedication To my lovely grandfather and parents 6 Acknowledgments I would like to express my special appreciation to my advisor, Dr. Jundong Liu, for his guidance and support over my Ph.D. studies. I am very grateful for his advice and many insightful discussions on my research, especially during the time where it seems there is not an obvious solution. My great appreciation also goes to Dr. Charles D. Smith, our long time collaborator, for his valuable advice on application directions, study interpretations and system designs. I would also like to thank my dissertation committee members, Dr. David Juedes, Dr. Razvan Bunescu, Dr. Chang Liu, Dr. Li Xu and Dr. Sergiu Azicovici, for all your professional guidance either on my research or on my coursework; your suggestions and feedbacks have been absolutely invaluable to me. I would like to express my sincere appreciation for all the service and time you devoted. My gratitude is also with my wonderful lab mates, Huihui Xu, Bibo Shi, who gave me a lot of guidance and help when I just came to Ohio University to pursue my Ph.D degree. Discussions with them and other lab mates, Pin Zhang, Zhewei Wang, Nidel Abuhajar, provided me with a lot of inspirations on my research. I am very grateful to all of you. Last but not least, I want to thank my family, especially my parents, who give me endless believe and love for supporting me anything without asking for any return. Particularly appreciative to my lovely grandfather who keeps encouraging me that I can be a better self. 7 Table of Contents Page Abstract.........................................3 Dedication........................................5 Acknowledgments....................................6 List of Tables......................................9 List of Figures...................................... 10 List of Acronyms.................................... 12 1 Introduction..................................... 13 1.1 Background – image segmentation...................... 13 1.2 Area overview................................. 14 1.3 Contributions................................. 17 1.4 Dissertation overview............................. 18 2 Preliminaries.................................... 20 2.1 Building blocks of CNN........................... 21 2.1.1 Convolution.............................. 22 2.1.2 Activation function.......................... 25 2.1.3 Pooling................................ 27 2.1.4 Upsampling.............................. 28 2.2 Training of CNN............................... 31 2.2.1 Optimization............................. 32 2.2.2 Data augmentation.......................... 34 2.2.3 Regularization............................ 35 2.2.4 Transfer learning........................... 37 2.3 RNN - recurrent neural network....................... 38 3 Literature Review.................................. 40 3.1 CNN based Segmentation........................... 41 3.1.1 Patch-wise CNN for segmentation.................. 42 3.1.2 Fully convolutional network..................... 43 3.1.3 SegNet................................ 44 3.1.4 U-Net................................. 45 3.2 RNN based Segmentation........................... 47 3.2.1 LSTM................................. 47 8 3.2.2 Convolutional LSTM and its application on segmentation...... 49 3.3 Segmentation based on combination of CNN and RNN........... 50 3.3.1 U-Net + Bi-directional CLSTM................... 50 4 Multi-view Ensemble ConvNet for Hippocampus Segmentation.......... 52 4.1 Motivation: Hippocampus segmentation................... 52 4.2 Method.................................... 55 4.2.1 U-Seg-Net.............................. 56 4.2.2 Ensemble-Net............................. 58 4.3 Data...................................... 60 4.4 Experimental settings............................. 61 4.5 Evaluation measurements........................... 61 4.6 Experiment results.............................. 62 4.6.1 U-Seg-Net on nine views....................... 62 4.6.2 Multi-view Ensemble ConvNets................... 63 4.7 Discussion................................... 66 5 Sequential FCN for Hippocampus Segmentation.................. 67 5.1 Method: U-Seg-Net + CLSTM........................ 67 5.1.1 3D U-Seg-Net............................ 70 5.2 Data and experimental setting........................ 71 5.3 Experiment results.............................. 72 5.3.1 CLSTMs............................... 72 5.3.2 Joint Model of U-Seg-Net and CLSTMs............... 74 5.3.3 Comparison with other methods................... 77 5.4 Discussion................................... 79 6 Application on Multi-class Brain Tumor Segmentation.............. 81 6.1 Motivation: Glioma segmentation...................... 81 6.1.1 Prior work.............................. 85 6.2 Proposed models............................... 87 6.2.1 U-Seg-Net+CLSTM with Squeeze-and-Excitation Unit....... 88 6.3 Data...................................... 90 6.4 Experimental settings............................. 91 6.5 Quantitative evaluations............................ 92 6.6 Experiment results.............................. 93 6.6.1 SE-U-Seg-Net + CLSTM...................... 96 6.7 Discussion................................... 97 7 Conclusion and Discussions............................ 99 References........................................ 102 9 List of Tables Table Page 4.1 Three different configurations for our Ensemble-Net. The convolutional layer parameters are denoted as ”conv[kernel size]-[number of kernels]”........ 59 4.2 Mean and standard deviation of Dice ratio (%) and HD for the hippocampus segmentation results using nine different single views. (L: left hippocampus, R: right hippocampus)............................... 63 4.3 Mean and standard deviation of Dice ratio (%) for the hippocampus segmenta- tion results using different combination methods................. 64 4.4 Comparison of the proposed method with other state-of-the-art existing methods on hippocampus segmentation...................... 65 5.1 Early stopping epochs used